Weirdly I have literally _just_ done that, using a GPX track dump to add a walking track that went right past a cache of ours. Though you do lose a lot moving from Google to OSM, particularly the satellite/earth view & streetview, both of which I use extensively in pre-trip planning. One of the first annoyances I noted with streetview was that the old red "teardrop" marking current coordinates was replaced by an almost invisible "X".