Exactly what I was going to say. We cache with our 4 1/2 year old and he loves the first cache of the day, is super excited about the second one (after getting to trade swag in the first) but by the third he can get bored and the fourth is usually a write off.
Stick to larger caches, let your little one trade, pack lots of snacks, do not go out near nap time and know when to give up and deal with a DNF if needed.
We live on Vancouver Island so seeing families wandering in the woods is really common, I've never worried about people looking at us strangely.