I've been taking my almost-5.5 year old caching for almost a year. If you are hiking on trails, you need to exercise the normal caution you do when you take a kid on a hike: give him a whistle, teach him to stay put no matter how long it takes if he gets separated, get him his own backpack with his own water/snacks, teach him not to stick his hands where he can't see.
Other than that, my experience is that it's much easier to look for urban/park caches with a little kid in tow than doing it solo. As above, a 5 year old poking around the underside of a park bench mostly removes any suspicion. The only time I have gotten much of a look was a week ago when I pulled off I-5 for GCM8WB and the boy wanted to take a direct assault on the hill on the side of the road and I figured I'd let him. A guy slowed down to try to figure out what I was doing, but we waved and the fellow moved on.
The only reason not to take your kids caching is if you are easily humiliated by their ability to find micros when you are staring slack-jawed at that pinecone for the fourth time.