The TSRORexx Crew are 5 and 3 years. I've found that Terrain of 3 or higher is more than they can handle. Many of the terrain 2's I wouldn't try if there's a significant distance to walk as you'll inevitably be carrying more than a GPSr.
As for your youngest, a child backpack is a necessity. The more you pay, the more comfortable they are and more stable they are when you set the pack down to crouch and poke around the bushes. I cannot remember the brand my buddy uses for his two year old when he's fishing/geo-caching, but it's quite the set-up and very durable. I'd say, eBay it! ::smirk::
As other posters have stated, you may want to stick to the parks and other urban locales when the kids are in tow (Terrain 1) and leave the more arduous caches for when the kids are with the grandparents/sitter/friends.
Happy caching!