Since there seems to be some insistence that the OP was unnecessarily endangering the children I would like to point out that cars go out of control all the time -- even to crashing through a restaurant window like happened in my area just a couple days ago. If you think you are 100% safe any place on this planet you're wrong. Maybe you feel safe sitting in your living room but cars have been known to come crashing into houses as well. Forget about home invasions, there's also things like earthquakes, tornadoes, and sinkholes swallowing up houses. Life is dangerous and nobody gets out of it alive. That doesn't excuse doing dangerous things but proper precautions can mitigate dangers but you will never ever eliminate them. People walk along streets all the time on a sidewalk in the city and some of them get hit by cars and killed -- I used to have a nephew that we lost that way. Someone on the side of 4 lane highway often is much farther away from traffic than the person on the sidewalk. Do what you think is prudent and let others do what they think is prudent.
As for the encounter, there's people around who get excited about stuff for no particular reason and often aren't interesting in actually hearing the answers they are demanding of you. Offering to call 911 often does defuse situations like this but I can see it escalating the situation as well. It's hard to know what to do in that situation especially when you have kids around you and you have to be concerned about their safety as well. All I know to offer is morons are like God -- they're everywhere.