I am female and regularly go geocaching alone in rural areas and woods (though being in the UK, the chances of me being eaten by a wild animal are relatively slim). I've often met lone men hiking or dog walking and the only exchange has been a cheery 'good morning' apart from one lovely old gent whose dog took a shine to me and we had a chat about geocaching. My partner has no problem at all with me going out alone and has never tried to tell me that it's not safe.
As with everything in life, there can be some risks, but if you never do anything - or worse, allow someone else to prevent you doing things - you will never enjoy life. In all honesty, I think his objection to geocaching is far-fetched. As others have said, there are far easier ways for predators to find victims! It sounds rather as is he's generally a bit paranoid or controlling.
Happy caching.