Unfortunately this is what caused us to purchase a handheld GPSr.
The TomTom just does not cut it for off road excursions, it will get you somewhat close but it keeps jumping back to the roads.
We have purchased 2 handhelds (Garmin Vista Cx & Garmin Legend) since & still use the TomTom for road navigation to the cache parking location.