Delorme plugin bug with illegal XML characters

[kfir]

I recently got a Delorme PN-40 and while trying to use the Delorme send to GPS plugin I was having geocaches disappearing. After several hours I tracked it down to a geocache log entry with a character which is not valid in the gpx XML file.

 

To reproduce on a PN-40,

1. Create a new geocache file and give it a name.

2. On the website find cache GC3383 and send to GPS. This will give us something which will disappear. The GPS lists 1 cache and reports 1 cache in memory.

3. On the website find cache GC1A7FR and send to GPS. The GPS now lists 2 caches and reports 2 caches in memory.

4. On the website find cache GC19NBY and send to GPS. This one has the invalid character in the log. The GPS now lists 3 caches and reports 3 caches in memory. So far so good.

5. On the GPS close the geocache file which saves the caches in memory to the file. The GPS now lists no caches and reports 0 caches in memory.

6. Open the geocache file you just created and the GPS lists only 1 cache and reports only 1 cache in memory.

 

The geocache file you created on the GPS really does have 3 caches in it. What happens is that when the GPS is reading the cache file, as soon as an invalid character is seen the rest of the caches in the file are skipped. I ran the XML file through an online XML validator which showed the problem. When I opened the file in an editor and removed the offending character ("TM") and reopened the file on the PN-40, all the caches showed up.

 

I don't know if the website should filter illegal XML characters or if the plugin should do it.

 

Windows 7

Internet Explorer 9

Latest Delorme Plugin

PN-40 version 2.8.543156

[kfir]

This week I loaded up 17 caches into a file on my GPS, drove 100 miles up to the mountains, opened the file on the GPS and it listed 0 caches. Argh. Back home with my PC I look inside of the file and all of the caches are there.

 

The bug is in the Delorme browser plugin. Who do I contact to get this bug fixed?

[Moun10Bike]

DeLorme is the developer of the plug-in so I would start there. Can you tell if there are illegal characters somewhere in the file that might be causing the plug-in to fail?

[kfir]

Thanks, I went ahead and contacted them.

 

Can you tell if there are illegal characters somewhere in the file that might be causing the plug-in to fail?

 

That is exactly what is causing the problem. There is a character in the visitor log that is not a valid XML character. I used an XML validation website to help me understand the problem and it pointed me to the offending character. The DTD for the Delorme XML file must have a reduced character set.

[Keystone]

To make things even worse, the geocacher who used the invalid XML character appears to do so in every one of their "found it" logs, as part of a signature line.

[Moun10Bike]

Can you point me to one of these logs? I want to see if we should be restricting that content.

[kfir]

Can you point me to one of these logs? I want to see if we should be restricting that content.

 

Cache GC19NBY

Log entry dated 1/29/2009

Look for the 'TM' character before the words "Happy Caching"

 

If the characters sent to the browser plugins were restricted that would be a good solution.

[jholly]

Can you point me to one of these logs? I want to see if we should be restricting that content.

 

Cache GC19NBY

Log entry dated 1/29/2009

Look for the 'TM' character before the words "Happy Caching"

 

If the characters sent to the browser plugins were restricted that would be a good solution.

It's the trade mark symbol, extended ascii octal 231. Looks like they are in every one of his logs. If your allowing HTML in logs this is probably valid. Probably the Delorme does not handle extended ascii.

 

Edit: loaded GC3GGN3 (recent find by Shelrik so his log was near the top) and my Garmin Etrex 30 handled the log just fine. Know it does not solve your problem, but it is a data point.

Edited July 6, 2012 by jholly

[kfir]

Probably the Delorme does not handle extended ascii.

 

Delorme uses XML files. When I copy the XML file that the browser plugin created, then run it through an online XML validator, the validator complains about that character. If the validator complains it makes sense that the GPS firmware will have a problem. I don't know enough about XML to track down character encoding problems.

 

I can post the XML file (300 lines) created with the steps in the original post if that would help.

[jholly]

Probably the Delorme does not handle extended ascii.

 

Delorme uses XML files. When I copy the XML file that the browser plugin created, then run it through an online XML validator, the validator complains about that character. If the validator complains it makes sense that the GPS firmware will have a problem. I don't know enough about XML to track down character encoding problems.

 

I can post the XML file (300 lines) created with the steps in the original post if that would help.

I ran the downloaded .gpx file from GC3GGN3 through the validator on http://www.xmlvalidation.com and it validated. As I pointed out my Garmin also does not have a problem with the file. I also validated the .gpx on my Garmin, which was downloaded via the Garmin plugin, without error. You may use this information as you wish. But at this point I would be talking to Delorme.

Edited July 7, 2012 by jholly

[kfir]

I ran the downloaded .gpx file from GC3GGN3 through the validator on http://www.xmlvalidation.com and it validated. As I pointed out my Garmin also does not have a problem with the file. I also validated the .gpx on my Garmin, which was downloaded via the Garmin plugin, without error.

Now that is interesting. I ran the same GC3GGN3 cache through the Delorme plugin, ran the resulting .gpx file through the validator (I'v been using the same one as you), and it failed on the 'tm' character. When I look at the file with my hex editor the ASCII code is 0x1A which is a control code.

 

Thanks for helping with this. The extra data helps.

[jholly]

I ran the downloaded .gpx file from GC3GGN3 through the validator on http://www.xmlvalidation.com and it validated. As I pointed out my Garmin also does not have a problem with the file. I also validated the .gpx on my Garmin, which was downloaded via the Garmin plugin, without error.

Now that is interesting. I ran the same GC3GGN3 cache through the Delorme plugin, ran the resulting .gpx file through the validator (I'v been using the same one as you), and it failed on the 'tm' character. When I look at the file with my hex editor the ASCII code is 0x1A which is a control code.

 

Thanks for helping with this. The extra data helps.

Pulling the .gpx file into notepad the tm symbol does display correctly in notepad. My hex editor shows different results. Have you tried downloading the gpx directly to disc and then dragging and dropping on your unit to bypass the plugin?

[kfir]

I am not a paid member so cannot download a gpx file from the website.

[jholly]

I am not a paid member so cannot download a gpx file from the website.

Interesting. I was not aware that the plugins could pull full cache data for regular members. Well at this time I think the problem is on Delorme's side of the fence, either the plugin can not handle extended ascii or the unit itself can not handle extended ascii, or both.

[+TomToad]

Tried the above with my Delorme PN-60. Have the same bug with it. Tried creating a PQ with the problem cache and transferred the PQ using both Cache Register, and with Drag-n-Drop, and the file behaved perfectly. Even edited the file to force a save/reload of the gpx file. It is definitely a bug with the plugin itself and not with the GPSr. Jholly would not have a problem since he is using the Garmin plugin and not the Delorme.

 

Probably the best way to get this problem solved would be in the Delorme forums.

 

Edit:

The Delorme forums are aware of the problem. Here is a description of the problem: http://forum.delorme.com/viewtopic.php?p=200195#p200195

 

Edit2:

I have created a nice workaround program for Windows here http://www.filedropper.com/delormepluginfix

Just unarchive the file to its own directory. No need to install anything. On your Delorme, choose Settings, Connect to Computer, When Connected, Open Internal Drive (or Open SD Card if you store your caches on SD). Then run the program DelormePluginFix.exe. a requester opens asking for the source file, just navigate to the Delorme folder (usually EM_USERMAPS), then waypoints, and finally the offending gpx file. Next requester asks where you want to save. You can save to the same folder, but use a different filename, such as GPXFileFixed.gpx. The program will then process the file and correct the problem. Now eject the device and click on Use GPS. Go to Geocaches and open the fixed file.

Edited July 9, 2012 by TomToad

[kfir]

Interesting. I was not aware that the plugins could pull full cache data for regular members.

It does not pull full cache data. It does not retrieve the description or hints for unpaid members. I have to copy those into the GPX file by hand.

Edited July 9, 2012 by kfir

[kfir]

Edit:

The Delorme forums are aware of the problem. Here is a description of the problem: http://forum.delorme.com/viewtopic.php?p=200195#p200195

Thanks for pointing me to this. I hope Delorme gets around to fixing it.

 

Edit2:

I have created a nice workaround program for Windows...

I tried it and found the following.

 

1. It does change the offending character. It changes the single hex character 0x1A to 3 hex characters 0xE2 0x84 0xA2. Notepad displays those as 3 separate characters (the ones in the ASCII chart). The file does pass XML validation. The file does read correctly on the PN-40. The character displayed on the PN-40 is a single question mark which is not one of those 3 hex characters.
   
2. I used my own editor to change the original offending character from 0x1A to 0x99 (the 'TM' character). Notepad displays the character correctly. The file does pass XML validation. The file does not read correctly on the PN-40.
   

I'm going to create my own Windows program instead. I want to be able to select a single file or a directory of files. I also want to exclude / convert the following characters (specified in hex):

 

• 0-1F - control characters
  
• 26 - ampersand (fails XML test)
  
• 3C - less than (fails XML test)
  
• 7F - delete character
  
• 80-FF - extended characters
  

I have tested the remaining character set on the PN-40 and they all allow the GPX file to open and the characters are displayed correctly.

 

Edit: Come to think of it, I cannot check for '&' or '<' since those are used by XML. The only way to detect those properly is with an XML parser. These being displayed properly in HTML might mean they are already encoded correctly in the XML file. I found an occurence of & and it is encoded correctly.

Edited July 9, 2012 by kfir

[+TomToad]

I checked both a bad gpx file and a good gpx file with a hex editor. In a bad gpx file, I have the character 1A. In a good gpx file, it is 3 characters E2 84 A2. That is why my program replaces the 1 character with 3. It works fine for me. The TM symbol shows up in Notepad with the Times New Roman font, and shows correctly on the GPSr. I am using the PN-60. Maybe the PN-40 is incapable of showing the TM symbol? If you click on "GPX File" from the cache page, then save the file to the PN-40, does the log show TM correctly then?

 

At the very least a "?" is better than disappearing caches.

 

Edit: After researching a bit, I find that E2 84 A2 is UTF-8 encoding for the TM symbol. Since the gpx file is encoded in UTF-8, that is why that sequence appears. If you don't see the TM in notepad, then you must be using an older version which doesn't support UTF-8.

Edited July 9, 2012 by TomToad

[jholly]

I checked both a bad gpx file and a good gpx file with a hex editor. In a bad gpx file, I have the character 1A. In a good gpx file, it is 3 characters E2 84 A2. That is why my program replaces the 1 character with 3. It works fine for me. The TM symbol shows up in Notepad with the Times New Roman font, and shows correctly on the GPSr. I am using the PN-60. Maybe the PN-40 is incapable of showing the TM symbol? If you click on "GPX File" from the cache page, then save the file to the PN-40, does the log show TM correctly then?

 

At the very least a "?" is better than disappearing caches.

 

Edit: After researching a bit, I find that E2 84 A2 is UTF-8 encoding for the TM symbol. Since the gpx file is encoded in UTF-8, that is why that sequence appears. If you don't see the TM in notepad, then you must be using an older version which doesn't support UTF-8.

The is exactly what I see in my hex editor, notepad on win7 shows the tm symbol and my etrex 30 displays the tm symbol. As I said before, the problem is on Delorme's side of the fence. And yes, the .gpx file is utf-8, says so right there on the first line.

[+dprovan]

By the way, I ran into this bug a long time ago with a character in some foreign language text in a cache's description, so it's not limited to logs or the TM symbol, just in case anyone doesn't understand that. Since I almost never use the plug-in, I just kinda forgot about it, but it is a very annoying problem.

[kfir]

If you click on "GPX File" from the cache page, then save the file to the PN-40, does the log show TM correctly then?

 

I am not a paid member so the "GPX File" option is not available to me.

 

Edit: After researching a bit, I find that E2 84 A2 is UTF-8 encoding for the TM symbol. Since the gpx file is encoded in UTF-8, that is why that sequence appears. If you don't see the TM in notepad, then you must be using an older version which doesn't support UTF-8.

 

I did a search and verified that. I discovered why Notepad was not showing the TM character for me. Notepad does not automatically select the type of character encoding. But when I chose the UTF-8 encoding option on the Open dialog then the TM character is displayed in Notepad.

 

Thanks guys, for all your help.

[kfir]

By the way, I ran into this bug a long time ago with a character in some foreign language text in a cache's description, so it's not limited to logs or the TM symbol, just in case anyone doesn't understand that. Since I almost never use the plug-in, I just kinda forgot about it, but it is a very annoying problem.

Thanks for the info. Another good reason for me not to bother with testing what extended characters may or may not cause errors on the PN-40.