Jump to content

Major image hosting problem


fizzymagic

Recommended Posts

Aye, if this were something, say a change in policy about allowable images, that was being newly enforced from this date moving forward, then a public announcement would suffice (the only anger likely being that new puzzles couldn't employ prior technical strategies), and existing listings would be 'grandfathered' (or given time to change/archive).

But this change is breaking old and existing puzzle caches, with many cache owners being none the wiser.  There is some very bad development/rollout practices being employed here for the sake of silently (relatively speaking) adhering to privacy laws. Things are breaking. People aren't being told, warned, informed. That ain't a good thing.  :(

  • Upvote 2
  • Helpful 4
Link to comment
45 minutes ago, thebruce0 said:

Aye, if this were something, say a change in policy about allowable images, that was being newly enforced from this date moving forward, then a public announcement would suffice (the only anger likely being that new puzzles couldn't employ prior technical strategies), and existing listings would be 'grandfathered' (or given time to change/archive).

But this change is breaking old and existing puzzle caches, with many cache owners being none the wiser.  There is some very bad development/rollout practices being employed here for the sake of silently (relatively speaking) adhering to privacy laws. Things are breaking. People aren't being told, warned, informed. That ain't a good thing.  :(

 

I'm scratching my head as to why a PNG image has to be converted to JPEG to protect the viewer's privacy. A quick search hasn't revealed anything about PNG privacy issues, but most of my search hits referred to Papua New Guinea's privacy laws and had nothing to do with image formats.

 

It does explain something odd though. Back in November, a new puzzle cache was published here which had a .png background image stored on GS's server as part of the cache gallery. Except the innards of that file weren't PNG but JPEG, which resulted in me spending several fruitless days picking it apart as I thought that was a clue to the puzzle. It wasn't and the CO had no idea how it happened.

 

I've done many puzzle caches that relied on having exact RGB values in some or all of the image, which is destroyed when converting to JPEG. This move would appear to pretty much put an end to a large class of image steganography in puzzle caches. What will be next to go under the streamroller?

  • Helpful 2
Link to comment
56 minutes ago, barefootjeff said:

I'm scratching my head as to why a PNG image has to be converted to JPEG to protect the viewer's privacy. A quick search hasn't revealed anything about PNG privacy issues, but most of my search hits referred to Papua New Guinea's privacy laws and had nothing to do with image formats.

 

I don't actually believe this has anything to do with privacy changes.  Many sites transcode images to save on bandwidth.  This situation may have been going on since last fall; I have been noticing the behavior and assuming it was something with Firefox.  Yesterday I looked a new puzzle and tried every possible way to get it to give me a png file, which it would not.  So I am not asserting that this is brand-new; only that it is a very serious problem.

  • Helpful 2
Link to comment
31 minutes ago, fizzymagic said:

 

I don't actually believe this has anything to do with privacy changes.  Many sites transcode images to save on bandwidth.  This situation may have been going on since last fall; I have been noticing the behavior and assuming it was something with Firefox.  Yesterday I looked a new puzzle and tried every possible way to get it to give me a png file, which it would not.  So I am not asserting that this is brand-new; only that it is a very serious problem.

 

In this age of rampant video streaming, much of it now moving to 4G, why are they quibbling over a few hundred kilobytes for a PNG file? The more I see now, the more I'm convinced I'm on the wrong planet.

  • Funny 1
  • Helpful 1
Link to comment
13 hours ago, barefootjeff said:

In this age of rampant video streaming, much of it now moving to 4G, why are they quibbling over a few hundred kilobytes for a PNG file? The more I see now, the more I'm convinced I'm on the wrong planet.

 

Here's my guess:

  1. They implement privacy features that interfere (intentionally or unintentionally) with externally-hosted images
  2. Due to 1., they recommend that members upload images directly to the cache listing.
  3. Due to 2., more images are having to be transferred from HQ's image storage rather than from external hosts, so bandwidth use is increasing.
  4. Due to 3., costs are increasing.
  5. Due to 4., transcoding is implemented to reduce bandwidth use and costs.

Of course, then there's the next step:

      6. Due to 5., puzzles that rely on the exact format and content of images are broken.

  • Upvote 5
Link to comment
On 3/1/2020 at 11:49 PM, thebruce0 said:

Aye, if this were something, say a change in policy about allowable images, that was being newly enforced from this date moving forward, then a public announcement would suffice (the only anger likely being that new puzzles couldn't employ prior technical strategies), and existing listings would be 'grandfathered' (or given time to change/archive).

But this change is breaking old and existing puzzle caches, with many cache owners being none the wiser.  There is some very bad development/rollout practices being employed here for the sake of silently (relatively speaking) adhering to privacy laws. Things are breaking. People aren't being told, warned, informed. That ain't a good thing.  :(

 

Someone just reached out for help on one of my caches, it involves using a background image hosted on a photo hosting site...

Originally in the source code the background image appeared as such:

<body background="i.imgur.com/0u38bBj.jpg" class="cache details page">

 

But now it's this monstrosity: 

<body background="https://imgproxy.geocaching.com/9660e2daac2e9e8461ceb68de1b8f1c378eb7ed1?url=https%3A%2F%2Fi.imgur.com%2F0u38bBj.jpg" class="CacheDetailsPage">

When did this change? Anyone find a work around? 

Edited by STNolan
  • Helpful 1
Link to comment
On 4/3/2020 at 10:14 PM, fizzymagic said:

I went to look at a puzzle that used a png today, and it came through without transcoding!   So whatever it was appears to be fixed.   Thanks, HQ.

 

No, it is not.  I tried to upload a png for creation of a puzzle cache yesterday and it cam back as a jpeg.

 

Could somebody from HQ pleasepleaseplease comment on this issue?   Hundreds of puzzles have been ruined.

  • Funny 1
  • Helpful 1
Link to comment

In a perhaps related issue, images have disappeared from several of my cache description pages:  this includes the multiple images tiles used as a background image and pictures embedded cache description.  Doesn't seem to be any pattern:  I open a cache page and they are either there or not and if not there are no longer in the "edit" section.  Weird.

edexter

Link to comment
1 hour ago, edexter said:

In a perhaps related issue, images have disappeared from several of my cache description pages:  this includes the multiple images tiles used as a background image and pictures embedded cache description.  Doesn't seem to be any pattern:  I open a cache page and they are either there or not and if not there are no longer in the "edit" section.  Weird.

edexter

 

Four of my mystery caches had the images included in the cache page description, and loaded as images on the cache page.  The loaded photos url had changed.  Not easy for this dolphin to figure out how to copy the url from the photo to copy into the edit of the cache page.  But I got them to work.  My other mystery caches had the same changes in the url, but the cache page included the new url into the page, and did not need anything to be done.  Very strange and inconsistent.

Link to comment

I'm wondering what might be required to have your personal domain added to the whitelist for hosting your own images. Agreement? Verifiable in/out of any public data to ensure you're not doing anything 'malicious'?  Maybe HQ could setup a process whereby users who know what they're doing and deemed trustworthy can host their own images on an approved domain.  How do flag counter websites become approved? SImply because of mass use? Or are they in communication to provide verification that the images are as provided with no under-the-hood actions going on?

It's not about file type, since other hosts can provide PNGs and unedited JPGs. It's not about dynamic content since flag sites and like are being allowed. So what might one need to agree to to have a personal domain approved for image hosting?

*wishful thinking*

  • Upvote 2
Link to comment
4 hours ago, thebruce0 said:

I'm wondering what might be required to have your personal domain added to the whitelist for hosting your own images...

*wishful thinking*

 

It probably is wishful thinking - but I would "apply" to use my own personal domain for image hosting.  All my images (until recently) are hosted on my own domain server.  I have a folder for caches (and hubby's caches as well) so all our image files are easily located; I enjoy having control over the server where they are hosted.  I'd probably be willing to sign or agree to verify whatever needed verifying, or go through whatever "process" is needed to continue hosting my own images.

 

All of our images seem to be intact (cache page backgrounds, puzzle images, etc) so far, but recently created caches have had to use GC's upload options.  It would be nice to get it all back in one place, on my domain!!

  • Funny 1
Link to comment

Add me to the wishful thinking queue.  I thought I'd read that the new hosting laws would not be backdated?  But they have, in a seemingly random manner.  Like others on here, I've been hosting images and other files on my own website since 2010.  I had fears about the Jan2020 release, but on checking quite a few of my puzzles, whatever I'd done to the images had stayed intact.  The recent removal of the image proxies, without notice. (I get global emails telling me what souvenirs I can collect, so why not tell everyone the image hosting rules are changing, again.)  I now have 200+ puzzles to redo.  On my caches, the line seems to stop on 22May2014.  I had 2 caches published that day, one of them still has the image proxy, and on the other it has been removed.  No sense to it at all.

Link to comment
2 hours ago, fizzymagic said:

Could somebody from HQ please say something?  Why are png images being returned as JPEGS?  Can somebody at least acknowledge that the problem exists?

 

Is this transcoding issue happening one day and not happening the next?  Is it retroactive for existing images?  Previous posts seem to suggest it seems kind of random.

Is there a particular size or kind of image that tends to be targeted?

 

I tried two uploads just now as a test on one of my caches, one image that is too small to require server-side re-sizing, 148x148 pixels, and one that seemed like a candidate for re-sizing, 1200x1207 (on upload, it was automagically shrunk to 636x640).  Both remain 24-bit PNG files.  I was kind of surprised at how small the 2nd one was re-sized.  I didn't test EXIF.

 

Will one or both be transcoded when I'm not looking?  

 

 

Edited by kunarion
An atom bomb went off, so I needed to hide in a refrigerator.
Link to comment
2 hours ago, kunarion said:

 

Is this transcoding issue happening one day and not happening the next?  Is it retroactive for existing images?  Previous posts seem to suggest it seems kind of random.

Is there a particular size or kind of image that tends to be targeted?

 

I tried two uploads just now as a test on one of my caches, one image that is too small to require server-side re-sizing, 148x148 pixels, and one that seemed like a candidate for re-sizing, 1200x1207 (on upload, it was automagically shrunk to 636x640).  Both remain 24-bit PNG files.  I was kind of surprised at how small the 2nd one was re-sized.  I didn't test EXIF.

 

Will one or both be transcoded when I'm not looking?  

 

 

 

You might think those two images are still PNG files, but they're not. Even though they still have the .png extension, internally they've been converted to jpg. Here are hex dumps of the start of each file:

 

image.png.40b36eee2801f8921d6435ab234673ce.png

 

image.png.2c4ef5ff018f1ae39b968001b3277fb7.png

 

The first two bytes (ff d8) is the JPEG signature, as is the string JFIF starting at offset 6.

Link to comment
1 hour ago, barefootjeff said:

 

You might think those two images are still PNG files, but they're not. Even though they still have the .png extension, internally they've been converted to jpg.

 

Ok, got it!  I probably should have paid more attention to the fact that the file size had become especially small.

Even Windows identifies them as 24-bit PNG.  I didn't do any further testing.

 

This reminds me of a thread a while back, where there were genuine png icon files, but other jpg icons had a .png extension, which a certain web browser couldn't handle.  So some were not loading.  Seems like that was similarly crazy, because sometimes you can't convince a web designer to use files that match their file extension.  You'd think it would be a normal thing to do, but, I guess not.

Edited by kunarion
Link to comment
32 minutes ago, lee737 said:

600x600 images? And we're back in 1997.....

 

That was weird.  My test today showed that 1200x1207 shrinks to about half its dimensions.  But lately I upload photos at 800x600, and they don't get re-sized by the server.  It used to be that up to about 1000x750 was safe. 

Link to comment

These days on the web you really can't pay attention to file extensions.  The server can send any kind of binary data, so it's a matter of the browser receiving it and interpreting the data properly. In the case of PNG files, they are being transcoded to JPG files, even though the URL of the image implies it's a ".PNG" file.  That's an inconsistency and can cause problems for anything that does assume a data type based on url (namely, functions that don't request the URL but have to deal with whatever is being requested - eg, if a url implies an image, the handling of that request may be different even though it might return text - file extension is still important, but can't assume to know exactly what will be received on request).

 

HQ, in your proxy image hosting: please either 1) don't change the data type and keep the file extension the same, or 2) change the file extension to match the new data type.  But preferably #1.

  • Helpful 1
Link to comment
3 hours ago, PatientRock said:

Came to the forum wondering why the hosted images in all my puzzle caches were no longer working... Had to replace them all. That, and my EXIF puzzle is trashed. Good times.

there is a workaround if you use external services like google drive, but it is not as easy to set this up.

 

On 11/5/2020 at 11:18 PM, fizzymagic said:

Still waiting for word From Above.

 

I've been wracking my brain trying to figure out why HQ is unwilling to even comment on this topic.  As a computer security researcher, I would have heard about any png-based exploit, so I am pretty sure it's not that.

 

It's just so...  odd.

 

You know, avoiding the topic and pretending it doesn't exist is the only way for them if they want to keep this that way. Nobody of course knows why, and thats not the first issue of this kind.

Link to comment

I did a new puzzle today that used a png image.  The CO was unaware that his png image had been transcoded to a jpg, making the puzzle much more difficult.  Don't you think common courtesy would dictate that Groundspeak let COs know that the images they upload (and are accepted by the upload page) are being altered into becoming completely different image types?

 

I am not sure how to put this delicately enough, so if I offend some rule please let me know, but the ongoing refusal to address this issue is starting to feel like an obscene gesture aimed in the direction of geocaching customers.   Please note, I am not saying that it is such a gesture; it just feels that way.

 

It is, in any event, rude.

 

ETA:  When I told the CO about the transcoding problem, he said, "but I uploaded it as a png!"  Making it very clear that reviewers are not checking with puzzle creators about whether the transcoding breaks their puzzles or not.  Could a reviewer comment on whether they have been informed of this change?

Edited by fizzymagic
add information
  • Upvote 7
  • Surprised 1
  • Helpful 2
Link to comment
10 hours ago, fizzymagic said:

Maybe I should start a new thread with a title like "BUG: png  images are sent as jpgs" or something?

 

I was thinking the same. It really IS a bug.  Non-JPG images are being converted and served as JPG images yet with the original file extension. That's a bug.  Even if it's intentional. :P

  • Upvote 1
Link to comment
2 hours ago, thebruce0 said:

 

I was thinking the same. It really IS a bug.  Non-JPG images are being converted and served as JPG images yet with the original file extension. That's a bug.  Even if it's intentional. :P

 

I think you are on to something here.  Maybe if it's intentional they can at least explain why they don't think it's a bug.

 

I am pursuing this fruitless quest because I have discovered that cache hiders don't know about the transcoding and reviewers don't tell them!

 

So I will try the bug route soon.

  • Funny 1
Link to comment
17 minutes ago, fizzymagic said:

I am pursuing this fruitless quest because I have discovered that cache hiders don't know about the transcoding and reviewers don't tell them!

 

Over ten years, I have seen these images transcoded and used by cache owners without any technical explanation or warnings from reviewers. For me, it took for a while to realize that images also have the original uploaded version instead of the processed jpg version, which was primarily offered for use. From this historical view, the current situation is an understandable continuum to the established practice.

  • Upvote 1
Link to comment
2 minutes ago, arisoft said:

For me, it took for a while to realize that images also have the original uploaded version instead of the processed jpg version, which was primarily offered for use. From this historical view, the current situation is an understandable continuum to the established practice.

 

Except that original uploaded version is no longer available. The original-sized version (without the _l suffix on the file name) is also transcoded to jpg while retaining the .png extension.

  • Helpful 1
Link to comment
17 minutes ago, arisoft said:

Over ten years, I have seen these images transcoded and used by cache owners without any technical explanation or warnings from reviewers. For me, it took for a while to realize that images also have the original uploaded version instead of the processed jpg version, which was primarily offered for use. From this historical view, the current situation is an understandable continuum to the established practice.


You are incorrect.   Years ago, images would be resized, and that was made clear on the image upload page.  But images have never been transcoded to a completely different format until now.

 

When an image is changed from png to jpg, many things change:

  • The pixel data is changed in a lossy way (most people know about this, but it cannot be reversed)
  • The colors are changed (there are many rgb values that cannot be displayed in a jpg image)
  • Transparency is not available in jpg images, but it is in png images
  • Images with large areas of the same color are less efficiently compressed by jpg than by png
  • Paletted images are not available in jpg
  • etc.

The problem is no minor continuation of previous policies.  It has broken hundreds of puzzle caches with no notification of the owners. Especially in the wake of the Photobucket debacle and various privacy laws, Groundspeak has been trying to encourage the storage of images to their cloud. The situation is not, to my mind, acceptable without some clear explanation of what is being done and why.

 

Edited by fizzymagic
  • Upvote 2
  • Surprised 1
  • Helpful 2
Link to comment

And just to add to this, sure there are quite often feathers ruffled and the same old forum folk raise a stink, often not given additional importance just by merit of the amount of discussion and criticism. But this is actually a problem, not just an opinion.

 

This is a technical inconsistency that should not exist (some might even say without good reason). The filename extension for non-JPGs no longer matches the resulting (force-transcoded) image content.  THAT is the technical problem.

 

As to whether images are to be force-transcoded to JPG when uploaded, that's a matter of policy. ie, if you choose to upload a non-jpg, then your images will become transcoded JPGs. If that's a policy that won't change, then that's HQ's decision. The alternative is, images can now be hosted on an approved 3rd part host and retain their original formats.

 

BUT - we go back to the technical issue that  the implied final image is not necessarily what the cache owner intended and is technically invalid, as well as highly misleading for people who do not know what's going on (especially those who are not technically proficient).

 

 

 

So in my mind, if HQ decides that all images uploaded and hosted natively MUST become JPGs, that's fine. I'll host non-jpgs on an approved 3rd party host.

 

BUT, either:

1. change the uploaded image URL to the correct .jpg extension, and ideally also inform the user that images uploaded to the listing will be converted to .jpg, or

2. disallow non-jpg image uploads straight-up, suggesting to use an approved 3rd party host (somehow inform the user while editing or uploading; just like most places disallow excessive file sizes, dimensions, or file types on uploading), or

3. simply stop transcoding approved non-jpg uploaded image types to jpg (gif, png, maybe even svg), even while the native url is still inserted (with the intended image type extension retained)

Edited by thebruce0
got my upload/proxy stuff mixed up --twice :P
  • Upvote 1
  • Funny 1
Link to comment
31 minutes ago, thebruce0 said:

So in my mind, if HQ decides that all images piped through their proxy MUST become JPGs, that's fine. I'll host non-jpgs on an approved 3rd party host.

 

The problem isn't with the proxy, in fact images on Mystery caches no longer go through that proxy server (that appears to have quietly changed a couple of months ago). The problem is with images hosted directly on the geocaching site, with the transcoding happening during the uploading process.

Link to comment

Yes I wasn't referring to mystery caches where the rules are slightly different.

 

However, on just testing now with one of my Traditional caches referencing a PNG image on my own site, it seems the proxy is employed but the PNG extension and proper image format are retained. So I got my mixed up proxy and upload issues mixed up :P I'll re-edit my prior comment back to referencing the upload issue (on both description and background uploads). heh

 

eta: just goes to show how confusing this image hosting thing can be to someone not in the know at all = P

Edited by thebruce0
Link to comment

And because I'm meticulous, I just ran through a test of the current proxy functionality just to confirm in detail... :laughing:

 

For new listings, referencing an image via IMG SRC (not the issue for this thread):
Traditional cache, unapproved 3rd party hosting PNG => Proxy, PNG
Traditional cache, unapproved 3rd party hosting JPG => Proxy, JPG
Mystery cache, unapproved 3rd party hosting PNG => CAN'T SAVE
Mystery cache, unapproved 3rd party hosting JPG => CAN'T SAVE
Traditional cache, approved 3rd party hosting PNG => SAVED
Traditional cache, approved 3rd party hosting JPG => SAVED
Mystery cache, approved 3rd party hosting PNG => SAVED
Mystery cache, approved 3rd party hosting PNG => SAVED

 

And for completion's sake on this post, as of this writing, any image uploaded is transcoded (and converted to JPG) for the native GC image hosting while keeping the original image file extension (which may not be JPG), even though the image data served is always JPG.

 

Any cache type, image uploaded PNG --transcoded--> JPG served with PNG extension
Any cache type, image uploaded JPG --transcoded--> JPG served with JPG extension

Edited by thebruce0
  • Helpful 1
Link to comment

In hopes to answer some of the questions: When uploading an image to Geocaching.com, it is converted to progressive jpeg. This means that the image will be progressively downloaded, improving website speed. What this looks like on a slow connection is that the image will become clearer and clearer with the progressive download (in comparison, non-progressive loading would show the image slowly appearing line by line from top to bottom).

  • Helpful 2
Link to comment
1 hour ago, nykkole said:

In hopes to answer some of the questions

 

Well, that can be helpful if the original image is a JPG and already progressive, or if being non-progressive is irrelevant to the CO :P

But the issue isn't with JPGs, it's with non-JPGs being converted to JPGs, both without the owner's knowledge and being technically incorrect in that the final URL is serving a different binary content type than implied by its extension (per my prior post).

Edited by thebruce0
grammar
  • Upvote 1
  • Helpful 1
Link to comment
52 minutes ago, nykkole said:

In hopes to answer some of the questions: When uploading an image to Geocaching.com, it is converted to progressive jpeg. This means that the image will be progressively downloaded, improving website speed. What this looks like on a slow connection is that the image will become clearer and clearer with the progressive download (in comparison, non-progressive loading would show the image slowly appearing line by line from top to bottom).

 

Thanks for the explanation. So in a nutshell, then, a whole class of image-based puzzles are being silently broken in order to make it look pretty for people using dial-up connections.

  • Upvote 1
  • Funny 1
  • Helpful 1
  • Love 1
Link to comment
10 hours ago, nykkole said:

In hopes to answer some of the questions: When uploading an image to Geocaching.com, it is converted to progressive jpeg. This means that the image will be progressively downloaded, improving website speed. What this looks like on a slow connection is that the image will become clearer and clearer with the progressive download (in comparison, non-progressive loading would show the image slowly appearing line by line from top to bottom).

That is a false dichotomy. Using a placeholder, which can be a low quality thumbnail, is a very common tactic that has no need to be paired with transcoding user submitted images to progressive jpeg. This can be coupled with lazy loading (download the full image only when the user tries to view it) which is natively supported by all major browsers except Safari, and there are readymade solutions for Safari too.

  • Surprised 1
Link to comment
On 11/20/2020 at 1:42 PM, nykkole said:

In hopes to answer some of the questions: When uploading an image to Geocaching.com, it is converted to progressive jpeg. This means that the image will be progressively downloaded, improving website speed. What this looks like on a slow connection is that the image will become clearer and clearer with the progressive download (in comparison, non-progressive loading would show the image slowly appearing line by line from top to bottom).

 

Thank you very much for responding.  At least we now have a basis upon which we can begin the discussion!

  • Upvote 1
  • Funny 1
Link to comment
On 11/20/2020 at 1:42 PM, nykkole said:

In hopes to answer some of the questions: When uploading an image to Geocaching.com, it is converted to progressive jpeg. This means that the image will be progressively downloaded, improving website speed. What this looks like on a slow connection is that the image will become clearer and clearer with the progressive download (in comparison, non-progressive loading would show the image slowly appearing line by line from top to bottom).

 

If all images are being converted to progressive jpegs for the reasons above, then it would make far better sense to use the WebP image format.  I mean, if you are willing to break thousands of puzzles, then why not do it right?   WebP renders progressively, supports lossless compression and transparency, and compresses better than jpeg and png.  It allows EXIF and works with (at least) Chrome, Firefox, Internet Explorer, Microsoft Edge, Opera, and Safari browsers.

 

So, actually, it would not break any puzzles, as the lossless mode is basically identical to png (with exif for free).  So if the reason for the change is either having progressive rendering or superior compression, progressive jpeg is not a wise choice.

  • Upvote 2
  • Helpful 2
  • Love 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...