Problem with huge Mars jp2 image

Ariel DK · February 17, 2020, 11:40:58 AM

After my vacations on Chile, i'm back to work on new projects again

But i fastly find problems. trying to create a Mars planet that survive to close-up shots, i find this great 92k resolution color dataset from here:

https://astrogeology.usgs.gov/search/map/Mars/Viking/MDIM21/Mars_Viking_MDIM21_ClrMosaic_global_232m

I didn't want the 12GB Tiff, so i tried with the 690Mb Jp2 image in the ancillary section.
TG can't even load it (ReadImage unable to load), in a Image Map shader.
so, i tried with Geo Image Map and, surprise! it load it, but the image don't contain georeferentation, and is not showing anything.
I'm not sure if i can manually make the referentation of the image, i tried with the supposed latitude: -90,90 and -180,180 used to define a standard rectangle, didn't work. Another fact, is that when i changed the load method from GDAL to Terragen, the ReadImage error appear again.
I only have 12 GB RAM, but i dont had any crash or load issue, indeed, it took just 1 minute to load the image, which is normal i guess.
So, anyone already had tried with this kind of files?

WAS · February 17, 2020, 11:51:22 AM

I can't seem to find the Jpeg version link. i just woke up though, literally brewing coffee now.

Ariel DK · February 17, 2020, 12:03:54 PM

black or decaffeinated coffee? I like me with cream if you ask.

cyphyr · February 17, 2020, 12:14:32 PM

A simple solution may be to try to open the 12gig image in photoshop a (or similar) and cut it in half, save it as east or west half and load it as a spherical projection but set the size to 0.5, 1.
There is not much point in loading data that you can not see.
You can of course cut the 12G image into smaller sections if you are close enough to get away with it.

WAS · February 17, 2020, 12:15:28 PM

Little bit of whole milk.

I'll see if i can get it to work when it's finished. Almost done. I see the openjp2.dll, so it should have support for this.

JP2 is a bitmap image format, lossy or lossless in any bit depth or metadata. It's not a dataset, at least shouldn't be used like that. It was intended for medical field as it can incorporate data on the image (metadata) which could be anything, such as information on a MRI. Metadata apparently can be proprietary for explicit uses to a program from what I'm seeing though too.

WAS · February 17, 2020, 12:27:53 PM

I am not sure TG can handle this specific file. When I do get it wrapped, there is no planet. It applies only black color data. Completely blended with BG.

Ariel DK · February 17, 2020, 12:40:13 PM

Quote from: cyphyr on February 17, 2020, 12:14:32 PMA simple solution may be to try to open the 12gig image in photoshop a (or similar) and cut it in half, save it as east or west half and load it as a spherical projection but set the size to 0.5, 1.
There is not much point in loading data that you can not see.
You can of course cut the 12G image into smaller sections if you are close enough to get away with it.

Well, there is a reason why i downloaded the 690Mb image instead the 12Gb Tiff, and is cuz i don't think that i would can open that tiff in PS, or any other program due my Ram limitation... but i can make the suicide try with that 12gig tiff tomorrow.

WAS · February 17, 2020, 12:46:10 PM

You could give Photoshop a custom scratch drive on disk space, set it to 64gb or so (head room). It should load the image into scratch mainly. This will be excruciatingly slow if it works. Photoshop Scratch is literally how we handled huge textures back when we had 512mb - 1024gb memory.

You could also give yourself more virtual memory to handle RAM overflow on HDD, again, slow.

Ariel DK · February 17, 2020, 01:21:51 PM

Quote from: WAS on February 17, 2020, 12:27:53 PMI am not sure TG can handle this specific file. When I do get it wrapped, there is no planet. It applies only black color data. Completely blended with BG.

Yes, thats true jordan. just as a one last try, i converted the jp2 into a Bmp file, and i get just the south pole, but nothing else (see attach).

Quote from: WAS on February 17, 2020, 12:46:10 PMYou could give Photoshop a custom scratch drive on disk space, set it to 64gb or so (head room). It should load the image into scratch mainly. This will be excruciatingly slow if it works.
(...)

I never did this before. there is some tutorial or instructions over there?

WAS · February 17, 2020, 01:24:43 PM

Well that's progress at least. I think now it's just a matter of dimension and wrapping to the planet.

Also have a look here: https://helpx.adobe.com/photoshop/using/scratch-disks-preferences.html

Ariel DK · February 17, 2020, 01:40:56 PM

Quote from: WAS on February 17, 2020, 01:24:43 PMWell that's progress at least. I think now it's just a matter of dimension and wrapping to the planet.

Also have a look here: https://helpx.adobe.com/photoshop/using/scratch-disks-preferences.html

I have another old hard disk compatible with my CPU, which is not probably the best option, but with hope it would be enough for this occasion.
Let me see what i can do...

WAS · February 17, 2020, 02:59:19 PM

Quote from: Ariel DK on February 17, 2020, 01:40:56 PM
Quote from: WAS on February 17, 2020, 01:24:43 PMWell that's progress at least. I think now it's just a matter of dimension and wrapping to the planet.

Also have a look here: https://helpx.adobe.com/photoshop/using/scratch-disks-preferences.html

I have another old hard disk compatible with my CPU, which is not probably the best option, but with hope it would be enough for this occasion.
Let me see what i can do...

USB flash drive should even work, and if USB 3.0 (and nothing else taking up much bandwidth) may be faster than a mechanical HDD (not SSD)

Before I had a dedicated drive to scratch/virtual memory I used a 64gb flash drive from a dedicated hub on front of PC.

Now... I have one HDD which streams most my content, cause my old PC is still fine so didn't pull HDDs. My SSD has the OS on it. So I don't use extended virtual memory at all or scratch, so I get lovely crashes all the time with 8gb memory (again... how many times have I upgraded to go back with upgrading a PC to new RAM type). Lol

My other PC is only 16gb RAM but can handle files upwards 25gb (in TG and stuff, not just PS) due to 128gb of dedicated HDD space to virtual memory.

Ariel DK · February 18, 2020, 08:48:24 AM

I downloaded the 12Gb tiff last night, and definitively doesn't load. I gonna see if i can get a flash drive today, before try with my hdd...

Ariel DK · February 18, 2020, 06:27:46 PM

Damn $:-\$ i couldn't get that flash drive today, and the drivers for my hdd is corrupted, so i will need to find and download those drivers again
Meanwhile, i gonna open a new thread about this Mars scene...

WAS · February 18, 2020, 06:56:26 PM

Drivers for a HDD? What sort of Wizardry is it? If you have the free space you can use your main HDD, but that will slow things down even further.

FYI; using Windows Virtual Memory may be more stable than Adobe's Scratch. I have seen issues where Adobe creates scratch files (temp files) that windows suddenly can't remove without a laborious task, and I've seen posts about corrupted temp files and trouble removing them, or crashes. I haven't experienced any issues with Windows clearing or managing it's own virtual memory, though there are sure no lack of posts. So I don't know.