Neural Network Texture up-scaling

Discuss the game Trespasser here!

Moderators: TresCom Support Team, TresCom Board Managers

User avatar
Isolt
Stegosaurus
Stegosaurus
Posts: 62
Joined: Wed Jun 06, 2018 8:02 pm

Neural Network Texture up-scaling

Post by Isolt »

I've been playing around with a few image up-scalers that use machine learning to guess what the enlarged image would look like rather than just average out the pixels.
I though it'd be fun to run the trespasser textures through it.
Doubling them seemed to be a good compromise between size and benefit. Unfortunately the original images are only 8 bit colour so there are colour banding issues. They might benefit from some blurring?

Current Version
V2.2 https://www.moddb.com/addons/start/2057 ... xture-pack
https://www.trescom.org/download/nnhd-t ... -version/#
V2.2 light https://www.moddb.com/addons/start/2057 ... xture-pack
https://www.trescom.org/download-catego ... llaneous/#
Pine Valley V1.0 https://www.moddb.com/addons/start/1868 ... xture-pack
https://www.trescom.org/download-catego ... llaneous/#
Attachments
PPrecisionSMLcrate_t2.jpg
PPrecisionSMLcrate_t2.jpg (47.96 KiB) Viewed 17431 times
Ppaintcan04t2.jpg
Ppaintcan04t2.jpg (55.23 KiB) Viewed 17431 times
Pfuelbarrel03t22.jpg
Pfuelbarrel03t22.jpg (41.33 KiB) Viewed 17431 times
PBrCanNoTopHi_lites00t2.jpg
PBrCanNoTopHi_lites00t2.jpg (58.25 KiB) Viewed 17431 times
Last edited by Isolt on Fri Feb 05, 2021 9:32 pm, edited 7 times in total.
User avatar
machf
T-Rex Killer
T-Rex Killer
Posts: 12368
Joined: Thu Apr 24, 2003 11:20 pm
Location: Lima, Peru
Contact:

Re: Neural Network Texture up-scaling

Post by machf »

Nice results...
Visit The Carnivores Saga - a forum devoted to modding Action Forms' Carnivores, Carnivores 2 and Carnivores: Ice Age games
Tres WIP: updated T-Script Reference and File Formats documents
Sound name listings for the Demo (build 117), Retail (build 116), Beta 103, Beta 99, Beta 97, Beta 96, Build 55, PC Gamer Alpha (build 32) and E3 1998 Alpha (build 22) TPA files
User avatar
Isolt
Stegosaurus
Stegosaurus
Posts: 62
Joined: Wed Jun 06, 2018 8:02 pm

Re: Neural Network Texture up-scaling

Post by Isolt »

Thanks.
They'll never be as clear as TheIdiot's remade ones, but they're better than any standard up-scaling.
It's not hugely noticeable in game but things are a bit more crisp.
Now to sort the texture overlaps from every level... There must be an easier way.

Image
Image
Image
Image
Spoiler: show
TPass005.jpg
TPass005.jpg (609.23 KiB) Viewed 17413 times
Spoiler: show
TPass004.jpg
TPass004.jpg (485.62 KiB) Viewed 17413 times
Spoiler: show
TPass002.jpg
TPass002.jpg (420.75 KiB) Viewed 17413 times
Spoiler: show
TPass001.jpg
TPass001.jpg (512.77 KiB) Viewed 17413 times
User avatar
machf
T-Rex Killer
T-Rex Killer
Posts: 12368
Joined: Thu Apr 24, 2003 11:20 pm
Location: Lima, Peru
Contact:

Re: Neural Network Texture up-scaling

Post by machf »

How many textures did you apply this process to?
And yes, the textures having been reduced to 8-bit palettes in the final step of the export for the game makes them somewhat problematic to improve... does the process get any advantage if you convert them to 24-bit RGB images first? At leas, normal resampling algorithms usually work better if you do that and then convert back to an 8-bit palette... and with CE, you don't really need to make them 8-bit again.
Visit The Carnivores Saga - a forum devoted to modding Action Forms' Carnivores, Carnivores 2 and Carnivores: Ice Age games
Tres WIP: updated T-Script Reference and File Formats documents
Sound name listings for the Demo (build 117), Retail (build 116), Beta 103, Beta 99, Beta 97, Beta 96, Build 55, PC Gamer Alpha (build 32) and E3 1998 Alpha (build 22) TPA files
User avatar
TheIdiot
T-Rex
T-Rex
Posts: 4343
Joined: Thu Feb 20, 2014 9:13 pm
Location: Canada, eh?

Re: Neural Network Texture up-scaling

Post by TheIdiot »

I was waiting for someone to do this. Had wanted to do it myself a while back but couldn't find a (free) tool to do so nicely, and retexturing everything from scratch takes a lot of time, especially considering a lot of Tres' objects have a large number of split-up textures, such as dinosaurs and vehicles.

The colour banding in the original images does suck, but there isn't a whole lot you can do about that without remaking the textures from scratch. You could try blurring the image a bit before upscaling but you'd lose some of the details. With CE you can use hardware textures in .dds format which should allow you to avoid having to convert the images over to 256 colors - I found saving them as .bmps causes some issues with lighting.

What I'd like to see now is a conversion of all of the original Trespasser bumpmaps to CE-compatible normalmaps so that they work in fan-made levels, since GeomAdd never got the ability to use standard Tres bumpmaps in custom levels.
User avatar
machf
T-Rex Killer
T-Rex Killer
Posts: 12368
Joined: Thu Apr 24, 2003 11:20 pm
Location: Lima, Peru
Contact:

Re: Neural Network Texture up-scaling

Post by machf »

TheIdiot wrote: What I'd like to see now is a conversion of all of the original Trespasser bumpmaps to CE-compatible normalmaps so that they work in fan-made levels, since GeomAdd never got the ability to use standard Tres bumpmaps in custom levels.
Technically, it shouldn't be too hard. Especially since, technically (yes, again), Trespasser isn't using bumpmaps but rather normal maps, only that instead of using cartesian coordinates to store the vector normals (X, Y, Z), it's using spherical coordinates (radius, azimuth and elevation).

Since by definition a normal vector always has a length of 1.0 (it's called a normal because it has been normalized), its radius in spherical coordinates will always be 0, so it doesn't need to be stored; the azimuth (horizontal angle) will go from 0 to 360 degrees (think of compass directions, with 0 being north, 90 being east, 180 being south, 270 being west and 360 back to north again), and while the elevation should go from -90 (pointing down) to +90 degrees (pointing up), the normal vector of a surface you're looking at will never have a negative angle, so it only has to go from 0 to 90 degrees. That's the reason why, from the 10 bits used for storing the normal maps (the other 6 bits being the palette index for a 64-color image), 6 bits are used for the azimuth and only 4 bits for the elevation. 6 bits mean 64 possible values, while 4 bits mean 16 possible ones, which is 4 times less, just kile 90 degrees is 1/4th of 360 degrees. The resolution is the same in both cases, 360/64 = 90/16 = 45/8 = 5.625 degrees.

Converting Trespasser's normal maps into bump maps has been more difficult than it would be converting them to regular normal maps, which only implies converting the spherical coordinates into cartesian ones and can be done by simple multiplication. Converting to bumpmaps, OTOH, has required numerical integration, and the results aren't accurate.

SWPext (don't remember if TresEd does too) has an option to extract the bumpmapped textures as raw data; then all we need to do is have another program process those files to write the corresponding standard normal maps as BMP or PNG images.
Visit The Carnivores Saga - a forum devoted to modding Action Forms' Carnivores, Carnivores 2 and Carnivores: Ice Age games
Tres WIP: updated T-Script Reference and File Formats documents
Sound name listings for the Demo (build 117), Retail (build 116), Beta 103, Beta 99, Beta 97, Beta 96, Build 55, PC Gamer Alpha (build 32) and E3 1998 Alpha (build 22) TPA files
User avatar
tatu
-=TresCom Website Manager=-
-=TresCom Website Manager=-
Posts: 5087
Joined: Fri Jun 24, 2005 9:40 pm
Antispam: No
Location: Sweden
Contact:

Re: Neural Network Texture up-scaling

Post by tatu »

A member on the Discord have done this to several LAB textures and upscaling some quite good.

IIRC the SWPext have issues and export the bumpmaps with some lines in them? However, you can use the GUIApp from CE and export textures along with the bumpmaps. :)
Active project: Trespasser: Isla Sorna
Status:
BE-PH1: Released
PH2-IT: Pre-released
PL-SUM: In production

"...there used to be more benches, but InGen's workers removed them during the evacuation in the name of framerate."
User avatar
Draconisaurus
T-Rex Killer
T-Rex Killer
Posts: 14032
Joined: Mon Dec 06, 2004 5:21 pm
Antispam: No
Location: Isla Sorna
Contact:

Re: Neural Network Texture up-scaling

Post by Draconisaurus »

Hey now, those upscaled images are great! I was unaware of this technique, am impressed...

Also.. spherical coordinate systems. Nice, I wonder if there are alien races who use this as their standard model rather than X,Y,Z box-coordinates..
User avatar
Isolt
Stegosaurus
Stegosaurus
Posts: 62
Joined: Wed Jun 06, 2018 8:02 pm

Re: Neural Network Texture up-scaling

Post by Isolt »

Hi machf.
I've done all of beach textures so that I could test it.
Last night I combined the textures from all the levels to avoid double ups. There was a note in the GUI app about having different textures with the same name in different levels. I went through them as I combined them but didn't notice any. Does anyone know of any exceptions? There are thousands of files and I may have overlooked some.
For my beach test I didn't upscale the normal maps, or even use the ones extracted along with the textures. Would those benefit from up-scaling?
I know ground textures aren't applied in CE so I won't do these. How do I identify them? "TGrass05t2" looks like a good candidate, what about things like "TBchClff30t2" and "TgiantBoulder01t2"
Would imposter sprites would benefit from up scaling, or are the new draw distances so good it's not necessary?

The Idiot, I want to avoid modifying the textures before up-scaling them. The less they've been modified the better the results are I believe. Hmm lighting issues with .BMP is concerning as that's what I was planning to use. TGA is an option, or I could batch convert them to .DDS with something?

My current plan is exporting all textures with GUIApp from CE. Combining them to remove doubles. Removing ground textures and normal maps. Upscale 2x with 24bit BMP's as the output. Then possibly add a bit of selective blurring on some of the worst banding.

Thanks tatu, you helped me get the GUIApp running over in the CE thread so that I could export these.
User avatar
machf
T-Rex Killer
T-Rex Killer
Posts: 12368
Joined: Thu Apr 24, 2003 11:20 pm
Location: Lima, Peru
Contact:

Re: Neural Network Texture up-scaling

Post by machf »

tatu wrote: IIRC the SWPext have issues and export the bumpmaps with some lines in them? However, you can use the GUIApp from CE and export textures along with the bumpmaps. :)
The idea is not to export them as bumpmaps (that's why in the process they get those lines), but the raw data, and then convert THAT into a regular normal map...
Isolt wrote:Hi machf.
I've done all of beach textures so that I could test it.
Last night I combined the textures from all the levels to avoid double ups. There was a note in the GUI app about having different textures with the same name in different levels. I went through them as I combined them but didn't notice any. Does anyone know of any exceptions? There are thousands of files and I may have overlooked some.
Nice amount of work that must have been...
The thing is, different levels use different palettes, for optimization reasons. So, what was originally the same 24-bit RGB texture in the source 3DS MAX files, may have been converted to 8-bit using a certain palette when exporting it for level A, and a different palette when exporting it for level B. And since the downconversion from 16 million colors to 256 colors just uses the "nearest color match" approach, they end up looking different.
Then there's the issue that, internally, Trespasser stores the texture names as CRC-32 hashes, and from a certain version onwards (at least Build 55), the hash is applied not just to the name itself but to the float Diffuse value appended to it... and when different objects with different Diffuse values use the same texture, it's actually stored twice for that reason, even if it's identical.
For my beach test I didn't upscale the normal maps, or even use the ones extracted along with the textures. Would those benefit from up-scaling?
Maybe, but let's wait until after we've generated the standard normal maps...
And I'm not really sure the same process would apply, since normal maps, even though they are stored as 24-bit RGB pictures, aren't really color pictures, just data that is stored as color pictures for convenience (X, Y and Z stored as the R, G and B values, I think, or maybe B, G and R).
I know ground textures aren't applied in CE so I won't do these. How do I identify them?
They usually have names that begin with "Trn", "TrnObj", or sometimes, "T" (or both: "TTrnObj")
"TGrass05t2" looks like a good candidate, what about things like "TBchClff30t2" and "TgiantBoulder01t2"
Unfortunately, the Giant Boulder isn't a terrain feature, but a regular object...
Would imposter sprites would benefit from up scaling, or are the new draw distances so good it's not necessary?
Imposter sprites are generated on the fly; I think you may mean the LoD meshes. And they sometimes use the same textures as the regular meshes.
Visit The Carnivores Saga - a forum devoted to modding Action Forms' Carnivores, Carnivores 2 and Carnivores: Ice Age games
Tres WIP: updated T-Script Reference and File Formats documents
Sound name listings for the Demo (build 117), Retail (build 116), Beta 103, Beta 99, Beta 97, Beta 96, Build 55, PC Gamer Alpha (build 32) and E3 1998 Alpha (build 22) TPA files
User avatar
Rebel
-=TresCom Developer=-
-=TresCom Developer=-
Posts: 6112
Joined: Sun Nov 10, 2002 10:26 pm
Location: That country nobody likes (you know the one)
Contact:

Re: Neural Network Texture up-scaling

Post by Rebel »

SWPext (don't remember if TresEd does too) has an option to extract the bumpmapped textures as raw data; then all we need to do is have another program process those files to write the corresponding standard normal maps as BMP or PNG images.
Presumably, Andres' bumpconvert utility does that, machf. I never used it, but if he released it, I'd assume that it does indeed work.
User avatar
Isolt
Stegosaurus
Stegosaurus
Posts: 62
Joined: Wed Jun 06, 2018 8:02 pm

Re: Neural Network Texture up-scaling

Post by Isolt »

The thing is, different levels use different palettes, for optimization reasons. So, what was originally the same 24-bit RGB texture in the source 3DS MAX files, may have been converted to 8-bit using a certain palette when exporting it for level A, and a different palette when exporting it for level B. And since the downconversion from 16 million colors to 256 colors just uses the "nearest color match" approach, they end up looking different.
Then there's the issue that, internally, Trespasser stores the texture names as CRC-32 hashes, and from a certain version onwards (at least Build 55), the hash is applied not just to the name itself but to the float Diffuse value appended to it... and when different objects with different Diffuse values use the same texture, it's actually stored twice for that reason, even if it's identical.
So slight colour variations but the same file names? Do you have any examples? I started off looking carefully at each double up but quickly gave up.
I could just keep each texture pack in it's own level folder and just live with double ups. The first level is about 250MB @ 2x with no normal maps, about 1GB at 4x.
Maybe, but let's wait until after we've generated the standard normal maps...
And I'm not really sure the same process would apply, since normal maps, even though they are stored as 24-bit RGB pictures, aren't really color pictures, just data that is stored as color pictures for convenience (X, Y and Z stored as the R, G and B values, I think, or maybe B, G and R).
Hmm no idea how they'd respond to upscale then, should make the download smaller without them. I will have to manually remove them though *sigh*
Unfortunately, the Giant Boulder isn't a terrain feature, but a regular object...
I suspected, so much for consistency. I'll remove the obvious terrain ones and leave any I'm unsure about, shouldn't be a large overhead in filesize.
Imposter sprites are generated on the fly; I think you may mean the LoD meshes. And they sometimes use the same textures as the regular meshes.
Ah I thought all the tiny tree textures were them
Hey now, those upscaled images are great! I was unaware of this technique, am impressed...
Drac I did cherry pick those :P I'm still experimenting, I made this beautiful watercolour barrel painting
Spoiler: show
PblFuelBarrelF00t2.bmp
PblFuelBarrelF00t2.bmp (1.5 MiB) Viewed 17388 times
User avatar
machf
T-Rex Killer
T-Rex Killer
Posts: 12368
Joined: Thu Apr 24, 2003 11:20 pm
Location: Lima, Peru
Contact:

Re: Neural Network Texture up-scaling

Post by machf »

Rebel wrote:
SWPext (don't remember if TresEd does too) has an option to extract the bumpmapped textures as raw data; then all we need to do is have another program process those files to write the corresponding standard normal maps as BMP or PNG images.
Presumably, Andres' bumpconvert utility does that, machf. I never used it, but if he released it, I'd assume that it does indeed work.
Nah, it converts Trespasser spherical normal maps into bump maps. What we need is another util to convert spherical normal maps into cartesian normal maps.

Stealing Andres' pictures from his original article to clear it up:

ImageImage
This is a representation of the 16-bit "bumpmapped" images in Trespasser after splitting the highest 6 bits a a paletted image map and the lower 10 bits as a curved normal map.

Then, if we split the 10 bits of the curved normal map into 6 bits for the azimuth and the lowest 4 bits for the elevation, it becomes:
ImageImageImage

Andres also showed what the X and Y components of the normal would look like after converting from spherical normals to cartesian normals:

ImageImage

He didn't calculate the Z component, but it's as easy as X and Y, and it would look similar too. Then we would just combine all three (8 bits each) into a 24-bit RGB image.
Here I've just averaged the X and Y images to create a fictional "Z" image (no, that's NOT the actual way in which it's calculated) just to show the results:
NormalImage9.jpg
NormalImage9.jpg (5.01 KiB) Viewed 17388 times
And after combining all three greyscale images into a single RGB one, we get:
NormalImage18.jpg
NormalImage18.jpg (6.09 KiB) Viewed 17388 times
That's what our standard cartesian normal map would look like, more or less. In some places it's grey, in others it veers towards cyan ("anti-red", so to speak) or towards magenta ("anti-green")...


Or just using the elevation part as an approximation of the Z component:
NormalImage19.jpg
NormalImage19.jpg (4.79 KiB) Viewed 17385 times
We then get this as the resulting normal map after combining all three:
NormalImage20.jpg
NormalImage20.jpg (6.35 KiB) Viewed 17385 times
Since here the Z component doesn't look too similar to the X and Y components, instead of a mostly grey image we get a mostly blue-violet one... if you look for normal maps online, you'll see that they tend to look like this.


Andres didn't perform this step, instead to generate bump maps, he performed numerical integration on both the X and Y components, with the results we already know, diagonal artifacts and all... but we don't need bump maps anymore for CE, we just need normal maps, and those don't need any complicated math that will end up with any artifacts.
Visit The Carnivores Saga - a forum devoted to modding Action Forms' Carnivores, Carnivores 2 and Carnivores: Ice Age games
Tres WIP: updated T-Script Reference and File Formats documents
Sound name listings for the Demo (build 117), Retail (build 116), Beta 103, Beta 99, Beta 97, Beta 96, Build 55, PC Gamer Alpha (build 32) and E3 1998 Alpha (build 22) TPA files
User avatar
machf
T-Rex Killer
T-Rex Killer
Posts: 12368
Joined: Thu Apr 24, 2003 11:20 pm
Location: Lima, Peru
Contact:

Re: Neural Network Texture up-scaling

Post by machf »

Isolt wrote: Ah I thought all the tiny tree textures were them
Ah, you mean the mipmaps... they are stored in order to save time, instead of having the came recalculate the texture at further distances (when it's shrunk), they're precalculated and stored in the .PID/.SWP files. I don't know how CE handles them It's likely that if you use .DDS textures, the mipmaps will be contained within them, too.
Visit The Carnivores Saga - a forum devoted to modding Action Forms' Carnivores, Carnivores 2 and Carnivores: Ice Age games
Tres WIP: updated T-Script Reference and File Formats documents
Sound name listings for the Demo (build 117), Retail (build 116), Beta 103, Beta 99, Beta 97, Beta 96, Build 55, PC Gamer Alpha (build 32) and E3 1998 Alpha (build 22) TPA files
User avatar
TheIdiot
T-Rex
T-Rex
Posts: 4343
Joined: Thu Feb 20, 2014 9:13 pm
Location: Canada, eh?

Re: Neural Network Texture up-scaling

Post by TheIdiot »

machf wrote:
Isolt wrote: Ah I thought all the tiny tree textures were them
Ah, you mean the mipmaps... they are stored in order to save time, instead of having the came recalculate the texture at further distances (when it's shrunk), they're precalculated and stored in the .PID/.SWP files. I don't know how CE handles them It's likely that if you use .DDS textures, the mipmaps will be contained within them, too.
So long as you generate mipmaps when you export your .dds files, yes, it does appear to use the mipmaps from the .dds files. The impostors however do not appear to benefit from HD textures and instead use the normal textures which are contained in the .swp instead.
Post Reply