Page 1 of 6

Neural Network Texture up-scaling

Posted: Sun Jun 30, 2019 10:00 pm
by Isolt
I've been playing around with a few image up-scalers that use machine learning to guess what the enlarged image would look like rather than just average out the pixels.
I though it'd be fun to run the trespasser textures through it.
Doubling them seemed to be a good compromise between size and benefit. Unfortunately the original images are only 8 bit colour so there are colour banding issues. They might benefit from some blurring?

Current Version
V2.2 https://www.moddb.com/addons/start/2057 ... xture-pack
https://www.trescom.org/download/nnhd-t ... -version/#
V2.2 light https://www.moddb.com/addons/start/2057 ... xture-pack
https://www.trescom.org/download-catego ... llaneous/#
Pine Valley V1.0 https://www.moddb.com/addons/start/1868 ... xture-pack
https://www.trescom.org/download-catego ... llaneous/#

Re: Neural Network Texture up-scaling

Posted: Mon Jul 01, 2019 4:21 pm
by machf
Nice results...

Re: Neural Network Texture up-scaling

Posted: Mon Jul 01, 2019 8:06 pm
by Isolt
Thanks.
They'll never be as clear as TheIdiot's remade ones, but they're better than any standard up-scaling.
It's not hugely noticeable in game but things are a bit more crisp.
Now to sort the texture overlaps from every level... There must be an easier way.

Image
Image
Image
Image
Spoiler: show
TPass005.jpg
TPass005.jpg (609.23 KiB) Viewed 17465 times
Spoiler: show
TPass004.jpg
TPass004.jpg (485.62 KiB) Viewed 17465 times
Spoiler: show
TPass002.jpg
TPass002.jpg (420.75 KiB) Viewed 17465 times
Spoiler: show
TPass001.jpg
TPass001.jpg (512.77 KiB) Viewed 17465 times

Re: Neural Network Texture up-scaling

Posted: Tue Jul 02, 2019 4:22 am
by machf
How many textures did you apply this process to?
And yes, the textures having been reduced to 8-bit palettes in the final step of the export for the game makes them somewhat problematic to improve... does the process get any advantage if you convert them to 24-bit RGB images first? At leas, normal resampling algorithms usually work better if you do that and then convert back to an 8-bit palette... and with CE, you don't really need to make them 8-bit again.

Re: Neural Network Texture up-scaling

Posted: Tue Jul 02, 2019 5:22 am
by TheIdiot
I was waiting for someone to do this. Had wanted to do it myself a while back but couldn't find a (free) tool to do so nicely, and retexturing everything from scratch takes a lot of time, especially considering a lot of Tres' objects have a large number of split-up textures, such as dinosaurs and vehicles.

The colour banding in the original images does suck, but there isn't a whole lot you can do about that without remaking the textures from scratch. You could try blurring the image a bit before upscaling but you'd lose some of the details. With CE you can use hardware textures in .dds format which should allow you to avoid having to convert the images over to 256 colors - I found saving them as .bmps causes some issues with lighting.

What I'd like to see now is a conversion of all of the original Trespasser bumpmaps to CE-compatible normalmaps so that they work in fan-made levels, since GeomAdd never got the ability to use standard Tres bumpmaps in custom levels.

Re: Neural Network Texture up-scaling

Posted: Tue Jul 02, 2019 7:43 am
by machf
TheIdiot wrote: What I'd like to see now is a conversion of all of the original Trespasser bumpmaps to CE-compatible normalmaps so that they work in fan-made levels, since GeomAdd never got the ability to use standard Tres bumpmaps in custom levels.
Technically, it shouldn't be too hard. Especially since, technically (yes, again), Trespasser isn't using bumpmaps but rather normal maps, only that instead of using cartesian coordinates to store the vector normals (X, Y, Z), it's using spherical coordinates (radius, azimuth and elevation).

Since by definition a normal vector always has a length of 1.0 (it's called a normal because it has been normalized), its radius in spherical coordinates will always be 0, so it doesn't need to be stored; the azimuth (horizontal angle) will go from 0 to 360 degrees (think of compass directions, with 0 being north, 90 being east, 180 being south, 270 being west and 360 back to north again), and while the elevation should go from -90 (pointing down) to +90 degrees (pointing up), the normal vector of a surface you're looking at will never have a negative angle, so it only has to go from 0 to 90 degrees. That's the reason why, from the 10 bits used for storing the normal maps (the other 6 bits being the palette index for a 64-color image), 6 bits are used for the azimuth and only 4 bits for the elevation. 6 bits mean 64 possible values, while 4 bits mean 16 possible ones, which is 4 times less, just kile 90 degrees is 1/4th of 360 degrees. The resolution is the same in both cases, 360/64 = 90/16 = 45/8 = 5.625 degrees.

Converting Trespasser's normal maps into bump maps has been more difficult than it would be converting them to regular normal maps, which only implies converting the spherical coordinates into cartesian ones and can be done by simple multiplication. Converting to bumpmaps, OTOH, has required numerical integration, and the results aren't accurate.

SWPext (don't remember if TresEd does too) has an option to extract the bumpmapped textures as raw data; then all we need to do is have another program process those files to write the corresponding standard normal maps as BMP or PNG images.

Re: Neural Network Texture up-scaling

Posted: Tue Jul 02, 2019 12:05 pm
by tatu
A member on the Discord have done this to several LAB textures and upscaling some quite good.

IIRC the SWPext have issues and export the bumpmaps with some lines in them? However, you can use the GUIApp from CE and export textures along with the bumpmaps. :)

Re: Neural Network Texture up-scaling

Posted: Tue Jul 02, 2019 7:48 pm
by Draconisaurus
Hey now, those upscaled images are great! I was unaware of this technique, am impressed...

Also.. spherical coordinate systems. Nice, I wonder if there are alien races who use this as their standard model rather than X,Y,Z box-coordinates..

Re: Neural Network Texture up-scaling

Posted: Tue Jul 02, 2019 8:52 pm
by Isolt
Hi machf.
I've done all of beach textures so that I could test it.
Last night I combined the textures from all the levels to avoid double ups. There was a note in the GUI app about having different textures with the same name in different levels. I went through them as I combined them but didn't notice any. Does anyone know of any exceptions? There are thousands of files and I may have overlooked some.
For my beach test I didn't upscale the normal maps, or even use the ones extracted along with the textures. Would those benefit from up-scaling?
I know ground textures aren't applied in CE so I won't do these. How do I identify them? "TGrass05t2" looks like a good candidate, what about things like "TBchClff30t2" and "TgiantBoulder01t2"
Would imposter sprites would benefit from up scaling, or are the new draw distances so good it's not necessary?

The Idiot, I want to avoid modifying the textures before up-scaling them. The less they've been modified the better the results are I believe. Hmm lighting issues with .BMP is concerning as that's what I was planning to use. TGA is an option, or I could batch convert them to .DDS with something?

My current plan is exporting all textures with GUIApp from CE. Combining them to remove doubles. Removing ground textures and normal maps. Upscale 2x with 24bit BMP's as the output. Then possibly add a bit of selective blurring on some of the worst banding.

Thanks tatu, you helped me get the GUIApp running over in the CE thread so that I could export these.

Re: Neural Network Texture up-scaling

Posted: Tue Jul 02, 2019 9:36 pm
by machf
tatu wrote: IIRC the SWPext have issues and export the bumpmaps with some lines in them? However, you can use the GUIApp from CE and export textures along with the bumpmaps. :)
The idea is not to export them as bumpmaps (that's why in the process they get those lines), but the raw data, and then convert THAT into a regular normal map...
Isolt wrote:Hi machf.
I've done all of beach textures so that I could test it.
Last night I combined the textures from all the levels to avoid double ups. There was a note in the GUI app about having different textures with the same name in different levels. I went through them as I combined them but didn't notice any. Does anyone know of any exceptions? There are thousands of files and I may have overlooked some.
Nice amount of work that must have been...
The thing is, different levels use different palettes, for optimization reasons. So, what was originally the same 24-bit RGB texture in the source 3DS MAX files, may have been converted to 8-bit using a certain palette when exporting it for level A, and a different palette when exporting it for level B. And since the downconversion from 16 million colors to 256 colors just uses the "nearest color match" approach, they end up looking different.
Then there's the issue that, internally, Trespasser stores the texture names as CRC-32 hashes, and from a certain version onwards (at least Build 55), the hash is applied not just to the name itself but to the float Diffuse value appended to it... and when different objects with different Diffuse values use the same texture, it's actually stored twice for that reason, even if it's identical.
For my beach test I didn't upscale the normal maps, or even use the ones extracted along with the textures. Would those benefit from up-scaling?
Maybe, but let's wait until after we've generated the standard normal maps...
And I'm not really sure the same process would apply, since normal maps, even though they are stored as 24-bit RGB pictures, aren't really color pictures, just data that is stored as color pictures for convenience (X, Y and Z stored as the R, G and B values, I think, or maybe B, G and R).
I know ground textures aren't applied in CE so I won't do these. How do I identify them?
They usually have names that begin with "Trn", "TrnObj", or sometimes, "T" (or both: "TTrnObj")
"TGrass05t2" looks like a good candidate, what about things like "TBchClff30t2" and "TgiantBoulder01t2"
Unfortunately, the Giant Boulder isn't a terrain feature, but a regular object...
Would imposter sprites would benefit from up scaling, or are the new draw distances so good it's not necessary?
Imposter sprites are generated on the fly; I think you may mean the LoD meshes. And they sometimes use the same textures as the regular meshes.

Re: Neural Network Texture up-scaling

Posted: Tue Jul 02, 2019 10:15 pm
by Rebel
SWPext (don't remember if TresEd does too) has an option to extract the bumpmapped textures as raw data; then all we need to do is have another program process those files to write the corresponding standard normal maps as BMP or PNG images.
Presumably, Andres' bumpconvert utility does that, machf. I never used it, but if he released it, I'd assume that it does indeed work.

Re: Neural Network Texture up-scaling

Posted: Tue Jul 02, 2019 10:55 pm
by Isolt
The thing is, different levels use different palettes, for optimization reasons. So, what was originally the same 24-bit RGB texture in the source 3DS MAX files, may have been converted to 8-bit using a certain palette when exporting it for level A, and a different palette when exporting it for level B. And since the downconversion from 16 million colors to 256 colors just uses the "nearest color match" approach, they end up looking different.
Then there's the issue that, internally, Trespasser stores the texture names as CRC-32 hashes, and from a certain version onwards (at least Build 55), the hash is applied not just to the name itself but to the float Diffuse value appended to it... and when different objects with different Diffuse values use the same texture, it's actually stored twice for that reason, even if it's identical.
So slight colour variations but the same file names? Do you have any examples? I started off looking carefully at each double up but quickly gave up.
I could just keep each texture pack in it's own level folder and just live with double ups. The first level is about 250MB @ 2x with no normal maps, about 1GB at 4x.
Maybe, but let's wait until after we've generated the standard normal maps...
And I'm not really sure the same process would apply, since normal maps, even though they are stored as 24-bit RGB pictures, aren't really color pictures, just data that is stored as color pictures for convenience (X, Y and Z stored as the R, G and B values, I think, or maybe B, G and R).
Hmm no idea how they'd respond to upscale then, should make the download smaller without them. I will have to manually remove them though *sigh*
Unfortunately, the Giant Boulder isn't a terrain feature, but a regular object...
I suspected, so much for consistency. I'll remove the obvious terrain ones and leave any I'm unsure about, shouldn't be a large overhead in filesize.
Imposter sprites are generated on the fly; I think you may mean the LoD meshes. And they sometimes use the same textures as the regular meshes.
Ah I thought all the tiny tree textures were them
Hey now, those upscaled images are great! I was unaware of this technique, am impressed...
Drac I did cherry pick those :P I'm still experimenting, I made this beautiful watercolour barrel painting
Spoiler: show
PblFuelBarrelF00t2.bmp
PblFuelBarrelF00t2.bmp (1.5 MiB) Viewed 17440 times

Re: Neural Network Texture up-scaling

Posted: Tue Jul 02, 2019 11:21 pm
by machf
Rebel wrote:
SWPext (don't remember if TresEd does too) has an option to extract the bumpmapped textures as raw data; then all we need to do is have another program process those files to write the corresponding standard normal maps as BMP or PNG images.
Presumably, Andres' bumpconvert utility does that, machf. I never used it, but if he released it, I'd assume that it does indeed work.
Nah, it converts Trespasser spherical normal maps into bump maps. What we need is another util to convert spherical normal maps into cartesian normal maps.

Stealing Andres' pictures from his original article to clear it up:

ImageImage
This is a representation of the 16-bit "bumpmapped" images in Trespasser after splitting the highest 6 bits a a paletted image map and the lower 10 bits as a curved normal map.

Then, if we split the 10 bits of the curved normal map into 6 bits for the azimuth and the lowest 4 bits for the elevation, it becomes:
ImageImageImage

Andres also showed what the X and Y components of the normal would look like after converting from spherical normals to cartesian normals:

ImageImage

He didn't calculate the Z component, but it's as easy as X and Y, and it would look similar too. Then we would just combine all three (8 bits each) into a 24-bit RGB image.
Here I've just averaged the X and Y images to create a fictional "Z" image (no, that's NOT the actual way in which it's calculated) just to show the results:
NormalImage9.jpg
NormalImage9.jpg (5.01 KiB) Viewed 17440 times
And after combining all three greyscale images into a single RGB one, we get:
NormalImage18.jpg
NormalImage18.jpg (6.09 KiB) Viewed 17440 times
That's what our standard cartesian normal map would look like, more or less. In some places it's grey, in others it veers towards cyan ("anti-red", so to speak) or towards magenta ("anti-green")...


Or just using the elevation part as an approximation of the Z component:
NormalImage19.jpg
NormalImage19.jpg (4.79 KiB) Viewed 17437 times
We then get this as the resulting normal map after combining all three:
NormalImage20.jpg
NormalImage20.jpg (6.35 KiB) Viewed 17437 times
Since here the Z component doesn't look too similar to the X and Y components, instead of a mostly grey image we get a mostly blue-violet one... if you look for normal maps online, you'll see that they tend to look like this.


Andres didn't perform this step, instead to generate bump maps, he performed numerical integration on both the X and Y components, with the results we already know, diagonal artifacts and all... but we don't need bump maps anymore for CE, we just need normal maps, and those don't need any complicated math that will end up with any artifacts.

Re: Neural Network Texture up-scaling

Posted: Tue Jul 02, 2019 11:34 pm
by machf
Isolt wrote: Ah I thought all the tiny tree textures were them
Ah, you mean the mipmaps... they are stored in order to save time, instead of having the came recalculate the texture at further distances (when it's shrunk), they're precalculated and stored in the .PID/.SWP files. I don't know how CE handles them It's likely that if you use .DDS textures, the mipmaps will be contained within them, too.

Re: Neural Network Texture up-scaling

Posted: Wed Jul 03, 2019 1:59 am
by TheIdiot
machf wrote:
Isolt wrote: Ah I thought all the tiny tree textures were them
Ah, you mean the mipmaps... they are stored in order to save time, instead of having the came recalculate the texture at further distances (when it's shrunk), they're precalculated and stored in the .PID/.SWP files. I don't know how CE handles them It's likely that if you use .DDS textures, the mipmaps will be contained within them, too.
So long as you generate mipmaps when you export your .dds files, yes, it does appear to use the mipmaps from the .dds files. The impostors however do not appear to benefit from HD textures and instead use the normal textures which are contained in the .swp instead.