Official VAR file compression poll

[meshedvr]

I think my announcement about image caching makes the decision easier. I also vastly improved audio loading from package files. I was doing something stupid there with only processing a chunk of data per rendered frame. I fixed to use a threaded loader. I also improved assetbundle loading. I was doing an extra memory copy and also loading in a way that resulted in a lot of buffer resizing. So that is faster as well.

What I will most likely do is compress any file that I know is not already compressed based on extension. So image files, audio files, assetbundles, etc. are going to be stored without compression because that doesn't save much space and just slows things down, but all other files (text and vam/vmb binaries, etc.) will be compressed. Fortunately the zip format allows for this as far as I can tell.

 

[Hunting-Succubus]

I already told in server when loading scene disk was at not active all the time, look like vam was loding file one by one , without multithreading..glad its fixed. what about dds format, it should greatly remove texture loading time.

 

[an3k]

I already told in server when loading scene disk was at not active all the time, look like vam was loding file one by one , without multithreading..glad its fixed.

I haven't noticed it being fixed. Is there a Beta Beta release?

what about dds format, it should greatly remove texture loading time.

Would be great if we could use it but don't make it mandatory since creating these is a hassle and many users/creators will not be able to. Maybe
if dds { straight to GPU memory } else { business as usual }

 

[Acid Bubbles]

Quick note about multithreading, loading files in parallel may be slower. What makes loading faster (except having multiple disks or counting the file system cache, which is very often forgotten in benchmarking) is loading less bytes, or parallelizing the CPU work. For example, if we're loading a file, then we decompress it, then we load the next file, we'll have a clear gain. If we parallelize, that means some IO work will block on multiple threads and the all threads may be busy decompressing or simply processing the bytes. You'll see marginal gain because CPU and IO may not always happen at the same time, which is where you'll see some gain.

 

[meshedvr]

The load time for images in 1.19.2.2 is 95% CPU processing (unzip (if in package), decompress (jpg/png/etc) to bitmap, process and format, generate mips, & DXT recompress) and 5% I/O. Most of the processing was done on a side thread to avoid hanging main thread, but it is correct that it is only doing an image at a time. That could be improved to convert multiple images at once using more threads.

@۞???????~???????? ۞ @an3k
1.20 still does all this same processing once, but saves result to a disk image, which is basically dds format without the header. I decided to not save a full dds because then the header has to be read and stripped (meaning another memory copy which slows down the load time) before passing to the Unity function that uploads the texture data to GPU. I don't think DDS makes sense as a user input format because it may still have to be converted if the data is not a type that works with the Unity function. VaM is using DXT1 and DXT5. Newer formats might be possible in 2.X.

I also experimented with compressing the cache file, which resulted in much smaller on-disk size, but this nullified much of the fast load gains so I decided against that. The goal of the cache is ultimate load speed, which I think it has accomplished.

 

[Hunting-Succubus]

@meshedvr I think texture browser/loader should allow to dds image side along png,jpg. if user has dds and jpg/png with same name, dds will be preferred over other slow format. this will allow to user to use both friendly png and fast dds texture. maybe a button to convert to dds image inside browser will help too.

 

[an3k]

Can we load bmp instead of jpg/png? I mean, since these get converted to bmp anyway ...

 

[hazmhox]

Can we load bmp instead of jpg/png? I mean, since these get converted to bmp anyway ...

I don't think you're in the good thread, but bmp is not really a format used in engines.

 

[meshedvr]

I'm not adding dds or bmp import support at this time. It makes no sense here now that caching will be a thing. Store images as png if you want max quality. 1st time they get converted they will be stored in dds-like cache file that is super fast to load (faster than DDS file) next time you use it. I added support for custom cache folder, so you can put the cache on a secondary drive and it is still blazingly fast even if 2nd drive is SATA HD.

 

[an3k]

I don't think you're in the good thread, but bmp is not really a format used in engines.

Meshed said himself BMP is used so ...

 

[Spacedog]

Meshed said himself BMP is used so ...

Meshed said 5 messages ago:

"VaM is using DXT1 and DXT5."

 

[meshedvr]

VaM is using bitmaps internally during 1st time conversion. This is required as you have to convert jpg/png to a uncompressed bitmap before it is then converted to compressed DXT1/DXT5. Then the cache is stored as DXT1/DXT5 binary (which is very close to DDS format). Next time that image is read, it doesn't have to go into internal bitmap format. It gets read straight from the compressed DXT1/DXT5 binary to the texture load function. This provides ultimate load speed which is what the goal was here.

What I'm not planning to add support for is bmp input format because png is superior in every way. Much smaller in size on disk and is lossless.

2.X will provide another option where you can make assets in Unity yourself that will work in VaM directly with no conversion and native load speed.

 

[Hunting-Succubus]

how much space cache storage will take. if I have 50 gb of texture then cache storage will be also 50 gb?

 

[AWWalker]

I suppose for speed junkies we could unpack and repack the vars with no compression?

 

[meshedvr]

how much space cache storage will take. if I have 50 gb of texture then cache storage will be also 50 gb?

The cache files will be larger than input png/jpg files because DXT compression does not reduce size as much. Also, normal maps are stored uncompressed because normal maps are very sensitive to compression artifacts.

512x512 thumbnails (generated when you browse a folder) are 128KB each if they don't have alpha channel and 256KB if they have an alpha channel (some pngs).
4096x4096 color textures (RGB) stored in DXT1 format are roughly 10.5MB each.
4096x4096 color textures with alpha stored in DXT5 format are roughly 21MB each.
4096x4096 uncompressed normal maps are roughly 85MB each.

So this can add up in size. I suggest using a large secondary drive if you are concerned about primary SSD space.

The cache can also be shared between multiple VaM installs.

 

[meshedvr]

I suppose for speed junkies we could unpack and repack the vars with no compression?

I'm not planning to support no compression in 1.20 packaging. I'm going to change it so some files are not compressed (mp3/ogg/png/jpg/etc.) since there is little or no gain to compressed size while hurting save/load performance, while the rest (json, vmi, vmb, etc.) will be compressed. Then the image caching being discussed will be used if enabled after 1st load of any image. Image caching speeds up both local images and packaged images.

I also already fixed a serious load time issue with packaged audio files that will be in 1.20.

 

[an3k]

So that means we do have to keep the compressed jpg/png texture as well as the uncompressed cached version. We can't delete the compressed one because then VaM throws a tantrum about missing sources and we can't delete the uncompressed one because it gets automatically generated when we lead the compressed one.

It would be really nice if we could store in uncompressed format to save space.

 

[meshedvr]

You can turn off caching if you are worried about space. And 1.20 will make it so textures in packages are not compressed twice (png/jpg + zip) so it has a speed improvement over 1.19 with that alone. So even with caching off 1.20 is better than 1.19.

IMO - it makes less sense to distribute the larger cached images to everyone as on average that takes more space for everyone and also takes longer to download. The packages folder can't easily be placed on a secondary drive. The cache, however, can be placed anywhere. Also the cache could have a max-size management system, and use a least recently used formula for cleanup. Lastly, if you were to distribute uncompressed cache images, those files could not be edited by end-users as they are in VaM-specific format. I'm not using DDS or raw bitmap because those require additional processing before load into GPU.

 

[TToby]

For some people disk size obviously doesn't matter, for many it does. I was in game development in a time, where all game content has to fit on a single CD then DVD and we did texture/file optimisation like crazy. Like CitizenX I absolutely can't stand having a 4k 25MB uncompressed normal map only for socks (one per pattern...) sitting on my HDD, so I edit every single damn download. But as slow as VaM sometimes is, speed surely does matter a lot, too. As the compressing poll is meaningless now because of Mesheds new way to load Vacs, We still have those other issue that IMHO eats speed and nerves: If I load a person or clothes with custom textures (=100% of my saved looks), VaM still loads the default textures first, then it loads the custom textures. But maybe this issue is already addressed in VaM 2.0 and this posting is completely useless ;-) .

 

[Deleted member 1503]

Storage seems cheap now days (just bought another 2tb m.2 for 160 bucks), seems better to have faster loads.

 

[aaron300]

I personally hate the new .VAR system. I unzip the var using 7-zip, and put the scene in my own filing system. So I would say uncompressed would work better for me. Thanks for asking! Love that you're looking for input.

 

[an3k]

You can turn off caching if you are worried about space. And 1.20 will make it so textures in packages are not compressed twice (png/jpg + zip) so it has a speed improvement over 1.19 with that alone. So even with caching off 1.20 is better than 1.19.

IMO - it makes less sense to distribute the larger cached images to everyone as on average that takes more space for everyone and also takes longer to download. The packages folder can't easily be placed on a secondary drive. The cache, however, can be placed anywhere. Also the cache could have a max-size management system, and use a least recently used formula for cleanup. Lastly, if you were to distribute uncompressed cache images, those files could not be edited by end-users as they are in VaM-specific format. I'm not using DDS or raw bitmap because those require additional processing before load into GPU.

I don't want to turn caching off. I want to be able to delete the compressed jpgs/pngs if there is a cached version and without VaM throwing an error because it can't find the textures. Ideally it should check for a cached version, then for a compression one and ONLY if both fail throw an error.

 

[cramonty]

jpeg will not compress either, but wav files will as far as I know.

(WAV files will not compress. They can be converted losslessly to FLAC, or lossy to AAC or OGG or mp3, though it would take time.)

 

[Mark Seven]

I too find the whole thing with VAR, VAM, VAC etc also very confusing. Many VAR files don't work out of the box. I have to decompress them and then drag the files one by one to each folder. Also I find it strange that if the VAR contains a look, I can't select it from Load Look, as Load Look refuses to go to the top directory so that I can never navigate to the AddOn Packages directory in that top directory. So I now unzip the VAR and drag the two look files to my Scene directory, where I can download them as a scene. It works, but it's way too messy.

 

[an3k]

I too find the whole thing with VAR, VAM, VAC etc also very confusing. Many VAR files don't work out of the box. I have to decompress them and then drag the files one by one to each folder. Also I find it strange that if the VAR contains a look, I can't select it from Load Look, as Load Look refuses to go to the top directory so that I can never navigate to the AddOn Packages directory in that top directory. So I now unzip the VAR and drag the two look files to my Scene directory, where I can download them as a scene. It works, but it's way too messy.

If you're "unzipping" the VAR manually: NEVER DO THAT, it will break stuff. If you really have to "unzip" a VAR, use the internal function in Virt-a-Mate in the Package Manager.

VAR packages do work out of the box, some may need additional VARs to be downloaded but they will tell you exactly the author name and the package name so you can easily search for them.

Containing a Look but you can't load them because the AddonPackages folder isn't accessible? You should have a list of VAR packages that contain looks at the far left of that file picker window. If you do NOT have it you really have to update your VaM installation using the Updater.