Official VAR file compression poll

[meshedvr]

Currently, VAR files contents are stored with compression. While this is great for reducing the size of the var, it can cause load times to be higher than if the files were stored uncompressed. There is an option for me to change it so VAR files are still in ZIP format, but contents could be stored as uncompressed. This would give identical performance to if the files we fully unpacked out of the var onto disk (what the Unpack feature can do).

I will make this change or not in 1.20 depending on poll results.

Thanks!

 

[MacGruber]

VAR files should use LZ4 chuck-based compression (or something similar) as it is an option for Unity AssetBundles. This allows you to load individual files without having to extract the entire archive. Since memory and CPU is much faster than disk access, even on normal SSD (*), loading compressed data should be faster or at least similar while using less disk space. Windows folder compression uses the same principle.

(*) maybe not these newer NWMe SSDs reaching like 3GB/sec speeds, but certainly normal SSD connected via SATA.

Edit: Question is of course what's the majority of data? If it is textures in JPG/PNG/etc. formats or already compressed AssetBundles, additional compression is pointless.

 

[meshedvr]

I like using ZIP for VAR files because it allows you to use popular tools on them for examining or modifying. ZIP with standard compression or no compression does not require extracting entire archive to pull out a single file. You can very efficiently pull just a single file out of large ZIP. VaM is already doing this. You can see this pretty clearly if you try out a bundle that contains a lot of textures and only load one. This poll is asking if we should move from current default of using compressed files in the zip, to simply have the full files in the zip. By using full files, there is nearly no performance difference compared to having the files directly on disk.

Assetbundles are a different topic, and I already decided to use LZ4 compression a long while back for this reason. If I use the other compression options, Unity actually extracts a full copy of the assetbundle to your AppData folder to use for caching, making disk usage much worse (more than 2X). Uncompressed is also option for assetbundles, but I'm happy with LZ4 performance for assetbundles and it does greatly reduce their size in most cases.

 

[Acid Bubbles]

Well I'm guessing compression may not make that much of a difference anyway. For example, I just tried compressing 4 png files, and uncompressed they weight 1.37MB, compressed they weight 1.35MB. You might want to run a few tests. For plugins I think it's really not useful, jpeg will not compress either, but wav files will as far as I know. So this is really the only time where it might be worth it.

If it's possible to only compress files we know would benefit within the same file, that could be an option (I didn't confirm), or it could be an option at the time of creation, or finally it could be determined based on total gain (e.g. you do the compression, check the size, if the gain is more than 30% or more then you take the compressed version). And even then, I'd only use the compressed version if the total weight was 200MB or more (throwing numbers here).

One last option to consider: people can zip themselves if their goal is to share a large var file.

 

[meshedvr]

Today, it is pretty obvious to me that loading a full set of character textures from a VAR is way slower than loading from disk. Someone else was just commenting on this today as well. That is what drove me to make this poll.

I might just offer a way to turn vars into uncompressed versions using the Package Manager (kind of like Unpack now, but leave everything inside the VAR) to offer same performance as unpacked, but in neater form.

 

[CitizenX]

Tangentially-related, something I'd really love to see is a way to optimize the textures within a VAC or VAR in-game, either natively or via a plugin. I wrote a php script to do it on my own install for newly-added clothing files, because I just don't need a 40mb normal file And for most clothing items 512x512 is barely distinguishable from 4096x4096 on my non-Pro VIVE. But if at some point there were an option to do this in-game it would be fantastic.

On the topic at hand, I voted for no-compression, because I agree that it probably doesn't actually make much difference in the file-size. Though I can find out... Since I'm going to be working with a ton of VAR files to do some testing, I'll run a script to see what the difference is compressed and uncompressed and let you know the results.

 

[Blazedust]

Tangentially-related too: The bottleneck during loading for me is the textures (I have VaM on a mechanical hard-drive). Reducing the texture size to half significantly decreases loading times for me. Especially noticable when you have a scene with a lot of textures to load. I also made a private tool just to reduce any textures down to a level-of-detail that's suitable for me (2048x2048). For all 256 var files I have installed and reduced textures for I have reduced the overall space required from 9.58 GB down to 5.54 GB. (I keep a backup of the original var files in case I want to restore anything and for comparison). For me the majority of space consist of textures (png and jpg).

Personally, I vote for no-compression, reduce loading times where you can.

 

[Spacedog]

I don't suppose there is some sort of slight compression that could be as fast but not compress much?

The only thing that I like compression for is animation data, which compresses very nicely. I think I'll take performance over that any day though.

 

[hazmhox]

As @Acid Bubbles said, on some resources, compression doesn't matter. And if the creator knows what he is doing, most of the time the resources will be optimized and compression, again, does not do much on optimized resources. And to be honest, before the var system we did not have compression at all.

Watching this from a game dev point of view, just like @Spacedog said, I'll take performances over compression any day. The experience is what is important in VaM not the size occupied on the hard drive. And almost all choices are made that way in an engine : you focus performances then size when you can. If your performances are drastically impacted by compression, you start to find a way to keep the perfs without compressing that much.

Seeing the few threads about who has the biggest VaM folder on reddit, I don't think that size matters after all

The goal of var being a way to compile all resources in one file, compression can be left out.

 

[NutellaBrah]

I dunno but I was not sure as a noob what was even going on. I was confused about the structure after i unpacked a var to save time for audio files because the folders were all over the place, not sure if i should have foders in the Addon packages Folder too. I was getting the red errors when loading but the scene seemed to mostly work and i followed all instructions but still got red errors.

i had to actually research the fact that Var goes into addon packages folder.

I figured a scene still went into scenes like a vac would.

To my dumb logic i was thinking "why isnt the addon folder called a Var folder?" and thinking that an add on was more like a plug in rather than a place to put a scene.

The compression was confusing and i wasnt sure if I have to do it or if its just recommended. I wasnt sure if I was supposed to repack it if i wanted to fix it. I saw then a file called .var.orig and wasnt sure but thought that was the unpacked form.

But then I still had red errors in a var scene and it would looking in the "custom" base folder and not finding something. but still semi worked but not sure if it was missing pieces.

I guess im thinking of the old way to do it but I would say the goal is to make it so no more red errors can ever occur.

even the dependency of having to download like 3 or 4 plugins to get a scene. why not include all those in your .var list.

A fully working delivery should be a zip with one or more .VARs that you put in a VAR folder and everything that scene needs should be in the zip so you can just literally unzip a group into a folder and you find it in game without need to decompress or prep.

this is coming from a clueless noob who didnt keep up with 1.19 and im just now like... ???

I had to google "VAM how to install VAR"

Surely the last thing im worried about is disk space on my drive. Id rather have uncompressed neatness and fast load speed over saved space.

 

[meshedvr]

@NutellaBrah I hear your frustration with VAR stuff. IMO, a lot of this is just a new-feature startup problem.

If you open the online browser in VaM today and navigate to the Hub and download a VAR, it will correctly install it for you into AddonPackages. If it has a scene in it, it will even open it up for you automatically. This is step 1. Step 2 is coming in 1.20 where we will have a dedicated way to browse the Hub and get addons. 1.20 will also add auto-download of dependencies (other var files) when those var files are hosted in the Hub. For VAR files hosted externally (mega, etc.), VaM might also be able to download via a prompt approach. VaM will know which files you need for content, and it could also alert you when it can't find something required (not uploaded or linked on Hub).

For now, the zip file with all vars needed in it is a solution, but that would lead to users downloading the same content over and over again.

 

[NSFV]

@meshedvr for a current solution for new people to VaM, I think there should be a quick guide to install VARs on the site that either creators can quickly link to or automatically be posted on all resource posts. Maybe there should be check boxes when uploading a resource based on the type of format it uses and based on that people will be given a pop up that tells them how to install it manually. Just like for mods on the nexus, I think there will be people who prefer to manually install things despite VaM having it's internal browser. Some people just won't want to install things in VR.

 

[NutellaBrah]

@NutellaBrah I hear your frustration with VAR stuff. IMO, a lot of this is just a new-feature startup problem.

If you open the online browser in VaM today and navigate to the Hub and download a VAR, it will correctly install it for you into AddonPackages. If it has a scene in it, it will even open it up for you automatically. This is step 1. Step 2 is coming in 1.20 where we will have a dedicated way to browse the Hub and get addons. 1.20 will also add auto-download of dependencies (other var files) when those var files are hosted in the Hub. For VAR files hosted externally (mega, etc.), VaM might also be able to download via a prompt approach. VaM will know which files you need for content, and it could also alert you when it can't find something required (not uploaded or linked on Hub).

For now, the zip file with all vars needed in it is a solution, but that would lead to users downloading the same content over and over again.

Oh thanks!

I was looking through the online browser and it looked like that's what it does and almost downloaded from it but wasn't sure. Will give it a try it looked like a good way to get stuff and just be in the scene without much folder work and pre planning.

And yeah good point. After you have the plugins you don't want to get them again every time.

 

[SelchidhVR]

Working with audio in the compressed format makes load times impractical for the type of audio-heavy scenes I'm doing.

I've been asking users to unpack before use, which, at best, taints the experience with extra steps, or more likely, simply pushes casual users who aren't carefully reading instructions to move on to the next scene when the load times become apparent.

Needless to say, I would love the packages to be uncompressed.

 

[NutellaBrah]

Working with audio in the compressed format makes load times impractical for the type of audio-heavy scenes I'm doing.

I've been asking users to unpack before use, which, at best, taints the experience with extra steps, or more likely, simply pushes casual users who aren't carefully reading instructions to move on to the next scene when the load times become apparent.

Needless to say, I would love the packages to be uncompressed.

Yeah yours were the scenes I had confusion with. (fixed it all in the end after a while)

I actually thought the fact that I unpacked it caused the errors lol

Two thoughts I had:

1. Perhaps we can have a big "compatability bundle" (download it if you don't already have it) that someone could put together that has a large majority of all the commonly used and shared assets all in one huge zip that you just dump into the add-on folder. Or would it be impossible for them to be added to the releases via updater?

2. What happens if you use the online browser and download something that has external dependencies? It won't auto find the external files right?

maybe all deliveries can have the main .var scene being delivered, and then the page can have all the dependencies also in a small optional bundle. Or like how download sites have a list of files you want to zip, and maybe have all the needed assets there, and people who already have them can skip downloading them.

I found it quite intense having to use the txt file to follow a link to not even a post, but a Patreon timeline where I then spent (sometimes whole minutes) scrolling down the timeline through more recent posts to find the said dependency post based on the name of it. (theres no search that I saw)

As I was hunting them all down on various Patreons, I was like am I really doing this? -but of course it was worth the fap.

So hopefully that helps seeing it from a noob/casual user perspective.

 

[Guy]

I have a NVME PCIE 4 ssd, very fast, and loading times are definitely tied up with something because they might hit 20%? Less? None of my 16 threads are maxed either. I'm a fan of any performance improvement we can get. VAM is an app that will prompt people to upgrade their machines if you take advantage of it. I say go nuts and don't try to be compatible with potatoes from 2005.

I am also NOT a fan of a complicated install. I would love to be able to filter out any content that doesn't follow the new hosted standard of easy installs. Auto-updating modules would be cool too.

 

[kemenate]

I agree with the two comments about textures above. I made the same experience. Especially when using noise in texture creation the file size can explode. At some point I started to run every image (png textures, as well as all the preview thumbnails) through tinypng.com (they should pay me, I recommend them so often...). And I deleted many of the auto-generated thumbnails in my old packages. This saves roughly 70% of package size and I can see 0 loss of quality.

Now the thumbnails are skippable, which helps to control image amount. Downside is, that many of my presets don't have a preview. Is there maybe a way to assign one thumbnail to multiple presets? I have this A LOT in hairstyles: 50 items, that all have the same 6 color presets. So instead of having 300 single images, 6 would suffice.

Maby there could also be a way to help creators find huge textures and raise the awareness: a note/icon in the material settings overview next to the texture name saying smth like "this image seems to be unusually big".

 

[CuteSvetlana]

I like the idea of letting the creator chosing his/her preferred compression mode. Personally, I am creating my var manually. So I would just prepare two zip files and see if the compressed version is worth or not.

 

[Maxman3D]

I vote for uncompressed. I haven't tested it, but do find that switching to VARs that textures do seems to be loading a bit slower

I like the idea of letting the creator chosing his/her preferred compression mode. Personally, I am creating my var manually. So I would just prepare two zip files and see if the compressed version is worth or not.

The problem I see with letting the creators decide is that if anything, the VAR system has shown that some of them can't follow directions since many VAR files are packed incorrectly and still including the dependent VARs which is making things worse than the VAC system IMO. I feel it just needs to be a standard one way or another.

 

[hazmhox]

You can default the base choice to the best one ( in this case uncompressed ) and let advanced creators who knows what they are doing do whatever they want.

It's not because a few newcomer creators made a mistake that you should think your tool based on that
Or you end up having Windows 10, an OS that makes the life of the most advanced users miserable without having a real administrator profile and a ton of features hidden and only editable if you fiddle inside the registry.

 

[CitizenX]

So I haven't done a ton of testing yet, but I did find one potential area for improvement within VaM... I was looking at a pack of 40 poses... 11mb VAR, 61mb uncompressed. The main culprit? The VaM-generated thumbnail images. It seems that when you take a screenshot for a thumbnail in VaM... for pose, look, scene, etc. it seems to be saving the jpg at 100 quality. A typical thumb at 512x512 (the size they are saved at by VaM) would be about 180k. Save the thumb at even just 60% jpg compression (which isn't even a noticeable difference) and it drops to about 40k.

So @meshedvr... if you're able to adjust the compression on saving thumbnails, that could make a big difference for storage space and optimization, not just for VARs, but for VaM in general.

 

[meshedvr]

@CitizenX Yeah I can fix that one issue so thumbnails are smaller. I'm also working on a much bigger load performance enhancement that will probably nullify this entire poll.

 

[meshedvr]

I fixed the thumbnail processing to use default compression (70%), and the thumbnails are coming out 4x smaller! Oops! Thanks for the catch @CitizenX

 

[AWWalker]

I like the packages, but personally I've got a huge amount of storage, so space isn't an issue, but the packages are awesome. I vote no compression. I have a dedicated nvme SSD just for VAM

 

[an3k]

I like using ZIP for VAR files because it allows you to use popular tools on them for examining or modifying.

VAR files should use LZ4 chuck-based compression (or something similar) as it is an option for Unity AssetBundles. This allows you to load individual files without having to extract the entire archive.

What about the 7-Zip software, the 7z container and the LZMA2 compression? I mean, 7-Zip is free (like open-source no ads free) and can open almost anything.

7-Zip - Wikipedia

en.wikipedia.org

7z - Wikipedia

en.wikipedia.org

Lempel–Ziv–Markov chain algorithm - Wikipedia

en.wikipedia.org

I generally agree with "speed before size". Especially in VR it's really annoying to sit there, staring at nothing and wait until stuff is loaded (which takes noticeably longer than before VARs)