VaM 1.x Dear VaM team, please pause V2 work for a day or two and fix the loading speed issues in V1

[gernb]

I love VAM. I love the team. But there is clearly some kind of O(N^2) or similar issue in VaM related to loading. There is no other app out there that takes more time to load with more unrelated files sitting around. Load up Pick VAM Produce 69 II and check the profile. The loading issue should be easily findable.

 

[atani]

You are refering to a really big scene with hundreds of dependencies, it is an outlier which is expected to have long loading times.

 

[hazmhox]

There is no loading issue, only scenes too big, assets unoptimized, installs way too massive... or eventually maybe, PCs too old.

 

[youngpeop]

Even with a 7800X3D system, loading times take over 30 seconds... many gamers probably find this scene slow even at home.

 

[hazmhox]

...

You are in an english speaking community, we tolerate foreign languages if the authors are speaking the same language in the same thread.
But out of respect for thread authors, follow the post main language and use Google translation if you are not speaking english. Thank you.

 

[youngpeop]

You are in an english speaking community, we tolerate foreign languages if the authors are speaking the same language in the same thread.
But out of respect for thread authors, follow the post main language and use Google translation if you are not speaking english. Thank you.

The translation wasn't switched before pasting, which caused the issue... Thanks for reminding me.

 

[hazmhox]

Thank you very much sir <3

 

[Origin69]

Vam isn't a "game" the way most are made. The built in stuff is well optimized. If you have a fresh install and use built in assets and textures, things start up and scenes load like lightning. The way it's set up to use community made content through vars is what takes scenes longer to load, and the more complex the scene the more dependencies you will have. Some content creators are better at optimization than others, but there are limits. If you have a lot of content in your custom folders or download every var you see on the hub to your addonpackages, then loading times suffer.

 

[gernb]

So running the visual studio profiler, AFAICT, every .var file is read at startup. No idea why. if it's actually loading the files that's bad. I'm going to assume it's just reading the. JSON. But why on every startup? Why not load them from the Cache folder on startup? Check the date. If the JSON file in the cache is newer than the VAR then just use the Cache?

Also, maybe profile the JSON parser, given you're reading 1000-2000 JSON files, if the JSON parser is slow or if the usage of it has a O(N) look up for properties instead of O(1) then that would also explain where lots of the time is going. Also if the JSON loader is loading the file piecemeal, that would be slow. It should be loading the entire JSON into memory and pointing to the string inline (though maybe that's not possible in .NET). It would still be possible to load it at once and parse from memory though (if it's not already doing that).

You could also consider Caching in a binary format so that it doesn't need to be re-parsed when loading from the cache. You could also consider using SqlLite or something and cache everything in an easy to lookup way. Because why read every single vam files JSON file.

At least in Visual Studio's Profiler it doesn't have any symbols so It's not clear where the time is going. For startup, it's clearly that it's reading so many files.

 

[Sizzle]

The big problem with everything, as always, is how VaM is made. I imagine it started out from a much more basic perspective, but in the end, people's creativity has gone much further, causing the ‘game’ to suffer as a result. We should definitely have a better system for basically everything. Using a base that is totally dependent on plugins and other external dependencies is a bad thing. Animation plugins should be integrated into VaM, as should many other dependencies that are standard nowadays for any type of scene. Let's hope that all this changes in VAM2, because VAM1 is currently an internal mess that makes it so slow and its performance so poor in scenes that shouldn't be a problem. VaM is made with Unity, I don't know why there are optimisation problems in scenes that simply have smoke effects, lights and other things. Even with an i9-13900k with a 4090, you cannot get high and stable performance in """complex""" scenes. Unity supports all kinds of particle accumulations, high-resolution assets, even raytracing and so on. As I said, unless VaM uses an ancient version of Unity I don't know what is causing all the performance issues that VaM suffers from, but hopefully they will be fixed in VaM2.

 

[hazmhox]

It won't be ( fixed in VAM 2 ).

I'm not undermining Meshed work here, but simply the concept of optimization. Optimization is generally made toward your target. Your target is "most probable situations". People having 3TB installs of VAM is not "most probable situations", and is a nightmare to deal with.

People want to have 3TB installation and have instant loading. This is NOT possible... not matter how you look at it.
The cache situation is somewhat the same deal. If you rely on cache, you're leaving potential updates behind until your cache expires. So the best compromise is looking at files (vars) to ensure they're not newer than the cache... which leads to this read at the launch of the game to ensure cache is not obsolete.
More or less it's a compromise between people whining because "I have obsolete versions in my game" and "I have longer loading times". No matter your approach, you'll have one side or another complaining anyway.

Unless you're, yourself, controlling your VAM install size... it's unlikely you'd have a massive install running flawlessly. VAM 2 will probably bring improvements... but will at some point hit some diminishing returns as the installs grow in size.

And the second aspect is authoring. @Sizzle, you're mentionning particles at some point... I've checked most particles made by the community... more than 50% are made without understanding overdraw or GPU instanciation. This is beyond VAM's ability to deal with performances. If modders/creators release poorly made content, VAM can't do nothing about it.

On top of that, you could have proper VFX released, and scene creators slapping 20 instances in a scene without accounting for optimization either.

The amount of layers for optimization is so insane that you can't hope for proper perfs unless you are, yourself, understanding every aspect of the puzzle.

 

[Calvados]

There are many things that vam1 cannot do, which is why they decided to create vam2.
Between 2018 and 2026, various hardware and software evolutions took place.
When was Unity 2018 announced?
When was Ray Tracing announced?
CPUs in 2018 were mainly Intel's 8th generation (Coffee Lake/Kaby Lake Refresh) and 9th generation (Coffee Lake Refresh).
For AMD, Ryzen 5 1600, Ryzen 7 1700, etc.
You can't have excessive expectations for Unity 2018.
If you need a more advanced plugin than the current ones, you'll need to use an external DLL with BepinEX.
Even if you create one, the base is from 2018, so its capabilities will be limited.

I used Google Translate. I apologize if my English is incorrect.

 

[Clownmug]

From what I've observed it doesn't seem like the content of scenes matter that much other than the number of person atoms. Loading person atoms feels like ~90% of the time it takes to load a scene. On a fresh start it might take over a minute to load a scene with two people, then if I load another scene with two people after that it might only take a few seconds. But if I load another scene with three people, suddenly it takes a long time again. I can see the progress bar zoom past the first two people, then hang on the third.

I'm assuming this is because most person atom data stays in memory. If you start an empty scene and start adding person atoms you'll even see them keep the names from previous scenes, then hang for some time when adding new ones. Technically you can "cache" things if make a scene that's just like five person atoms and then load it immediately after starting VaM. Most scenes after that should load relatively quickly. Overall I don't think it's possible to "optimize" anything without compromising the quality of character models though.

 

[AshAuryn]

VaM 1.x is a successful application that has reached the end of its development lifecycle (excluding any potential future security updates). Development is frozen for 1.x.

The team is, and will continue to be, working full speed on VaM 2 development. VaM 2 solves many of the loading issues present in VaM 1.x. For example, here is a video of the VaM2 beta loading person atoms.

 

[Origin69]

The Vam community would like to reassure you that this video is not intended as a threat. Any fear response from witnessing AshAuryn and her army of G8F clones is entirely on you.

 

[gernb]

People want to have 3TB installation and have instant loading. This is NOT possible... not matter how you look at it.

It is possible. My music app does it just fine. Kodi, my media app does it just it just fine. Both of them scan each file ONCE and put all the metadata in a database. After that, both launch instantly. If I kill the database and then run them again they will do exactly what VaM is doing, and read every media file. It will take it 15-20 minutes until it's finished. This is EXACTLY THE SAME SITUATION

 

[Anand]

People want to have 3TB installation and have instant loading. This is NOT possible... not matter how you look at it.

With all due respect, it is absolutely possible. I have almost 4TB of VARs and yet my game load time is almost as fast as a virgin VAM installation.

The key is to NOT allow VAM to load the entirety of AddonPackages (more specifically, the morphs) when the game starts.

sFisherE first showed the way when he posted his Var Browser plugin in Dec 2022. JayJayWon more recently took a similar but slightly different approach with his Browser Assist. BOTH of these solutions when used properly enables VAM to function smoothly even with arbitrarily large VAR collections. (Shoutout to gicstin's VPB which is a recent fork and update of Var Browser). What these solutions do is to "hide" VARs from VAM so as to prevent it from loading all the VARs. But they give the user a way to browse and selectively load VARs (and associated dependencies) while the game is running.

(There are other Var Management tools recently published. While they also hide VARs from VAM at load, they don't provide a way to access those VARs during game play).

These plugins are workarounds, but are definitive proof that one can have tons of VARs but still have fast load times and no dreaded "Too Many Heap Sections" error.

VAM's requirement to have everything pre-loaded at the start as opposed to as-needed loading during game play is the fatal design flaw.

For some reason, the pillars of VAM here on the hub (who I respect) just will not accept that there are other ways of skinning the cat. Users have a tendency to collect large libraries of VARs (whether by intent or laziness in collection management). VAM's current approach to loading everything causes a lot of problems and is not how modern games or media applications are implemented. While VAM 1 is pretty much frozen (and we are forced to use workarounds), I can only hope that VAM 2 will not repeat the mistake of VAM 1.

For those who not yet familiar with Var Browser, take a look at VamEssentials' guide on Var Browser (https://hub.virtamate.com/resources/how-to-setup-and-use-var-browser.55752/). It was very well written and if it were available when I first tried Var Browser, that guide would have saved me countless Heap Errors and hours of banging my head against the wall.

 

[hazmhox]

The key is to NOT allow VAM to load the entirety of AddonPackages (more specifically, the morphs) when the game starts.

In before "It's is possible sir, if you don't load anything". You are litterally proving my point... which btw I stated in my post : "Unless you're, yourself, controlling your VAM install size"
With a normal installation (not tweaked/modified), you cannot reach the same speed as a clean install, or small install... and the amount of people not knowing that var and especially morphs and auto-loaded assets (like clothes/hairs) are the choking point of a VAM install is bigger than those knowing you need to control that.

For some reason, the pillars of VAM here on the hub (who I respect) just will not accept that there are other ways of skinning the cat. Users have a tendency to collect large libraries of VARs (whether by intent or laziness in collection management). VAM's current approach to loading everything causes a lot of problems and is not how modern games or media applications are implemented. While VAM 1 is pretty much frozen (and we are forced to use workarounds), I can only hope that VAM 2 will not repeat the mistake of VAM 1.

I think you're confusing accepting and stating a fact.

When you say that you control your VAM install through unloading vars to speed it up. You are simply creating a workaround which will require you to have several setups to access the content. We have people that just don't want to micro-manage data, plain an simple. They want to access everything, all the time... and on top of that want a perfectly fast app... I'm simply saying it's not possible factually and implicity it means you either control your VAM install, or you just accept the side effects.

Suggesting a load "on demand" thinking it would solve everything is also not true. Most recent scenes with the insane amount of custom textures on characters is a great example. Custom textures are loaded on demand and caching on the fly. This is probably the biggest hit on a scene loading and can make the VRAM skyrocket if you don't hard reset before playing the scene.

A bunch of people were complaining of the stutters not so long ago, and I'm seeing people scripting scene with presets load at playthrough triggering the VAM caching/loading window. This is something I really do not enjoy in a scene, and this is "load on demand". Streaming and loading on demand will just shift the problem and creates other issues.

Also "modern games and media applications" are not "implemented that way"... because there is not a single game even close enough to the very insane installs we saw over the years. Games considered very big hover around 110 to 150gigs. This is 10 to 20 folds less than a very big VAM install. One of the biggest game of the market which is combining several "sub games" (and an insane amount of pointless data) is COD which at some point reached 300+gigs, this is still 3 to 4 times less than a somewhat "low tier" big install of VAM at 1TB.

You are apparently organized and managing your VAM install, and I think it's a super great approach.
But you gotta understand that you are not considering the average joe. The random user discovering VAM will grab it, download half the hub and play. Unless they are motivated/passionnate enough about VAM. They will probably never know that you need to control your data in order to have a proper experience.

 

[Anand]

OK, in carefully parsing your words: here is what I distilled as being your position on this matter:

It is NOT possible for an un-modded VAM installation that has a multi-terabyte AddonPackages directory to load quickly.

Is this a correct synopsis? If yes, I concede. That has been my experience. You win.

OK, for the rest of you kiddies who want to learn how to "work around" this limitation of VAM [oooooh, mommy, mommy, that bad man said a naughty word], follow me to the back of the schoolhouse. And pssst, bring that pack of smokes with you when you come join me.

sFisherE and JayJayWon figured out some technical wizardry to allow us to have our cake and eat it too. They have modified VAM be like modern media players or Triple A games to access a huge library of content without pre-scanning all of its metadata. The downside is that the solutions are mods to the program. For some segment of users, installing mods and using them is hard. (i.e. relatively sparse documentation etc etc). But my point again: it is possible to have a smooth VAM experience while also having a large VAR collection. I and many other users are living testimony to this.

As to your point about needing to organize and control a large VAR collection - that is the EXACT opposite of what I actually do. Offloading gives me the freedom from managing VARs. This is because I know I can load the file whenever the fancy strikes me - as opposed to consciously placing VARs I plan on using into AddonPackages BEFORE VAM starts. Without limits on the number of VARs in the VAM directory, I don't have to sift through my collection and think about what I want to keep or throw away. I can keep all the VARs I want in a flat file structure within a subdirectory under /VAM/ (i.e. with zero management or control). And still have all of these files available to me while running VAM. I've done this and it works - until I run out of space on the 8TB SSD. (BTW a flat file structure is stupid, so I ended up with some organization of sub directories. But I spend almost no time in managing the collection).

[A slight walkback - I do curate a special subdirectory in AddonPackages that I ensure always gets loaded. This dir contains critical plugins for Default Session Plugins, and my must-have plugins such as your most excellent hazmhox.vamlive.6.var. Maintaining this "core" set of plugins that I use all the time is probably the only important management that I perform. And of course I do discard scenes/looks that don't appeal to me. I mean, I am not a mindless hoarder]

Anyway, feel free to continue to debate this and to disparage my using mods to "work around" VAM's limitations. But I'll respectfully bow out. Back in the day when I only had approximately 700GB of VARs, VAM took me 20 minutes to load, and every third or fourth scene load would result in a crash with heap errors. I was deep in the bowels of despair and ready to throw in the towel. But I found salvation through Var Browser and I just want to spread the Good News to all who would accept it.

;-) Tongue in cheeks, folks. Tongue in cheeks. Put down the pitchforks.

 

[VaMStaff]

We are closing this thread since VaM1 is in version freeze while we focus on VaM2 development.