Solved Too many heap sections

[Vesolye]

I have 64gb ram now, so there should be no problems theorically, but it keeps crashing my game.

Yesterday, it requires 10-13 scenes to crash, but today only one scene with timeline is enough to crash with this message.

Maybe it was caused by my enormous amount of addons? I cannot guess...

 

[hazmhox]

Why doesn't Meshed just release the V1 code to the community? There's a FAR better chance for us to fix it and make it amazing, than there is for V2 to be a fully fleshed out savior. Every project that becomes open-source inevitably becomes amazing, and bugs get fixed!
Gotta do it for the good of the community.

Why don't you ask meshed? Are you ready to dive yourself into the code and fix those bugs if it goes open source? And I wouldn't even put that under "bugs", that's optimization. And scaling the optimization for situations like this is (to me) kind or proposterous.

Open source is not something you decide one morning out of the blue. VAM has several licenced assets inside its core, meshed couldn't release it. Since the whole lighting system is based on that, you would need to reverse engineer the shader and build something that would work the same way.

And beyond that... not everyone wants to do open source. That's a business decision.

And to be honest, the last part of my message was a polite way of saying: "No one is ever gonna optimize a game for the three dudes that have a 3TB folder of stuff they never use".

 

[VRmatt]

Thanks for the reply, I did just throw that idea out there willy-nilly, but I am aware there's a huge amount of money and business property involved. Call it devils advocate, or you know just getting some ideas out in the open.
Anyway - I wouldn't want to take on a complete rewrite, but I definitely would like some visibility into the backend code, to find out where this GC error is stemming from.
As far as I can tell, there's multiple threads with people just randomly guessing at what the solution is. Which I definitely am thankful for, because I've tried every solution that was mentioned.
I'd definitely like to know what is happening under the hood, that's all I'm saying
I absolutely love VAM and I use the shit out of it, obviously.

 

[VRmatt]

I rolled back the v13 memory fix, and installed v12. It works WAY better and hasn't crashed yet.

 

[everlaster]

Thanks for your feedback, I'm glad it helps, I was really happy when I figured that out.

Here's my script, it's my first time writing Python, so bear with me ;-)

You'll have to rename it from morphbloatscanner.txt to morphbloatscanner.py, because the forum wouldn't let me upload a file with the .py extension.

To execute it, you can just run it from your AddonPackages folder (or any folder with VARs) like this:
python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py
Of course, you'll need python.

If you want to customize the max number of morphs by VAR allowed you can use "-m".
If you want to execute the script from anywhere and specify the path to a VAR folder you can use "-p"
Here's an example with both:
python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py -m 900 -p C:\<ANOTHER_PATH>\virt-a-mate\AddonPackages

I've only tested with Windows PowerShell

Also, in the script, there's a whitelist of VAR files that are ignored, that was mostly for my use, but that you can customize by editing he script.

With the power of GPT, I updated this script to sort the results by morph count descending, and print out in a tabular format.

As above, the file extension needs to be changed to .py.

Usage (Command line)

E:
cd VaM\AddonPackages
python .\morphbloatscanner-1.1.0.py --min 1 --path .

Usage (Powershell)

cd E:\VaM\AddonPackages
python .\morphbloatscanner-1.1.0.py --min 1 --path .

Output example

 

[atani]

@everlaster

In the example parameters you need to change -min to either -m or --min.

And one request if easy to include, could you include a morphs sum at the end of the results?

Thank you for sharing, @everlaster and @le_hibou , it's really useful. Perhaps this could be a resource of its own in the Other category?

 

[everlaster]

@everlaster

In the example parameters you need to change -min to either -m or --min.

Fixed

And one request if easy to include, could you include a morphs sum at the end of the results?

Not sure how useful a total would be, since VAM will generally only load morphs from the latest version of a package. You'd just get tons of morphs in the total count that aren't actually loaded in VAM. Easier to check the real number of loaded morphs within VAM at any given time:

This number * 12 which is the number of morphs per page, so 253*12 = 3036, for example.

Filter by Show: Custom Package to get a result corresponding to morphs loaded from var packages.

Thank you for sharing, @everlaster and @le_hibou , it's really useful. Perhaps this could be a resource of its own in the Other category?

Maybe, or a Guide for dealing with this heap sections issue, with this script as an attachment. I'll leave that to @le_hibou if they want to do it - feel free to use my edited script.

 

[atani]

In my case I wouldn't see the older versions as I have none because I'm a freak in maintaining VaM . But I see your point, would be useless to everyone else unless the count would ignore the older versions too.

The option of the morphs manager only shows the preloaded morphs, wouldn't show the others not set to preload. And somewhere was mentioned a correction that even non-preloaded morphs still influence VaM performance, so that's why I was looking to have a full count.
But it's something I can do too i think.

 

[Makimax]

The creators should read the messages because more than a year after the owl's message, Universens continues to make vars with thousands of morphs in each file and with 1 file per sex position. This completely slowed down my system. I deleted them. there are also creators like Wolverine who seem to create their vars poorly and have a completely crazy and useless list of dependencies.

 

[justkiddingbutnot]

The creators should read the messages because more than a year after the owl's message, Universens continues to make vars with thousands of morphs in each file and with 1 file per sex position. This completely slowed down my system. I deleted them. there are also creators like Wolverine who seem to create their vars poorly and have a completely crazy and useless list of dependencies.

We need an app that scans and consolidates all morphs and removes duplicates.

 

[everlaster]

We need an app that scans and consolidates all morphs and removes duplicates.

You'd need to extract morphs from all packages to your actual Custom/Morphs folder to do that. This would prevent VAM from loading morphs on demand, all morphs would be loaded at all times. Even after removing all duplicates, you would probably just increase the number of morphs you have loaded at any given time.

For the most part, you can avoid too many morphs from loading by ensuring Preload Morphs is unchecked in the package manager for any package that has a lot of morphs.

 

[justkiddingbutnot]

You'd need to extract morphs from all packages to your actual Custom/Morphs folder to do that. This would prevent VAM from loading morphs on demand, all morphs would be loaded at all times. Even after removing all duplicates, you would probably just increase the number of morphs you have loaded at any given time.

For the most part, you can avoid too many morphs from loading by ensuring Preload Morphs is unchecked in the package manager for any package that has a lot of morphs.

Would it be possible to programmatically do this with symlinks? Feels like there has to be a way around morph bloat. You’re imminently more talented than most, so if you say there isn’t, there isn’t.

 

[everlaster]

Would it be possible to programmatically do this with symlinks? Feels like there has to be a way around morph bloat. You’re imminently more talented than most, so if you say there isn’t, there isn’t.

To do what with symlinks?

The way around morph bloat is to not download packages that contain too many morphs, and set Preload Morphs to false on most packages containing morphs, so that morphs from that package are only loaded when a scene or preset references them. That way, even if you have 1000 duplicates of a morph, only a few of them will be loaded at any one time since they're all going to be referenced/used by different scenes.

 

[virtamater]

To do what with symlinks?

The way around morph bloat is to not download packages that contain too many morphs, and set Preload Morphs to false on most packages containing morphs, so that morphs from that package are only loaded when a scene or preset references them. That way, even if you have 1000 duplicates of a morph, only a few of them will be loaded at any one time since they're all going to be referenced/used by different scenes.

Is it documented anywhere how to set preloaded Morphs to false? I don't remember seeing that option

 

[virtamater]

* Pretty sure that your issue is smth different -sounds rather like something to do with the rendering pipeline or GPU drivers than a memory-related issue (it's pretty much accepted wisdom by now that the heap sections CTD is memory-related) - also pretty sure that you'd like to have @hazmhox advice.

* Five ppl is ... extremely ambitious, for ppl with hardware three geneerations ahead of yours. Unless you're into slideshow's that is.

However, the crashing sounds ... ominous.

I've no advice I could give you but this: Most likely, asking for help in this thread isn't the best best way for you to optimize the chances of a helpful response - you want to attract the attention of ppl who have either encountered your problem before, or those who have a very good understanding of Vam's rendering pipeline (there are a couple of those active here)

TL;DR - Best thing you can do IMO is create your own thread.

Since this post, I have updated my system considerably the extra texture files isnt an issue anymore.
My system is poky with some ffm scenes from Kittymocap, but it seems to be more of an issue with my headset in VR as everything runs crazy fast w no issues on Desktop. As for Morphs, I have a lot of extraneous plug ins on my girls. so if theres some lag, its due to that.. but its only in VR if theres some pokyness its the Kitty stuff.

 

[SlimerJSpud]

Is it documented anywhere how to set preloaded Morphs to false? I don't remember seeing that option

It's in Package Manager. You select the package, or step through them, then go to the User Prefs tab and uncheck the box for that. Any package that has "morphs" in the name should have it checked. Ordinary looks generally do not need it.

 

[Makimax]

It's in Package Manager. You select the package, or step through them, then go to the User Prefs tab and uncheck the box for that. Any package that has "morphs" in the name should have it checked. Ordinary looks generally do not need it.

Don't know "Vam thin twin from VamLightYear" and "BrowserAssist". Are they Good solutions to speed up when we have a lot of vars (morphs) ?
Is there a tip to keep only Last version of a var and delete old versions. I keep all versions otherwise Manager ask me to download oldiers...

 

[Vamuser123]

About this whole setting Preload Morphs to false thing, i was this close to going through ~4500 vars one by one in package manager to make them all false (if i understand what this process entails), but i asked ChatGPT for directions how to do so and it told me it was risky, is this true? This is what it said, i asked it to tldr it as well for the sake of this post:


If you turn off "Preload Morphs," VaM won’t load all morphs into memory upfront. Instead, it loads them as needed, which saves memory but can cause problems:

1. Path Issues: The game might struggle to find morphs if the paths aren't set up correctly.
2. Performance Hit: Loading morphs on-the-fly can lead to lag or stuttering.
3. Inconsistent Behavior: Morphs might not apply or display consistently.

It's usually better to keep "Preload Morphs" on to avoid these issues.


Can someone please confirm that this is just the normal ChatGPT making a slight fib situation and that I am safe to go set all my Preload Morphs to false with no risk?

 

[everlaster]

About this whole setting Preload Morphs to false thing, i was this close to going through ~4500 vars one by one in package manager to make them all false (if i understand what this process entails), but i asked ChatGPT for directions how to do so and it told me it was risky, is this true? This is what it said, i asked it to tldr it as well for the sake of this post:


If you turn off "Preload Morphs," VaM won’t load all morphs into memory upfront. Instead, it loads them as needed, which saves memory but can cause problems:

1. Path Issues: The game might struggle to find morphs if the paths aren't set up correctly.
2. Performance Hit: Loading morphs on-the-fly can lead to lag or stuttering.
3. Inconsistent Behavior: Morphs might not apply or display consistently.

It's usually better to keep "Preload Morphs" on to avoid these issues.


Can someone please confirm that this is just the normal ChatGPT making a slight fib situation and that I am safe to go set all my Preload Morphs to false with no risk?

All three points are complete BS, these are not problems that occur if you turn off preload morphs. Well, maybe in theory loading morphs on the fly can "lead to lag or stuttering", but I think that's an inaccurate way of putting it. I've never noticed any lag from loading morphs on demand, but technically there has to be some file reads and processing that occur. Generally morphs that get loaded on demand during a scene as opposed to while loading the scene itself are referenced in appearance presets, and the loading of skin, clothing etc. is going to cause much more "lag or stuttering" than the part where morphs are loaded.

Also, while turning off preload morphs proably does save memory a little bit, the main reason to do that is due to the performance hit of having a lot of morphs loaded, and the "too many heap sections" error.

going through ~4500 vars one by one

Here's where ChatGPT might be able to help you. Ask it to write a script that
1. loops over all .var files in AddonPackages and its subfolders
2. ensures each .var has a corresponding AddonPackagesUserPrefs/*.prefs file (not including .var extension, not nested in a subfolder)
3. in each .prefs file, which is in JSON format, ensure there is a block customOptions with the key preloadMorphs set to false

   "customOptions" : {
      "preloadMorphs" : "false"
   }

 

[Vamuser123]

All three points are complete BS, these are not problems that occur if you turn off preload morphs. Well, maybe in theory loading morphs on the fly can "lead to lag or stuttering", but I think that's an inaccurate way of putting it. I've never noticed any lag from loading morphs on demand, but technically there has to be some file reads and processing that occur. Generally morphs that get loaded on demand during a scene as opposed to while loading the scene itself are referenced in appearance presets, and the loading of skin, clothing etc. is going to cause much more "lag or stuttering" than the part where morphs are loaded.

Also, while turning off preload morphs proably does save memory a little bit, the main reason to do that is due to the performance hit of having a lot of morphs loaded, and the "too many heap sections" error.


Here's where ChatGPT might be able to help you. Ask it to write a script that
1. loops over all .var files in AddonPackages and its subfolders
2. ensures each .var has a corresponding AddonPackagesUserPrefs/*.prefs file (not including .var extension, not nested in a subfolder)
3. in each .prefs file, which is in JSON format, ensure there is a block customOptions with the key preloadMorphs set to false

   "customOptions" : {
      "preloadMorphs" : "false"
   }

First off thanks for all the mods you make, I minimum add Naturalis to every single scene immediately.
That doesnt surprise me at all, i just spent(wasted) $500 to have my meta 2 with PCVR cable link + power bank. ChatGPT spoke with such conviction while sitting on a throne of lies. I will never trust it again haha.
I just wanted to make sure and was kind hoping it even was the case so i didnt have to sift through 4500 vars LOL. TBH its time i got rid of uneeded vars anyways as well. I will try this suggestion as well for the script to make it easy, tyvm, much appreciated.

 

[SlimerJSpud]

The issue with wiping out all preloads is that there are many very useful morph packs on the hub that are essential to a lot of scenes. You really do want those on. The problem is, you might not know which is which by the filename. Just today, there's a legit morph pack called "Eyelids_look" that isn't a look, it's a morph pack that would need to be preloaded. My simple filename test of "Is it a look or a morph pack" would have failed on that one.

The issue with deleting all but the last version of all vars is that plugins are generally referenced in scenes by the exact version number. That's why you keep getting missing references and the tool wants to download them again. There's no good way to fix that either, as some plugin creators may change the options between versions. You can use this Python app to scan for things that are not being called by anything else:

Other - VAR Inspector

VAR INSPECTOR CC-BY-NC-ND What is it: Var Inspector is a desktop application written in Python which will scan the save files and presets in your var and tell you where the dependencies are located How does it work: Var Inspector creates a...

hub.virtamate.com

One issue with this inspector is that it will almost always tell you the latest version of a plugin is not needed, because you let the tool download the update, but you don't have any scenes that call it yet!

 

[Vamuser123]

The issue with wiping out all preloads is that there are many very useful morph packs on the hub that are essential to a lot of scenes. You really do want those on. The problem is, you might not know which is which by the filename. Just today, there's a legit morph pack called "Eyelids_look" that isn't a look, it's a morph pack that would need to be preloaded. My simple filename test of "Is it a look or a morph pack" would have failed on that one.

The issue with deleting all but the last version of all vars is that plugins are generally referenced in scenes by the exact version number. That's why you keep getting missing references and the tool wants to download them again. There's no good way to fix that either, as some plugin creators may change the options between versions. You can use this Python app to scan for things that are not being called by anything else:

Other - VAR Inspector

VAR INSPECTOR CC-BY-NC-ND What is it: Var Inspector is a desktop application written in Python which will scan the save files and presets in your var and tell you where the dependencies are located How does it work: Var Inspector creates a...

hub.virtamate.com

One issue with this inspector is that it will almost always tell you the latest version of a plugin is not needed, because you let the tool download the update, but you don't have any scenes that call it yet!

Actually for once ChatGPT gave me information that worked in practice instead of in Chat's head. It created a script for me that took 3 seconds to make a user pref file for every of my ~4500 vars (went up from 70 user pref files since its a pretty new file install of VaM).
I tested a PC killer scene I made quick with 9 women eating each others butts (lol), dropped 1 FPS from my usual 16FPS, so I just called it there after doing a quick check to realize that most of my vars already had Preload Morphs set to false anyway.
So i basically just created 4500 text files bringing my FPS down 1 from what i could tell. I mean it seemed from a quick check like only 1% (if that) of my ~4500 vars were set to Preload Morphs in the first place. My focus is now on trimming down my var files as i misunderstoood that all vars were already set to true for Preload Morphs. I feel like my 4500 vars is doing much more damage to my performance than the few preload morphs set to true. 4500 is crazy.

 

[catbird01]

superb work thank you

Example output:

 

[SlimerJSpud]

The items above that say "has many morphs" are the ones you really do want preload enabled for.

 

[Vamuser123]

While I can't speak on the matter of "too many heap sections" error and the result of reducing preload morphs, I wanted to share the information I found through a bit of testing lately.

I set about 200 pages ofr my 440 morph pages to preload morphs off, going down to about 240 pages of female morphs and 210 pages of male morphs. This was a large enough differential I assumed, to test the results of having so many less preload morphs enabled. I did not see any change at all, not even slightly.

That was after I actually used a script to create the user pref files that reference Preload Morphs being set to false, which actually brought me down 1FPS because of adding 4500 user pref files (as mentioned a couple posts above).

So, again, while I definitely can't speak on the too many heap sections problem I just wanted to share for anyone considering diving into the Preload Morphs problem in an attempt to get more FPS, that I personally found zero gain in FPS doing so.

This might be as I've noticed in different themes that there is just so much going on that the preload morph thing, although it was a big change, it didnt show it. This is seen in examples like, when I reach close to max performance output of my PC and start hitting bottlenecks, my FPS will remain the exact same no matter what I do. I can remove a 3 light setep and have 0 lights. Or move 1 cm away from a group of 9 person atoms to 1 mile away from them with literally 0 change in FPS.

I think this is similar to whats happening in that case, that it might show a FPS increase in certain cases but when you have a full scene with plugins and custom person atom/textures, etc. etc., the change doesnt even show up when having less preload morphs.