Solved Too many heap sections

[Vesolye]

I have 64gb ram now, so there should be no problems theorically, but it keeps crashing my game.

Yesterday, it requires 10-13 scenes to crash, but today only one scene with timeline is enough to crash with this message.

Maybe it was caused by my enormous amount of addons? I cannot guess...

 

[DJ]

Typically, Vam has trouble with 16GB of Ram or less, and works fine with 32GB. And even that problem has been reduced with the "memory optimization" options in the more recent versions. 64GB should not be any problem at all.

So while your problem may well be memory related, it's very unlikely that the problem is that you don't have enough.


The number of addons in your collection wouldn't affect this, unless you're using them all in one scene!

Maybe you have a bad stick of Ram. If you're not overclocking, you could try turning the memory speed down below what it's rated for. Or, if you have a 2x32GB or 4x16GB kit, try running Vam with only one half of the memory installed. Then with the other half.

 

[Vesolye]

No. It was nothing with my RAMs.
No. It was nothing with my GPU and power setting.
And no. It was nothinf with my PBOs, because I've been turning it on for 3 years, even before I install this.
Can a single, certain plugin make this situation?

BTW, I just checked out that the heap size of the game was 15.48gb at the startup. Maybe it's with this...?

 

[jimmyxpop]

I am running into the same thing. Not sure what it is as well. I have a top of line comp, with I-9, 64 mem, and a 3090. Mostly happens when I am using the timeline plugin with long animations. Would love to find a fix to this, especially since I am not maxing out my system.

 

[Vesolye]

After spending a day for this crapshit error message, I gained one fact: it's depending on my addon number. Yes, it was. I have 28000 addon and reduce some of them, it works. But it could be not the solution of the problem because it STILL bothers my play.

I don't know it's because of my tons of addon or certain one addon, the heap size of the game increased as 15GB at the startup. So it makes the crash.

Ram problem? No. It has 11GB of plenty spaces. Virtual Memory? No. It also crashes when my disk has 113GB of free space to page. Bad stick? No. Both half of my sticks got the same crash message.

Also the memory option of the game didn't make any great change. Rather, it only caused problems.

I mean geez, you guys really need to notice about this oldass unity garbage collector shit. I thought it's garbage collector but it's just garbage itself.

 

[jimmyxpop]

Yeah, I am with you on that. I read your post and decided to download the VAR Manager program from the hub looking for a solution. I have 27,062 addons. I am now using VAR Manager to move the vars I rarely use to a back-up folder and then restore them if needed. This is definitely not a ideal solution, but I was starting to get heap errors constantly every time I ran the program. It also has features like move old versions of vars and a Duplicate items resolver which brought me down to around 25,000 Vars now total. Its the best I know of besides manually moving your Vars in and out the addon folder.

 

[hazmhox]

Honestly, I'm gonna answer the thing you're not gonna like: I would bet that meshed has never stressed tested VAM with 25K vars.
Even tho the game does not prevent you to hoard that much content, I'm convinced that the overall code is not perfectly made to handle that much data. And the fact that reducing the amount of vars fixes the problem does prove that this is the issue.

You have to understand that beyond a handful of devs around here that know exactly how to optimize their plugins/content, the rest does not. You will end up having content that is not perfectly optimized and could lead to memory issues or potential crashes.

So the only solution would be to have several builds with content spread across these.

Unless meshed optimize the game engine to be able to handle that much content, AND that all creators would release content that is optimized af... it's likely that the problem will never go away. ( And 1.x is not gonna receive that kind of heavy change anymore )

 

[hazmhox]

BTW, if you're not only a "player"... but you are creating content also (and crashing through content creation)... do like pretty much all advanced creators around here: have a custom duplicated build of VAM with the minimum vars you need.

And another one clean to test out your content.

 

[le_hibou]

So I’ve been struggling with this for a while and I think I found a solution.

Short version: remove VARs with too many morphs, namely those by Universens and noheadnoleg.

Longer version:

I wasn’t able to open scenes with more than 2 people in it, without getting the dreaded "Too Many Heap Sections". I’ve read that it was not about not having enough physical RAM, which I can confirm from my experience. I do have LOTS of VARs and I was pretty sure that was the reason, but I was not sure why.

I found a very interesting thread about VAM performance where MeshedVR explained that VAM performance wasn’t directly related to the number of VARs, but it is related to the number of morphs, including the ones in VARs and all duplicates. He recommends "disabling or removing VAR files that have a lot of morphs and you are not regularly using" ( https://hub.virtamate.com/threads/benchmark-clear-install-vs-many-vars.37256/post-103453 )
I assumed that if all those morphs needed to be applied, they probably had to be loaded in memory too, so this may be the cause of my problem.

So I put together a small Python script (I can share it if people are interested) to detect VARs that are bloated with morphs.
I noticed that some vars contain over 2000 morphs. MeshedVR recommends having less than 10000 morphs overall, so you see how quickly you gan get there with only a few of those.
It makes sense for some VARs to have lots of morphs, like AshAuryn’s expression stuff, or things that are clearly morph collections. But for some, it absolutely does not. A single scene should not contain 2000 morphs. The worst offender is Universens. Almost each of his VARs does. I guess, it’s not surprising when his VARs are often larger than 1Gb… Noheadnoleg seems to be a close second. I proceeded to remove most of these VARs, and it worked!
Now I’ve been able to open 3-person scenes that I had never been able to open before. And it seems to have helped a lot with FPS too (but I haven’t measured anything).
I’d also recommend removing older versions of VamX, which contains lots of morphs too (but that’s expected).

I hope this helps.

Edit: script added here: https://hub.virtamate.com/threads/too-many-heap-sections.38525/post-108718

 

[jimmyxpop]

Wholly moly Le_hibou!!! Our HERO! Yes! I would love for you to share your Python script. I am experiencing the same issue when loading multiple characters too. Just yesterday I tried loading an older Vecterror scene with two models that I have loaded easily many times in the past, and now was getting heap errors constantly. It was maddening, especially since it worked in the past, and I have been going back through my vars and deleting duplicates and moving rarely used scenes out the folder to reduce the var count. Got them down to 20,000 from 25,000, but still get that dreaded heap error. I also have been a patron of both Universen and noheadnoleg so I have a huge Var library of both their works. Now it makes more sense after reading your post. Hopefully with the script I can identify these bloated morph vars and move them in and out of my add-ons library, vs. the mass var purge I’ve been doing this week to hopefully reduce heap errors.

 

[Endiness2k]

Ah nice, this seems to be working well so far me too.

 

[le_hibou]

Wholly moly Le_hibou!!! Our HERO! Yes! I would love for you to share your Python script. I am experiencing the same issue when loading multiple characters too. Just yesterday I tried loading an older Vecterror scene with two models that I have loaded easily many times in the past, and now was getting heap errors constantly. It was maddening, especially since it worked in the past, and I have been going back through my vars and deleting duplicates and moving rarely used scenes out the folder to reduce the var count. Got them down to 20,000 from 25,000, but still get that dreaded heap error. I also have been a patron of both Universen and noheadnoleg so I have a huge Var library of both their works. Now it makes more sense after reading your post. Hopefully with the script I can identify these bloated morph vars and move them in and out of my add-ons library, vs. the mass var purge I’ve been doing this week to hopefully reduce heap errors.

Thanks for your feedback, I'm glad it helps, I was really happy when I figured that out.

Here's my script, it's my first time writing Python, so bear with me ;-)

You'll have to rename it from morphbloatscanner.txt to morphbloatscanner.py, because the forum wouldn't let me upload a file with the .py extension.

To execute it, you can just run it from your AddonPackages folder (or any folder with VARs) like this:
python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py
Of course, you'll need python.

If you want to customize the max number of morphs by VAR allowed you can use "-m".
If you want to execute the script from anywhere and specify the path to a VAR folder you can use "-p"
Here's an example with both:
python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py -m 900 -p C:\<ANOTHER_PATH>\virt-a-mate\AddonPackages

I've only tested with Windows PowerShell

Also, in the script, there's a whitelist of VAR files that are ignored, that was mostly for my use, but that you can customize by editing he script.

 

[jimmyxpop]

Thanks for your feedback, I'm glad it helps, I was really happy when I figured that out.

Here's my script, it's my first time writing Python, so bear with me ;-)

You'll have to rename it from morphbloatscanner.txt to morphbloatscanner.py, because the forum wouldn't let me upload a file with the .py extension.

To execute it, you can just run it from your AddonPackages folder (or any folder with VARs) like this:
python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py
Of course, you'll need python.

If you want to customize the max number of morphs by VAR allowed you can use "-m".
If you want to execute the script from anywhere and specify the path to a VAR folder you can use "-p"
Here's an example with both:
python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py -m 900 -p C:\<ANOTHER_PATH>\virt-a-mate\AddonPackages

I've only tested with Windows PowerShell

Also, in the script, there's a whitelist of VAR files that are ignored, that was mostly for my use, but that you can customize by editing he script.

Thank you! your script and directions were spot on. I just scanned my addon vars folder for packages with 1,000 or more, and now going through the filtered list and moving them to my backup folder. The script also helps me see that I had some duplicate Var packages that have 1,000+ morphs in them. So even after moving unused vars all week, I still had my VAM session loading thousands of duplicate morphs as well. Using your script and VAR manager together will make it easy to just move the VARs with lots of morphs to backup, and just retrieve the backup vars when I want. Kudos to you!

 

[lalaland3321]

Typically, Vam has trouble with 16GB of Ram or less, and works fine with 32GB. And even that problem has been reduced with the "memory optimization" options in the more recent versions. 64GB should not be any problem at all.

So while your problem may well be memory related, it's very unlikely that the problem is that you don't have enough.


The number of addons in your collection wouldn't affect this, unless you're using them all in one scene!

Maybe you have a bad stick of Ram. If you're not overclocking, you could try turning the memory speed down below what it's rated for. Or, if you have a 2x32GB or 4x16GB kit, try running Vam with only one half of the memory installed. Then with the other half.

i have 128gig of ram and still have this problem.

 

[jimmyxpop]

i have 128gig of ram and still have this problem.

It's not a memory thing. Its the amount of morphs in your add-ons type issue.

Go download Le_hibou 's script in the above post. Run the script and scan your add-ons directory. Then remove the Var files that have lots of morphs out your add-ons folder to a backup folder you create to store them until you want to use them. I started with a scan and display any Var's that has over 1,000 morphs (follow instructions in the post above to see how). Then I did one at 900 morphs (and so on). Since you are moving the biggest offending morphs out the add-ons folder, it did not require me to move too many VARs to see a dramatic improvement to where I am now not receiving any heap errors so far. Everything runs soother and can load large scenes with multiple characters again. If this works for you give Le_hibou's script a thumbs up.

Lastly, pro-tip, if your situation was like mine: Note: duplicate vars and their morphs count toward the total morphs in the add-on folder that seems to effects heap errors. Since, I try to keep my scenes and looks vars in a folder with the creators name, I tend to have many duplicate dependencies scattered throughout my add-ons folder. You can use the script to scan and show every var with at least 1 morph. Then highlight and copy the results and paste the results in the first column of excel and highlight all duplicates. Lastly copy and paste the dups vars name in the file manager search and delete all dups but one. There were many dup files like poses, titty majic, make-up vars that were small by there file size, but had 600 or more morphs in them.

 

[Vesolye]

Thanks for your feedback, I'm glad it helps, I was really happy when I figured that out.

Here's my script, it's my first time writing Python, so bear with me ;-)

You'll have to rename it from morphbloatscanner.txt to morphbloatscanner.py, because the forum wouldn't let me upload a file with the .py extension.

To execute it, you can just run it from your AddonPackages folder (or any folder with VARs) like this:
python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py
Of course, you'll need python.

If you want to customize the max number of morphs by VAR allowed you can use "-m".
If you want to execute the script from anywhere and specify the path to a VAR folder you can use "-p"
Here's an example with both:
python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py -m 900 -p C:\<ANOTHER_PATH>\virt-a-mate\AddonPackages

I've only tested with Windows PowerShell

Also, in the script, there's a whitelist of VAR files that are ignored, that was mostly for my use, but that you can customize by editing he script.

Geez, thanks! I should give it a try at home! I've been total frustrated that I couldn't solve this, i was thinking about give up VAM! You are my hero of a day. Thank you again!

 

[hazmhox]

Geez, thanks! I should give it a try at home! I've been total frustrated that I couldn't solve this, i was thinking about give up VAM! You are my hero of a day. Thank you again!

Thanks to you for downvoting answers that are actually explaining what to do... and upvoting a script that does exactly what I'm suggesting : D
I think I'm gonna stop answering on the forums, people are asses.

 

[Vesolye]

Dude, it's my right to upvote or downvote the answer what acually helped me or not. Or why does it exist here with that 'sort by votes'?

 

[hazmhox]

Dude, it's my right to upvote or downvote the answer what acually helped me or not. Or why does it exist here with that 'sort by votes'?

Ho yeah dude, it's your "right".

The thing is, there is valid information in my answers, which also covers either a user/player approach OR a creator approach. Downvoting an answer with valid information makes users think it can be dismissed. You could have upvoted only the thing that is helping you a lot and stopped there to avoid confusing the potential future readers of this thread.

If your goal was simply to "sort", then you should have downvoted pretty much every single "out-of-the-subject answer", kept the useful information with no vote, and upvoted the most important ones.

 

[Vesolye]

Ho yeah dude, it's your "right".

The thing is, there is valid information in my answers, which also covers either a user/player approach OR a creator approach. Downvoting an answer with valid information makes users think it can be dismissed. You could have upvoted only the thing that is helping you a lot and stopped there to avoid confusing the potential future readers of this thread.

If your goal was simply to "sort", then you should have downvoted pretty much every single "out-of-the-subject answer", kept the useful information with no vote, and upvoted the most important ones.

Took a shower a minute ago and I thought about it. Yeah, you're right. I'll apologize. Maybe I was too hotheaded because of all the annoyance by this infinite-seemed chains of error with 'real-life shitstacks' those recently occured to me.
Sorry for my rudeness, with my heart. I just voted and liked your answers.

 

[Fantano]

So I’ve been struggling with this for a while and I think I found a solution.

Short version: remove VARs with too many morphs, namely those by Universens and noheadnoleg.

Longer version:

I wasn’t able to open scenes with more than 2 people in it, without getting the dreaded "Too Many Heap Sections". I’ve read that it was not about not having enough physical RAM, which I can confirm from my experience. I do have LOTS of VARs and I was pretty sure that was the reason, but I was not sure why.

I found a very interesting thread about VAM performance where MeshedVR explained that VAM performance wasn’t directly related to the number of VARs, but it is related to the number of morphs, including the ones in VARs and all duplicates. He recommends "disabling or removing VAR files that have a lot of morphs and you are not regularly using" ( https://hub.virtamate.com/threads/benchmark-clear-install-vs-many-vars.37256/post-103453 )
I assumed that if all those morphs needed to be applied, they probably had to be loaded in memory too, so this may be the cause of my problem.

So I put together a small Python script (I can share it if people are interested) to detect VARs that are bloated with morphs.
I noticed that some vars contain over 2000 morphs. MeshedVR recommends having less than 10000 morphs overall, so you see how quickly you gan get there with only a few of those.
It makes sense for some VARs to have lots of morphs, like AshAuryn’s expression stuff, or things that are clearly morph collections. But for some, it absolutely does not. A single scene should not contain 2000 morphs. The worst offender is Universens. Almost each of his VARs does. I guess, it’s not surprising when his VARs are often larger than 1Gb… Noheadnoleg seems to be a close second. I proceeded to remove most of these VARs, and it worked!
Now I’ve been able to open 3-person scenes that I had never been able to open before. And it seems to have helped a lot with FPS too (but I haven’t measured anything).
I’d also recommend removing older versions of VamX, which contains lots of morphs too (but that’s expected).

I hope this helps.

Edit: script added here: https://hub.virtamate.com/threads/too-many-heap-sections.38525/post-108718

Thanks! I was also suffered by this problem. Found all over the discord but I couldn't, finally it's here.

 

[Case]

Ho yeah dude, it's your "right".

The thing is, there is valid information in my answers, which also covers either a user/player approach OR a creator approach. Downvoting an answer with valid information makes users think it can be dismissed. You could have upvoted only the thing that is helping you a lot and stopped there to avoid confusing the potential future readers of this thread.

If your goal was simply to "sort", then you should have downvoted pretty much every single "out-of-the-subject answer", kept the useful information with no vote, and upvoted the most important ones.

It's maybe not my place to say this, but I have a strong feeling it needs to be said, and that we, as a community aren't saying it often enough:
This community owes all its plugin devs an immense debt of gratitude. MeshedVR built VaM, true - but it's you guys who've turned smth more akin to "an exiting tech demo" into a sandbox that can be used without exceeding the user's frustration tolerance (too often. Ok, not more than twice a day ...).

And even amongst that group, there's five or so people that are indispensable - Acidbubbles, for example. And you.

Your contributions and expertise are greatly appreaciated, and loosing you would be felt by the whole community. We've already lost MacGruber - please stay, and stay active.

 

[Case]

Thanks! I was also suffered by this problem. Found all over the discord but I couldn't, finally it's here.

Same here. Recently upgraded from 32GB to 64Gb. Updated BIOS (Zen7 5800x, ASUS-TUF 570 Gaming plus, 4x8Gb Ram @ 3600Hz - not exactly new hardware that still has teething problems, but enough to run VaM). I don't even HAVE memory OC settings I could revert - and I won't consider running below vendor-recommended XMP settings (Yes that was actually a suggestion on the Discord a couple months ago ...).

No other app but VaM has ever said a peep about memory to me.

P.S.: I've long ago removed UniversenS scenes (no offense to UniversenS, but ... it's 2023. The grace-period for being that bad at packaging expired two years ago ...). Same for NoheadNolegs.

Unfortunately, some of the most common morphpacks clock in at close to 1.000 morphs - and there's considerable overlap between some of them (Iiirc, the "Morph.Morphs_Mega_Pack" contains large parts of the "import_reloaded" pack(s) - and both are referenced far too often to cull either of them).

 

[Case]

Thanks for your feedback, I'm glad it helps, I was really happy when I figured that out.


Here's my script, it's my first time writing Python, so bear with me ;-)


You'll have to rename it from morphbloatscanner.txt to morphbloatscanner.py, because the forum wouldn't let me upload a file with the .py extension.


To execute it, you can just run it from your AddonPackages folder (or any folder with VARs) like this:

python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py

Of course, you'll need python.


If you want to customize the max number of morphs by VAR allowed you can use "-m".

If you want to execute the script from anywhere and specify the path to a VAR folder you can use "-p"

Here's an example with both:

python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py -m 900 -p C:\<ANOTHER_PATH>\virt-a-mate\AddonPackages


I've only tested with Windows PowerShell


Also, in the script, there's a whitelist of VAR files that are ignored, that was mostly for my use, but that you can customize by editing he script.

Kudos on the find that morph-dupes also count. That being said, some "nagging":
* Your script "double.-counts" morphs - it counts both the .vmi and the .vmb for each morph as separate morphs. They're not. Not really important (If x>y, then 2x > 2y, too), but still.
* There's a limit to how usefull this can be: The problem with the .var system isn't "getting rid of bad vars" per se (otherwise, we'd solved it years ago) - it's the references to bad .vars.

For example, look at your own whitelist - you've whitelisted both the "V2.___UniversalMorphCollection___.100" and "vecterror._MorphCollection_.26". If you take a closer look, you'll see that "vecterror._MorphCollection_.26" is completely duplicated within "V2.___UniversalMorphCollection___.100" - unsuprisingly, since the latter was meant as a replacement for all prior packs. So why are both in your whitelist? (SPOILER: Bcs vecterror is smart, but his Vam's packagemanager is not ... eg. grab his Lana-look. Both packs are referenced in the .vap ... Why? I've actually tried editing the references manually & repackaging that look - Vam just adds the references again. Probably bcs they're both listed in my cache.)

Same applies for other morphpacks - the "import_reloaded" pack(s) are duplicated in another common package ... etcetc

 

[virtamater]

I'm not sure if this works or makes any difference on your system.. I'm able to add up to 5 girls and 1 guy no problem if no one has any Spec, gloss or Norm textures on.
I'm not sure what the deal with the extra maps is on my system and I've even created my own gloss, spec and norm maps all as jpgs. Even those files that are 90kbs crash my scenes the same way the pngs do... If anyone here happens to have a way to fix that I'd be interested..

I have an older EyeBuyPower with 32 gigs of ram. My graphics cards a gtx 1060ti. Its not amazing but it still works awesome for me. I set the render scale to 85. I cant really see a difference in appearance from 100, but it sure loads up faster. I'm not sure what the deal with the shaders is from my end and why those other maps are so buggy for me.