Solved Too many heap sections

[Vesolye]

I have 64gb ram now, so there should be no problems theorically, but it keeps crashing my game.

Yesterday, it requires 10-13 scenes to crash, but today only one scene with timeline is enough to crash with this message.

Maybe it was caused by my enormous amount of addons? I cannot guess...

 

[Fantano]

Same here. Recently upgraded from 32GB to 64Gb. Updated BIOS (Zen7 5800x, ASUS-TUF 570 Gaming plus, 4x8Gb Ram @ 3600Hz - not exactly new hardware that still has teething problems, but enough to run VaM). I don't even HAVE memory OC settings I could revert - and I won't consider running below vendor-recommended XMP settings (Yes that was actually a suggestion on the Discord a couple months ago ...).

No other app but VaM has ever said a peep about memory to me.

P.S.: I've long ago removed UniversenS scenes (no offense to UniversenS, but ... it's 2023. The grace-period for being that bad at packaging expired two years ago ...). Same for NoheadNolegs.

Unfortunately, some of the most common morphpacks clock in at close to 1.000 morphs - and there's considerable overlap between some of them (Iiirc, the "Morph.Morphs_Mega_Pack" contains large parts of the "import_reloaded" pack(s) - and both are referenced far too often to cull either of them).

I also removed a lot of universense's bloated ahh scenes from my game, he is great, but really need to fix this for both himself and his fans.
There are also smaller but greater scenes like flyroxy's, NLVAM... etc. I think enjoying their scenes also good method!

 

[Case]

I'm not sure if this works or makes any difference on your system.. I'm able to add up to 5 girls and 1 guy no problem if no one has any Spec, gloss or Norm textures on.
I'm not sure what the deal with the extra maps is on my system and I've even created my own gloss, spec and norm maps all as jpgs. Even those files that are 90kbs crash my scenes the same way the pngs do... If anyone here happens to have a way to fix that I'd be interested..

I have an older EyeBuyPower with 32 gigs of ram. My graphics cards a gtx 1060ti. Its not amazing but it still works awesome for me. I set the render scale to 85. I cant really see a difference in appearance from 100, but it sure loads up faster. I'm not sure what the deal with the shaders is from my end and why those other maps are so buggy for me.

* Pretty sure that your issue is smth different -sounds rather like something to do with the rendering pipeline or GPU drivers than a memory-related issue (it's pretty much accepted wisdom by now that the heap sections CTD is memory-related) - also pretty sure that you'd like to have @hazmhox advice.

* Five ppl is ... extremely ambitious, for ppl with hardware three geneerations ahead of yours. Unless you're into slideshow's that is.

However, the crashing sounds ... ominous.

I've no advice I could give you but this: Most likely, asking for help in this thread isn't the best best way for you to optimize the chances of a helpful response - you want to attract the attention of ppl who have either encountered your problem before, or those who have a very good understanding of Vam's rendering pipeline (there are a couple of those active here)

TL;DR - Best thing you can do IMO is create your own thread.

 

[le_hibou]

Kudos on the find that morph-dupes also count. That being said, some "nagging":
* Your script "double.-counts" morphs - it counts both the .vmi and the .vmb for each morph as separate morphs. They're not. Not really important (If x>y, then 2x > 2y, too), but still.
* There's a limit to how usefull this can be: The problem with the .var system isn't "getting rid of bad vars" per se (otherwise, we'd solved it years ago) - it's the references to bad .vars.

For example, look at your own whitelist - you've whitelisted both the "V2.___UniversalMorphCollection___.100" and "vecterror._MorphCollection_.26". If you take a closer look, you'll see that "vecterror._MorphCollection_.26" is completely duplicated within "V2.___UniversalMorphCollection___.100" - unsuprisingly, since the latter was meant as a replacement for all prior packs. So why are both in your whitelist? (SPOILER: Bcs vecterror is smart, but his Vam's packagemanager is not ... eg. grab his Lana-look. Both packs are referenced in the .vap ... Why? I've actually tried editing the references manually & repackaging that look - Vam just adds the references again. Probably bcs they're both listed in my cache.)

Same applies for other morphpacks - the "import_reloaded" pack(s) are duplicated in another common package ... etcetc

Thanks for your feedback, and sorry for the late answer.
* You're absolutely right about the double count. I'll try to fix that at some point.
* And yeah, I agree that because of dependencies there's a limit to what we can do... But at least now, we know that it's the morphs that cause this issue, and hopefully creators will pay more attention to the number of morphs they package (and maybe also avoid depending on bloated VARs).

 

[PiddlyD]

A problem with this script - or really, a problem with the hub - is that if some of these bloated vars with tons of morphs are hub hosted dependencies - you're going to have to run the script and delete those dependency vars every time you update dependencies in VAM.

But it does seem to work. i7, 64GB, RTX2080, crashing all the time. Ran this, removed almost all vars with 1000+ morphs - running far more stable now.

 

[le_hibou]

A problem with this script - or really, a problem with the hub - is that if some of these bloated vars with tons of morphs are hub hosted dependencies - you're going to have to run the script and delete those dependency vars every time you update dependencies in VAM.

Or you could give up on those and whitelist them, that's what I did... But yeah, it's not ideal.

 

[PiddlyD]

Or you could give up on those and whitelist them, that's what I did... But yeah, it's not ideal.

I suppose if you made empty files with the .var extension with those names in your addons folder, it wouldn't download the dependencies again after you deleted the actual vars. That would be a nifty option to the script - "delete var and create dummy file as placeholder".

 

[adamgali]

Cutting edge solution without disabling vars, morphs, ..etc

Disable caching. The impact is not very noticeable since I have nvme/i9/64 memory. I have over 1.5TB vam and using caching requires additional 1TB of free space, eats a lot of memory, heap, and causes a lot of crashes. The difference between caching/no-caching loading time is not a big deal.
You will have slower VAM but no more crashes, no deleting vars.

Also, caching will reduce the lifetime of your ssd/nvme.

 

[SlimerJSpud]

I think I found a minor issue with the python script. In the past, one way to eliminate errors from broken dependencies was to unpack the .var file contents to a .var directory with the same name, then edit the files within. The problem here is, the python script is trying to unzip a folder to check it, which doesn't work. It throws an error and ignores the content. In my AddonPackages, this is a folder, not a file:
ERROR: .\Ark1F1.Violet_Update_1.1.var: [Errno 13] Permission denied: '.\\Ark1F1.Violet_Update_1.1.var'

 

[krch]

Thanks for your feedback, I'm glad it helps, I was really happy when I figured that out.

Here's my script, it's my first time writing Python, so bear with me ;-)

You'll have to rename it from morphbloatscanner.txt to morphbloatscanner.py, because the forum wouldn't let me upload a file with the .py extension.

To execute it, you can just run it from your AddonPackages folder (or any folder with VARs) like this:
python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py
Of course, you'll need python.

If you want to customize the max number of morphs by VAR allowed you can use "-m".
If you want to execute the script from anywhere and specify the path to a VAR folder you can use "-p"
Here's an example with both:
python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py -m 900 -p C:\<ANOTHER_PATH>\virt-a-mate\AddonPackages

I've only tested with Windows PowerShell

Also, in the script, there's a whitelist of VAR files that are ignored, that was mostly for my use, but that you can customize by editing he script.

My two cents: As I'm beginning to understand the anatomy of variables and VAM in general, I found your approach very useful. I got rid of two-thirds of repeated/unused morphs and had better timing loading the game

#!/usr/bin/python3
# Original code by: le_hibou
# Modified by krch for redundancy in subfolders (idea by slimerjspud) and counting only once when both vmb and vmi files are present (idea by case), default morphs threshold reduced to 250 (from 500)
# Version: 1.0.1

            
import os
import glob
import zipfile
import sys
import argparse
import pathlib

whiteList = []
whiteList.append("AshAuryn.AshAuryn's_174_Asymmetry_Morphs_for_Mouth.2.var")
whiteList.append("AshAuryn.Pose_Tools.4.var")
whiteList.append("AshAuryn.Sexpressions.5.var")
whiteList.append("cotyounoyume.ExpressionBlushingAndTears.37.var")
whiteList.append("kemenate.Morphs.10.var")
whiteList.append("Oeshii.ReloadedMorphs.1.var")
whiteList.append("SPQR.SPQRAlive.40.var")
whiteList.append("V2.___UniversalMorphCollection___.100.var")
whiteList.append("CheesyFX.ShakeIt.35.var")
whiteList.append("ParticlePinnacle.ppthighcompressorandvibrations.19.var")
whiteList.append("vamX.Base.1.var")
whiteList.append("V2.___UniversalMorphCollection___.103.var")
whiteList.append("vecterror._MorphCollection_.26.var")
whiteList.append("SPQR.SPQRAlive.53.var")
whiteList.append("vamX.1.32.var")

# Initialize the total morph count
totalMorphCount = 0

def logerror(message):
    print("ERROR: " + message, file=sys.stderr, flush=True)

argParser = argparse.ArgumentParser(prog='morphbloatscanner.py', description='List Virt-a-mate VAR files that have both vmi and vmb extensions with the same filename')
argParser.add_argument("-m", "--max", type=int, default=250, help="maximum number of morphs by VAR allowed")
argParser.add_argument("-p", "--path", type=pathlib.Path, default='.', help="path to VAR folder")

args = argParser.parse_args()

print()
print("VAR files with more than or equal to %i morphs will be listed" % args.max)
print()

vars = glob.glob(str(args.path) + "/**/*.var", recursive=True)

for var in vars:
    head_tail = os.path.split(var)
    varFilename = head_tail[1]

    if varFilename not in whiteList:
        # Initialize varMorphCount to 0 for each VAR file
        varMorphCount = 0

        try:
            with zipfile.ZipFile(var) as myzip:
                vmb_files = set()
                vmi_files = set()

                for file in myzip.namelist():
                    filename = file.split("/")[-1].lower()
                    if file.lower().find("custom/atom/person/morphs") != -1:
                        if filename.endswith('.vmb'):
                            vmb_files.add(filename)
                        elif filename.endswith('.vmi'):
                            vmi_files.add(filename)

                counter = 0
                for vmb_filename in vmb_files:
                    vmi_filename = vmb_filename.replace('.vmb', '.vmi')
                    if vmi_filename in vmi_files:
                        counter += 1
                #Debug
                #print("VAR File:"+ varFilename + "Number of pairs:" + str(counter))
                varMorphCount += counter
              
        except Exception as e:
            logerror(var + ": " + str(e))

        # Check if varMorphCount is greater than or equal to args.max
        if varMorphCount >= args.max:
            print(varFilename + " has " + str(varMorphCount) + " morphs")

    # Increment totalMorphCount by varMorphCount for each .var file
    totalMorphCount += varMorphCount

# Print the total morph count for all analyzed VAR files
print("Total morphs found in the directory: " + str(totalMorphCount))

 

[Boogaloo1]

Thanks for your feedback, I'm glad it helps, I was really happy when I figured that out.

Here's my script, it's my first time writing Python, so bear with me ;-)

You'll have to rename it from morphbloatscanner.txt to morphbloatscanner.py, because the forum wouldn't let me upload a file with the .py extension.

To execute it, you can just run it from your AddonPackages folder (or any folder with VARs) like this:
python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py
Of course, you'll need python.

If you want to customize the max number of morphs by VAR allowed you can use "-m".
If you want to execute the script from anywhere and specify the path to a VAR folder you can use "-p"
Here's an example with both:
python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py -m 900 -p C:\<ANOTHER_PATH>\virt-a-mate\AddonPackages

I've only tested with Windows PowerShell

Also, in the script, there's a whitelist of VAR files that are ignored, that was mostly for my use, but that you can customize by editing he script.

Hi there,
Can you please tell me where the results of the scan are saved to?
I can see it has ran successfully but can't find where the results have been saved to.

 

[FreeMaster69]

Thanks for your feedback, I'm glad it helps, I was really happy when I figured that out.

Here's my script, it's my first time writing Python, so bear with me ;-)

You'll have to rename it from morphbloatscanner.txt to morphbloatscanner.py, because the forum wouldn't let me upload a file with the .py extension.

To execute it, you can just run it from your AddonPackages folder (or any folder with VARs) like this:
python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py
Of course, you'll need python.

If you want to customize the max number of morphs by VAR allowed you can use "-m".
If you want to execute the script from anywhere and specify the path to a VAR folder you can use "-p"
Here's an example with both:
python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py -m 900 -p C:\<ANOTHER_PATH>\virt-a-mate\AddonPackages

I've only tested with Windows PowerShell

Also, in the script, there's a whitelist of VAR files that are ignored, that was mostly for my use, but that you can customize by editing he script.

I'm using powershell but it says python can't open the file.

 

[everlaster]

I'm using powershell but it says python can't open the file.

Run as admin?

 

[FreeMaster69]

Run as admin?

Thanks for the reply but it was me being a dumbass as I did not open powershell in addon packages folder first lol. I got rid of so many vars with 1,000s of morphs its unreal. I have a feeling that this script is not scanning my sub folders though as it says "error this file is not a zip" on a lot of packages.

 

[VRmatt]

I'm having the same problem. Too many heap sections.
Recently failed loading the default scene. Is MeshedVR or another dev available to take a look?
<<< Crash dump attached >>>

 

[everlaster]

I'm having the same problem. Too many heap sections.
Recently failed loading the default scene. Is MeshedVR or another dev available to take a look?
<<< Crash dump attached >>>

the file doesn't contain the text "too many heap sections". Where is that error?

Did you try reducing the number of morphs in your installation?

 

[VRmatt]

the file doesn't contain the text "too many heap sections". Where is that error?

Did you try reducing the number of morphs in your installation?

Sorry, the too many heap error is in the popup on the desktop.
It doesnt show in the logs.



This section looks off-putting.
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00007FF9E63694B4)
0x00007FF9E63694B4 (UnityPlayer) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00007FF9E6366910)
0x00007FF9E6366910 (UnityPlayer) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00007FF9E634CC37)
0x00007FF9E634CC37 (UnityPlayer) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00007FF9E60820F9)
0x00007FF9E60820F9 (UnityPlayer) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00007FF9E624B962)
0x00007FF9E624B962 (UnityPlayer) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00007FF9E625887A)
0x00007FF9E625887A (UnityPlayer) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00007FF9E68A188C)
0x00007FF9E68A188C (UnityPlayer) (function-name not available)

 

[VRmatt]

I'd be happy to send you a scene file or screen share with you to look at it in realtime.
perhaps you'd know where to look, better than I would.

 

[5DVoom]

I latest times i start more often catch this error too (64gb here), suppose this relate with with increased var's, but as already was noted above that in moment of crash the memory far from maximum load, well ok about half, but i don't think it's can cause problem in normal case. But ok t's not critical, just for fyu. If will explore some dependency (among var amount) then will post.

 

[cyberbloney]

so if I wanna get rid of these morphs individually in a file, how am i gonna be able to? Im already having trouble keeping phython up even. it runs and it reads but it doesnt really stay up long enough for me to read it.

 

[FreeMaster69]

so if I wanna get rid of these morphs individually in a file, how am i gonna be able to? Im already having trouble keeping phython up even. it runs and it reads but it doesnt really stay up long enough for me to read it.

If you are using python it won't stay up. Use powershell.

 

[VRmatt]

Well, I mean what about the case where you load a default person atom.
If you add 2 more - you'll crash with GC error.

So then, are these loading duplicate morphs? Or are they using the same stack?
I can't load any 3 people, at all. I have trouble believing its a 'morph count' bug. More like a memory leak bug.

I have 64 RAM, and lots of JVM ram available.
I will say, VAM thin twin works most of the time, without any crashing. But obviously its extremely hard to setup a viable working session with it.

 

[virtamater]

Could a folder for Clothes and things that suck up memory be referenced outside of the VaM folder? My VaM folder's 100 gigs.. I use a 8 terabite external hard drive for it, I'd say a good 60 gig of its clothes, does that all really need to b in the addon folder is it's just there for a few scene as presets?
a map to place things not physics or core game related outside of the VaM folder but still be able to be referenced for the use in a particular scene might help if this issue is in fact related to the amount of Vars in your add on folder.

 

[tokyoidolsafk]

Hi all,

I tried everything here from all the helpful people. I removed a whole bunch of Vars with high morphs and did everything but Produce 69 Season 2 just refused to load and got the Heap error no matter what I did. I may not have removed enough, as I have a huge folder of VARS (1TB).

What DID work in the end was to turn off cache. I turned it off. reloaded VAM and BOOM.. Produce 69 season 2 fully functional.
Thought I'd share that with u guys.


My system:
i7 12700K OC
RTX 3080Ti
64GB DDR5 RAM
Samsung 990 Pro NVME 2TB

Cheers.

 

[VRmatt]

I did everything in this thread as well, also cache off, and also with thin twin.
I'm still hitting this error at random times - probably based on model sizes or texture sizes (in simple scenes).

Anyone know if its possible to increase the JVM Heap size for Unity/Vam??
Any devs care to chime in here?

 

[VRmatt]

Why doesn't Meshed just release the V1 code to the community? There's a FAR better chance for us to fix it and make it amazing, than there is for V2 to be a fully fleshed out savior. Every project that becomes open-source inevitably becomes amazing, and bugs get fixed!
Gotta do it for the good of the community.

Honestly, I'm gonna answer the thing you're not gonna like: I would bet that meshed has never stressed tested VAM with 25K vars.
Even tho the game does not prevent you to hoard that much content, I'm convinced that the overall code is not perfectly made to handle that much data. And the fact that reducing the amount of vars fixes the problem does prove that this is the issue.

You have to understand that beyond a handful of devs around here that know exactly how to optimize their plugins/content, the rest does not. You will end up having content that is not perfectly optimized and could lead to memory issues or potential crashes.

So the only solution would be to have several builds with content spread across these.

Unless meshed optimize the game engine to be able to handle that much content, AND that all creators would release content that is optimized af... it's likely that the problem will never go away. ( And 1.x is not gonna receive that kind of heavy change anymore )