• Hello Guest!

    We have recently updated our Site Policies regarding the use of Non Commercial content within Paid Content posts. Please read the new policy here.

    An offical announcement about this new policy can be read on our Discord.

    ~The VaMHub Moderation Team
  • Hello Guest!

    We posted an announcment regarding upcoming changes to Paid Content submissions.

    Please see this thread for more information.

Answered Too many heap sections

Vesolye

New member
Messages
10
Reactions
5
Points
3
I have 64gb ram now, so there should be no problems theorically, but it keeps crashing my game.

Yesterday, it requires 10-13 scenes to crash, but today only one scene with timeline is enough to crash with this message.

Maybe it was caused by my enormous amount of addons? I cannot guess...
 
Same here. Recently upgraded from 32GB to 64Gb. Updated BIOS (Zen7 5800x, ASUS-TUF 570 Gaming plus, 4x8Gb Ram @ 3600Hz - not exactly new hardware that still has teething problems, but enough to run VaM). I don't even HAVE memory OC settings I could revert - and I won't consider running below vendor-recommended XMP settings (Yes that was actually a suggestion on the Discord a couple months ago ...).

No other app but VaM has ever said a peep about memory to me.

P.S.: I've long ago removed UniversenS scenes (no offense to UniversenS, but ... it's 2023. The grace-period for being that bad at packaging expired two years ago ...). Same for NoheadNolegs.

Unfortunately, some of the most common morphpacks clock in at close to 1.000 morphs - and there's considerable overlap between some of them (Iiirc, the "Morph.Morphs_Mega_Pack" contains large parts of the "import_reloaded" pack(s) - and both are referenced far too often to cull either of them).
I also removed a lot of universense's bloated ahh scenes from my game, he is great, but really need to fix this for both himself and his fans.
There are also smaller but greater scenes like flyroxy's, NLVAM... etc. I think enjoying their scenes also good method!
 
Upvote 0
I'm not sure if this works or makes any difference on your system.. I'm able to add up to 5 girls and 1 guy no problem if no one has any Spec, gloss or Norm textures on.
I'm not sure what the deal with the extra maps is on my system and I've even created my own gloss, spec and norm maps all as jpgs. Even those files that are 90kbs crash my scenes the same way the pngs do... If anyone here happens to have a way to fix that I'd be interested..

I have an older EyeBuyPower with 32 gigs of ram. My graphics cards a gtx 1060ti. Its not amazing but it still works awesome for me. I set the render scale to 85. I cant really see a difference in appearance from 100, but it sure loads up faster. I'm not sure what the deal with the shaders is from my end and why those other maps are so buggy for me.

* Pretty sure that your issue is smth different -sounds rather like something to do with the rendering pipeline or GPU drivers than a memory-related issue (it's pretty much accepted wisdom by now that the heap sections CTD is memory-related) - also pretty sure that you'd like to have @hazmhox advice.

* Five ppl is ... extremely ambitious, for ppl with hardware three geneerations ahead of yours. Unless you're into slideshow's that is.

However, the crashing sounds ... ominous.

I've no advice I could give you but this: Most likely, asking for help in this thread isn't the best best way for you to optimize the chances of a helpful response - you want to attract the attention of ppl who have either encountered your problem before, or those who have a very good understanding of Vam's rendering pipeline (there are a couple of those active here)

TL;DR - Best thing you can do IMO is create your own thread.
 
Last edited:
Upvote 0
Kudos on the find that morph-dupes also count. That being said, some "nagging":
* Your script "double.-counts" morphs - it counts both the .vmi and the .vmb for each morph as separate morphs. They're not. Not really important (If x>y, then 2x > 2y, too), but still.
* There's a limit to how usefull this can be: The problem with the .var system isn't "getting rid of bad vars" per se (otherwise, we'd solved it years ago) - it's the references to bad .vars.

For example, look at your own whitelist - you've whitelisted both the "V2.___UniversalMorphCollection___.100" and "vecterror._MorphCollection_.26". If you take a closer look, you'll see that "vecterror._MorphCollection_.26" is completely duplicated within "V2.___UniversalMorphCollection___.100" - unsuprisingly, since the latter was meant as a replacement for all prior packs. So why are both in your whitelist? (SPOILER: Bcs vecterror is smart, but his Vam's packagemanager is not ... eg. grab his Lana-look. Both packs are referenced in the .vap ... Why? I've actually tried editing the references manually & repackaging that look - Vam just adds the references again. Probably bcs they're both listed in my cache.)

Same applies for other morphpacks - the "import_reloaded" pack(s) are duplicated in another common package ... etcetc
Thanks for your feedback, and sorry for the late answer.
* You're absolutely right about the double count. I'll try to fix that at some point.
* And yeah, I agree that because of dependencies there's a limit to what we can do... But at least now, we know that it's the morphs that cause this issue, and hopefully creators will pay more attention to the number of morphs they package (and maybe also avoid depending on bloated VARs).
 
Upvote 0
A problem with this script - or really, a problem with the hub - is that if some of these bloated vars with tons of morphs are hub hosted dependencies - you're going to have to run the script and delete those dependency vars every time you update dependencies in VAM.

But it does seem to work. i7, 64GB, RTX2080, crashing all the time. Ran this, removed almost all vars with 1000+ morphs - running far more stable now.
 
Upvote 0
A problem with this script - or really, a problem with the hub - is that if some of these bloated vars with tons of morphs are hub hosted dependencies - you're going to have to run the script and delete those dependency vars every time you update dependencies in VAM.
Or you could give up on those and whitelist them, that's what I did... But yeah, it's not ideal.
 
Upvote 0
Or you could give up on those and whitelist them, that's what I did... But yeah, it's not ideal.
I suppose if you made empty files with the .var extension with those names in your addons folder, it wouldn't download the dependencies again after you deleted the actual vars. That would be a nifty option to the script - "delete var and create dummy file as placeholder".
 
Upvote 0
Cutting edge solution without disabling vars, morphs, ..etc

Disable caching. The impact is not very noticeable since I have nvme/i9/64 memory. I have over 1.5TB vam and using caching requires additional 1TB of free space, eats a lot of memory, heap, and causes a lot of crashes. The difference between caching/no-caching loading time is not a big deal.
You will have slower VAM but no more crashes, no deleting vars.
no caching.png

Also, caching will reduce the lifetime of your ssd/nvme.
 
Last edited:
Upvote 0
I think I found a minor issue with the python script. In the past, one way to eliminate errors from broken dependencies was to unpack the .var file contents to a .var directory with the same name, then edit the files within. The problem here is, the python script is trying to unzip a folder to check it, which doesn't work. It throws an error and ignores the content. In my AddonPackages, this is a folder, not a file:
ERROR: .\Ark1F1.Violet_Update_1.1.var: [Errno 13] Permission denied: '.\\Ark1F1.Violet_Update_1.1.var'
 
Upvote 0
Thanks for your feedback, I'm glad it helps, I was really happy when I figured that out.

Here's my script, it's my first time writing Python, so bear with me ;-)

You'll have to rename it from morphbloatscanner.txt to morphbloatscanner.py, because the forum wouldn't let me upload a file with the .py extension.

To execute it, you can just run it from your AddonPackages folder (or any folder with VARs) like this:
python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py
Of course, you'll need python.

If you want to customize the max number of morphs by VAR allowed you can use "-m".
If you want to execute the script from anywhere and specify the path to a VAR folder you can use "-p"
Here's an example with both:
python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py -m 900 -p C:\<ANOTHER_PATH>\virt-a-mate\AddonPackages

I've only tested with Windows PowerShell

Also, in the script, there's a whitelist of VAR files that are ignored, that was mostly for my use, but that you can customize by editing he script.
My two cents: As I'm beginning to understand the anatomy of variables and VAM in general, I found your approach very useful. I got rid of two-thirds of repeated/unused morphs and had better timing loading the game
Code:
#!/usr/bin/python3
# Original code by: le_hibou
# Modified by krch for redundancy in subfolders (idea by slimerjspud) and counting only once when both vmb and vmi files are present (idea by case), default morphs threshold reduced to 250 (from 500)
# Version: 1.0.1

            
import os
import glob
import zipfile
import sys
import argparse
import pathlib

whiteList = []
whiteList.append("AshAuryn.AshAuryn's_174_Asymmetry_Morphs_for_Mouth.2.var")
whiteList.append("AshAuryn.Pose_Tools.4.var")
whiteList.append("AshAuryn.Sexpressions.5.var")
whiteList.append("cotyounoyume.ExpressionBlushingAndTears.37.var")
whiteList.append("kemenate.Morphs.10.var")
whiteList.append("Oeshii.ReloadedMorphs.1.var")
whiteList.append("SPQR.SPQRAlive.40.var")
whiteList.append("V2.___UniversalMorphCollection___.100.var")
whiteList.append("CheesyFX.ShakeIt.35.var")
whiteList.append("ParticlePinnacle.ppthighcompressorandvibrations.19.var")
whiteList.append("vamX.Base.1.var")
whiteList.append("V2.___UniversalMorphCollection___.103.var")
whiteList.append("vecterror._MorphCollection_.26.var")
whiteList.append("SPQR.SPQRAlive.53.var")
whiteList.append("vamX.1.32.var")

# Initialize the total morph count
totalMorphCount = 0

def logerror(message):
    print("ERROR: " + message, file=sys.stderr, flush=True)

argParser = argparse.ArgumentParser(prog='morphbloatscanner.py', description='List Virt-a-mate VAR files that have both vmi and vmb extensions with the same filename')
argParser.add_argument("-m", "--max", type=int, default=250, help="maximum number of morphs by VAR allowed")
argParser.add_argument("-p", "--path", type=pathlib.Path, default='.', help="path to VAR folder")

args = argParser.parse_args()

print()
print("VAR files with more than or equal to %i morphs will be listed" % args.max)
print()

vars = glob.glob(str(args.path) + "/**/*.var", recursive=True)

for var in vars:
    head_tail = os.path.split(var)
    varFilename = head_tail[1]

    if varFilename not in whiteList:
        # Initialize varMorphCount to 0 for each VAR file
        varMorphCount = 0

        try:
            with zipfile.ZipFile(var) as myzip:
                vmb_files = set()
                vmi_files = set()

                for file in myzip.namelist():
                    filename = file.split("/")[-1].lower()
                    if file.lower().find("custom/atom/person/morphs") != -1:
                        if filename.endswith('.vmb'):
                            vmb_files.add(filename)
                        elif filename.endswith('.vmi'):
                            vmi_files.add(filename)

                counter = 0
                for vmb_filename in vmb_files:
                    vmi_filename = vmb_filename.replace('.vmb', '.vmi')
                    if vmi_filename in vmi_files:
                        counter += 1
                #Debug
                #print("VAR File:"+ varFilename + "Number of pairs:" + str(counter))
                varMorphCount += counter
              
        except Exception as e:
            logerror(var + ": " + str(e))

        # Check if varMorphCount is greater than or equal to args.max
        if varMorphCount >= args.max:
            print(varFilename + " has " + str(varMorphCount) + " morphs")

    # Increment totalMorphCount by varMorphCount for each .var file
    totalMorphCount += varMorphCount

# Print the total morph count for all analyzed VAR files
print("Total morphs found in the directory: " + str(totalMorphCount))
 
Last edited:
Upvote 0
Thanks for your feedback, I'm glad it helps, I was really happy when I figured that out.

Here's my script, it's my first time writing Python, so bear with me ;-)

You'll have to rename it from morphbloatscanner.txt to morphbloatscanner.py, because the forum wouldn't let me upload a file with the .py extension.

To execute it, you can just run it from your AddonPackages folder (or any folder with VARs) like this:
python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py
Of course, you'll need python.

If you want to customize the max number of morphs by VAR allowed you can use "-m".
If you want to execute the script from anywhere and specify the path to a VAR folder you can use "-p"
Here's an example with both:
python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py -m 900 -p C:\<ANOTHER_PATH>\virt-a-mate\AddonPackages

I've only tested with Windows PowerShell

Also, in the script, there's a whitelist of VAR files that are ignored, that was mostly for my use, but that you can customize by editing he script.
Hi there,
Can you please tell me where the results of the scan are saved to?
I can see it has ran successfully but can't find where the results have been saved to.
 
Upvote 0
Thanks for your feedback, I'm glad it helps, I was really happy when I figured that out.

Here's my script, it's my first time writing Python, so bear with me ;-)

You'll have to rename it from morphbloatscanner.txt to morphbloatscanner.py, because the forum wouldn't let me upload a file with the .py extension.

To execute it, you can just run it from your AddonPackages folder (or any folder with VARs) like this:
python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py
Of course, you'll need python.

If you want to customize the max number of morphs by VAR allowed you can use "-m".
If you want to execute the script from anywhere and specify the path to a VAR folder you can use "-p"
Here's an example with both:
python.exe C:\<YOUR_PATH_HERE>\morphbloatscanner.py -m 900 -p C:\<ANOTHER_PATH>\virt-a-mate\AddonPackages

I've only tested with Windows PowerShell

Also, in the script, there's a whitelist of VAR files that are ignored, that was mostly for my use, but that you can customize by editing he script.
I'm using powershell but it says python can't open the file.
 
Upvote 0
Run as admin?
Thanks for the reply but it was me being a dumbass as I did not open powershell in addon packages folder first lol. I got rid of so many vars with 1,000s of morphs its unreal. I have a feeling that this script is not scanning my sub folders though as it says "error this file is not a zip" on a lot of packages.
 
Upvote 0
Back
Top Bottom