Resource icon

Guides Legacy Morph Cleanup Guide

Juno

Well-known member
Featured Contributor
Messages
226
Reactions
2,818
Points
93
Juno submitted a new resource:

Legacy Morph Cleanup Guide - Guide to dedupe legacy morphs

As many of you will know VAM didn't always have the current VAR system in place and many of us will have tons of content from the dark days of VAC packaging and content installation being folder merges.

I for one have been fed up with the large number of duplicate morphs showing in my morph browser for quite some time due to the old content so I decide this morning to deal with this one and for all.

So here I present to you Juno's guide to cleaning up and deduping morphs from the legacy...

Read more about this resource...
 
Nice one Acid I will have a play and update as needed (y)
 
You might want to look into this: https://hub.virtamate.com/resources/varbsorb.164/

It'll handle morph references and things like that for you.
I'm a fan of your work Acid! Just a few questions regarding varbsorb...

1) How confident are you that this will not damage scenes? (assuming symbolic links are not used)

2) Is there a detailed description on github of how this works? I've been brainstorming a way to do something like this but couldn't think of a methodology that was idiot proof.

3) Is this based on a methodology that has already been proven? Is this why your are relatively confident?

I'm just starting out as a programmer, these type of file management scripts scare the sh** out of me because of they damage they can do. Although, I wrote a batch script that does something similar. It, however, doesn't change references in the scene files! So, I know it's do-able.
I have a program that allows me to find and replace across entire directory tree, text within scripts. When I use it, however, I'm usually on the edge of my seat, making sure I didn't make a mistake. When you f***-up you f***-up big!
Thanks a bunch! Great work! (y) :D
 
@fubar_life123 I completely understand, and in the end something that deletes and/or modifies files in a recursive way should always be used with caution. I would recommend making a backup of your saves folder before running varbsorb, just in case. However I ran it myself a few times, here's why I think it's fairly safe:

1. It's open source, you can inspect the code here: https://github.com/acidbubbles/vam-varbsorb
2. It's unit tested (65% code coverage) and has an A rating of code quality by Codacy and 0 alerts by lgtm. There won't prevent bugs but at least they mean basic reviews and typical pitfalls were seen and addressed.
3. Instead of deleting any files, it sends them to the recycle bin. So you can easily see what was modified and restore the files if needed. Note that the scene json files however will be modified.

The methodology is based on file hashes and type-awareness. A hash is a unique representation of a file's content (it's heavily used in security, I'm using SHA1 because security is not a concern in this specific case however). I'm scanning both the scenes and var files, and I find matching files. For morphs, I only match based on .vmi+.vmb pairs to avoid broken references. After cleaning up a file, I update scene files so they point to the .var file instead. In case of multiple var files having the same content, I'll prefer the .var with the fewest files.

Hope that helps, to be honest I cannot say with confidence that this code is flawless (there was no peer review and there's no metrics of success other than on my own machine and from the few people who were nice enough to report back problems) but if you backup your files, it should be fine. I have mechanisms in place to prevent the program from going outside the "safe" folders.
 
@fubar_life123 I completely understand, and in the end something that deletes and/or modifies files in a recursive way should always be used with caution. I would recommend making a backup of your saves folder before running varbsorb, just in case. However I ran it myself a few times, here's why I think it's fairly safe:

1. It's open source, you can inspect the code here: https://github.com/acidbubbles/vam-varbsorb
2. It's unit tested (65% code coverage) and has an A rating of code quality by Codacy and 0 alerts by lgtm. There won't prevent bugs but at least they mean basic reviews and typical pitfalls were seen and addressed.
3. Instead of deleting any files, it sends them to the recycle bin. So you can easily see what was modified and restore the files if needed. Note that the scene json files however will be modified.

The methodology is based on file hashes and type-awareness. A hash is a unique representation of a file's content (it's heavily used in security, I'm using SHA1 because security is not a concern in this specific case however). I'm scanning both the scenes and var files, and I find matching files. For morphs, I only match based on .vmi+.vmb pairs to avoid broken references. After cleaning up a file, I update scene files so they point to the .var file instead. In case of multiple var files having the same content, I'll prefer the .var with the fewest files.

Hope that helps, to be honest I cannot say with confidence that this code is flawless (there was no peer review and there's no metrics of success other than on my own machine and from the few people who were nice enough to report back problems) but if you backup your files, it should be fine. I have mechanisms in place to prevent the program from going outside the "safe" folders.
Sounds similar to what I did in the batch file using fail-safes and deleting files using "for loops" with strict conditions on deletion. Hash is fairly safe it is incredibly unlikely to have collisions and it is a lot faster than bit for bit comparison. Works for me!
This has the potential to really trim-down the fat with things like Custom textures, Custom Unity Assets and plugins! Then there would be no more complaining about bloat! 😃

correction: I just noticed in the recycling bin that indeed it does those assets as well! :love:
 
Last edited:
Sounds similar to what I did in the batch file using fail-safes and deleting files using "for loops" with strict conditions on deletion. Hash is fairly safe it is incredibly unlikely to have collisions and it is a lot faster than bit for bit comparison. Works for me!
This has the potential to really trim-down the fat with things like Custom textures, Custom Unity Assets and plugins! Then there would be no more complaining about bloat! 😃

correction: I just noticed in the recycling bin that indeed it does those assets as well! :love:
And restore all!!! Bunch of references were not updated!! Thanks anyway!
 
Oh noes! Well it's been a while, but there shouldn't be much missing for it to work, but since I've cleaned up my own saves I moved on and didn't maintain it :( Sorry it didn't work out!
 
Oh noes! Well it's been a while, but there shouldn't be much missing for it to work, but since I've cleaned up my own saves I moved on and didn't maintain it :( Sorry it didn't work out!
It's a very complicated problem to solve... There may have been changes made on my behave that resulted in the dead links, but of one of the dead link problems (missing textures) was definitely due to varbsorb. When I restored the deleted files she was back to her former glory! There needs to be a solution to find dead links and try to connect them with either var (if the resource is available) or some predesignated location in custom folder before migrating healthy links to var. Otherwise, it will appear that varbsorb didn't work (In those cases). The existing dead links need to be addressed first before everything else. Generally, the file name and ext would be sufficient to locate it's twin. Then pitch the dups! I would try to do something myself but I'm struggling to find time to learn (programming).
 
Oh noes! Well it's been a while, but there shouldn't be much missing for it to work, but since I've cleaned up my own saves I moved on and didn't maintain it :( Sorry it didn't work out!
Actually, something just occurred to me; did you exclude the "Default.json" from being updated? The textures in that file were still pointing to the original location. I'm just curious if you perhaps intentionally excluded that file being that it is the default load scene? If so, then the rest of the process might have completed without issue. It's hard to verify success when dealing with so many files!!! o_O

I suppose the only way to validate success is if the link is no longer dead.... ;)
 
Last edited:
I get an error no matter from what I select.
 

Attachments

  • doesnt work for me.JPG
    doesnt work for me.JPG
    199.5 KB · Views: 0
Might be a silly question but what version of vam are you running? The file path in your screenie says 1.3. i think the path for morphs was different on the older versions. it so long i cant remember where.
 
Might be a silly question but what version of vam are you running? The file path in your screenie says 1.3. i think the path for morphs was different on the older versions. it so long i cant remember where.

At the time of my VaM install, it was version 1.3 but i have 1.19.2 and I never changed the root folder name.

EDIT: I had to go to options and disable the ignore specify folder because It is located in program files on my D drive. It works now.
 
Last edited:
There appears to be about 11,000 files in my morphs/female folder.

Most of my duplicates reported on first run appear to be in some subfolder called AUTO. ie. over 3000 of them.

Is this some automatically generated VAM folder generated from existing morphs?

Seems all its doing is equivalent to just deleting the AUTO folder, but for some reason I seem to remember that was there for some reason.
 
Back
Top Bottom