Other [Utility] VAMUpdater - Always Use the Latest Addons!

[droseofc]

Any way to specify where it backups the files to?

 

[droseofc]

Also, is it safe to cancel the process and if so, will it pick up where it left off the next time?

 

[venkman]

Any way to specify where it backups the files to?

Not presently, but implementing this is fairly trivial. I could probably have it look for a config file where you could specify persistent settings, this being one of them.

Also, is it safe to cancel the process and if so, will it pick up where it left off the next time?

Define "safe"? For the most part, you shouldn't break anything if you kill the process as backups are made before files are written, etc, but if you kill it in the middle of writing a backup or updating a VAR, then there's really no telling what will happen.

I don't track where in the scanning you are when stuff is scanned, only for the presence of that file's backup. If a backup exists, then I don't touch the file again, so by virtue of that, it will pick up where it left off.

 

[slimjxmmi]

I have about 7000 vars. It got through 1500 and then just quit without updating the rest. When I tried restarting, it went through a few and then quit without telling me which one it had an issue with. I've tried deleting the last VAR package in the list. (It might help to know how the utility orders the var packages or to have a prompt before exiting.) (or if it has trouble with foreign characters i.e. Chinese characters) Any help? @venkman

 

[stylesi]

EDIT: Nvm, I think it works again now for me.. gg

 

[slimjxmmi]

EDIT: Nvm, I think it works again now for me.. gg

How did you fix it?

 

[venkman]

venkman updated [Utility] VAMUpdater - Always Use the Latest Addons! with a new update entry:

Kickin' tires and lightin' fires!

What's new? A LOT.

For starters, I've added a GUI. The GUI has all of the same settings/options as the command-line, but has the added benefit of storing your settings for future use.

View attachment 137974

You can now specify which elements you want to scan for and move/update, versus a sort of mish-mashed sequence of flags that did weird stuff if input improperly.

For debugging, I threw in a "Verbose Output" flag, which is REALLY verbose. The point of this is to figure out...

Read the rest of this update entry...

 

[venkman]

I have about 7000 vars. It got through 1500 and then just quit without updating the rest. When I tried restarting, it went through a few and then quit without telling me which one it had an issue with. I've tried deleting the last VAR package in the list. (It might help to know how the utility orders the var packages or to have a prompt before exiting.) (or if it has trouble with foreign characters i.e. Chinese characters) Any help? @venkman
View attachment 137487

So, the SILVER ALERT bit is something I had for debugging, and can be ignored. I'll be sure to remove it in the next update.

If you use the "Update_All.bat" file, it should instruct the app to scan *everything* again.

How did you fix it?

Give the newest update a go. I've added a nice GUI, plus options for more Verbose Messaging. Enable that, ensure you've told it to force rescan, and see how far you get. If it still fails, let me know what the last lines were from the messages.

 

[venkman]

Any way to specify where it backups the files to?

A custom backup location is now part of the options you can configure via command-line flag OR GUI.

 

[slimjxmmi]

So, the SILVER ALERT bit is something I had for debugging, and can be ignored. I'll be sure to remove it in the next update.

If you use the "Update_All.bat" file, it should instruct the app to scan *everything* again.


Give the newest update a go. I've added a nice GUI, plus options for more Verbose Messaging. Enable that, ensure you've told it to force rescan, and see how far you get. If it still fails, let me know what the last lines were from the messages.

@venkman

The problem still occurs. I couldn't copy and paste the output since the "Status" screen doesn't allow it (only one line at a time). The program seems to only check for duplicates that are in the plugins.txt file and not all multiple copies (don't know if this was intended or not; it would be a pain to add all the VAR names to the plugins.txt file). Also the "Update File Dates" didn't do anything. On the status screen it seemed that it was rechecking all the VARs but it should have detected some VARs that do not reference the .latest in the meta.json. The "ignore dependencies" option makes the scanning even quicker, not slower, despite the warning (even with rescan on).


For example, it would be nice if this kept only the lastest (this is still in the AddonPackages folder):


I don't know if the utility is supposed to update every dependency package to ".latest" or just the ones in plugins.txt.

It appears that it will examine and enumerate all the addon packages. But then it will only scan the ones in plugins.txt:

 

[venkman]

@venkman

The problem still occurs. I couldn't copy and paste the output since the "Status" screen doesn't allow it (only one line at a time). The program seems to only check for duplicates that are in the plugins.txt file and not all multiple copies (don't know if this was intended or not; it would be a pain to add all the VAR names to the plugins.txt file). Also the "Update File Dates" didn't do anything. On the status screen it seemed that it was rechecking all the VARs but it should have detected some VARs that do not reference the .latest in the meta.json. The "ignore dependencies" option makes the scanning even quicker, not slower, despite the warning (even with rescan on).
View attachment 138080

For example, it would be nice if this kept only the lastest (this is still in the AddonPackages folder):
View attachment 138081

I don't know if the utility is supposed to update every dependency package to ".latest" or just the ones in plugins.txt.

Ah, yeah. It's only targeting the ones in plugins.txt. You can edit that file as desired to include any other packages you want updated automagically.

Unfortunately, I can't quite account for every single addonpackage out there, but I'm happy to include suggestions if people have tested them and are sure that updating won't break something else.


I'll take a look at the file dates bit, if that's selected, it should be using the current time...but then again, I could have borked something in the rewrite.

And I'll also update the instructions to point out that it's actually faster. I assumed it'd take longer because it has to check every var, even without the meta saying so, but it's possible that not having to parse and check the meta is making it quicker.

Thanks for the feedback!

 

[slimjxmmi]

Ah, yeah. It's only targeting the ones in plugins.txt. You can edit that file as desired to include any other packages you want updated automagically.

Unfortunately, I can't quite account for every single addonpackage out there, but I'm happy to include suggestions if people have tested them and are sure that updating won't break something else.


I'll take a look at the file dates bit, if that's selected, it should be using the current time...but then again, I could have borked something in the rewrite.

And I'll also update the instructions to point out that it's actually faster. I assumed it'd take longer because it has to check every var, even without the meta saying so, but it's possible that not having to parse and check the meta is making it quicker.

Thanks for the feedback!

No problem. Great plugin nonetheless. Is there a way for it to auto detect multiple instances of the addon packages since it enumerates them anyway? Like for example:
(1) create a list of all the VAR packages from AddonPackages directory
(2) get rid of {#}.var at the end of each file
(3) convert to all lowercase
(4) check the list for any repeat lines (because that would mean there are multiple instances)
(5) keep unique line for each repeat line
(6) generate new plugins.txt

I believe you can code it this way instead of having a static plugins.txt and that would save us all time finding the duplicate packages and would be unique to our AddonPackages folder.

I was manually able to do this with Command Prompt to get the AddonPackages file list into a .txt file. Then using Notepad++ you can change it to all lowercase and delete the ending #.var. Last, remove duplicates and overwrite to new plugins.txt. I'm sure you could code that.

This method, in theory, should made your plugin work properly to get rid of all duplicates, however it stalls on certain packages.

Also could I make a request for the "Status" window be able to copy and paste more than 1 line at a time? I need to debug some broken packages and it would be useful if I could copy and paste for a Ctrl+F.

The reason for this is because I stopped the program half way through for some tests. Even when selecting "rescan," some of the packages still prompt: "Skipping previously updated VAR" which leads me to believe that they are broken and the ones I stopped on. It would be good if these skips can be recorded on the .txt output file.

 

[venkman]

venkman updated [Utility] VAMUpdater - Always Use the Latest Addons! with a new update entry:

Features & Fixes

Add option to update plugins.txt before scan. This will create a new "plugins_custom.txt" file, which will be appended to the included plugins.txt file. Plugins_custom.txt will not be overwritten on application updates.

Add "Select all" and "copy" buttons to the status page, because that's kind of useful.

Ensure any jpg files for a var package are moved as well.

Read the rest of this update entry...

 

[venkman]

No problem. Great plugin nonetheless. Is there a way for it to auto detect multiple instances of the addon packages since it enumerates them anyway? Like for example:
(1) create a list of all the VAR packages from AddonPackages directory
(2) get rid of {#}.var at the end of each file
(3) convert to all lowercase
(4) check the list for any repeat lines (because that would mean there are multiple instances)
(5) keep unique line for each repeat line
(6) generate new plugins.txt

I believe you can code it this way instead of having a static plugins.txt and that would save us all time finding the duplicate packages and would be unique to our AddonPackages folder.

I was manually able to do this with Command Prompt to get the AddonPackages file list into a .txt file. Then using Notepad++ you can change it to all lowercase and delete the ending #.var. Last, remove duplicates and overwrite to new plugins.txt. I'm sure you could code that.

This method, in theory, should made your plugin work properly to get rid of all duplicates, however it stalls on certain packages.

Also could I make a request for the "Status" window be able to copy and paste more than 1 line at a time? I need to debug some broken packages and it would be useful if I could copy and paste for a Ctrl+F.

The reason for this is because I stopped the program half way through for some tests. Even when selecting "rescan," some of the packages still prompt: "Skipping previously updated VAR" which leads me to believe that they are broken and the ones I stopped on. It would be good if these skips can be recorded on the .txt output file.

Ask, and ye shall receive.

Added a new option to the UI/flags to allow auto-creation of an additional plugins_custom.txt file, which will contain any duplicate entries found for ANY addonpackage. Use with caution, obviously.

Also added a "select all" and "copy" feature to the status, for better logging/support.

Enjoy!

 

[Sam Gold]

Since windows10 update in end of June, your awesome prog is broke. Before the update using your version 3 worked fine. It took 11hrs but expected since I have over 12k vars and repo on HDD NAS so no big deal (I use symlinks for vam to see the vars which work fine without taking valuable SSD space). It will scan on local drives the vars fine even the vars that turn up in error log. And as long as just scan around 100 vars on the NAS site its fine, but when doing in thousands, it chokes up error not enough resources. I thought maybe file path name length might be problem but on average file path characters are under 200.

So I think you did nothing wrong since it worked before windows update. I even reinstalled dotnet runtimes made no difference.

Also the GUI recent version won't let me choose a backup location outside the repo of vars it's scanning with Select button. I have to paste or type in new location for it to create the backup folders. My plan it to leave vars on repo NAS and have backup files on my local SSD.

I hope your genius can undo what I believe windows undid. This was very useful in trimming down my var collection.

-Thanks

-FYI

After first run I had a backup folder of 85GB. I zip it and it dropped to 3.5GB since it was just text files really. Does it skip the files to scan from folder name or actual json file it modifies? It would be great to either leave the folders empty so it saves space and speed up future runs from reading file names from backup folder. Even better have a database with stored var changes so if needed it can recreate json file. Most of that big file size is from mocap atom plot points that are not needed. Having just changes would make it like Z-Delta backups so backup and restore would be super fast. Just a thought. I know for small var collectors not necessary but once you go over 5k I think it would helpful and appreciated.

Spoiler: LOGS

ver.3 log

ver 5 errors

ver 7 errors

 

[droseofc]

A custom backup location is now part of the options you can configure via command-line flag OR GUI.

I seen the implementation... very nice of you to do that. I just did a test and I did specify a backup location, however it still created the backup in the same directory of VAM. Could it be that it I selected a different drive to back up to? Such as, my VAM is on I:/ and I was backing up to B:/

 

[xadamadax]

Great great great utility! - Any chance to make it compatible with unpacked vars (i.e. folders in addonpackages with .var tagged onto the end of them vs. the original compressed package). Thanks for all your hard work.

 

[stylesi]

I seen the implementation... very nice of you to do that. I just did a test and I did specify a backup location, however it still created the backup in the same directory of VAM. Could it be that it I selected a different drive to back up to? Such as, my VAM is on I:/ and I was backing up to B:/

yes, I have the same bug. Folder option is borked.

 

[Rayt]

I've just used your utility selecting addonpackages and unchecked the update file dates, but my new>old are all jumbled.

Additionally, I then run the redundant addons and specified the backup folder in the GUI but it seems to not have moved the backups there.

 

[xadamadax]

Great great great utility! - Any chance to make it compatible with unpacked vars (i.e. folders in addonpackages with .var tagged onto the end of them vs. the original compressed package). Thanks for all your hard work.

Just circling back, any chance to get this working with Non compressed var (folders with .var extension in addonpacks folder). Once again, great tool. I use it on my smaller instance, this would enable me to run it on my large instance

 

[stylesi]

Other than having the backup folder bug, this tool is really great! For new users, you just need to update plugins.txt to be more comprehensive on your own.

 

[TechnoLazer]

I have a utility that Ive done for myself to maintain my ridiculous count of files and give very quick parses and moves to find files that depend on an installation and move them around and Ive noticed that it looks like every file that has had its zip altered by your tool leaves the entry bytes borked. Im using system.io.compression.ziparchive to extract. To give you specifics on the issue, the two files below show what happens when I simply make a copy of the file, open in a utility which opens fine, save the meta.json, and then push it back down into the file. nothing in the files change, but the lengths get properly recorded. I hadnt noticed because it seems most zip utilities fix this, but it makes every file error for me unless its fixed. so about 2000 of my vars now have this error. I will have to find a way to correct it on my end but itll help tremendously if it could be fixed, or at minimum maybe reply with what compression library youre using (i know an old version of chilkat has this problem)
Thanks

 

[xadamadax]

I have a utility that Ive done for myself to maintain my ridiculous count of files and give very quick parses and moves to find files that depend on an installation and move them around and Ive noticed that it looks like every file that has had its zip altered by your tool leaves the entry bytes borked. Im using system.io.compression.ziparchive to extract. To give you specifics on the issue, the two files below show what happens when I simply make a copy of the file, open in a utility which opens fine, save the meta.json, and then push it back down into the file. nothing in the files change, but the lengths get properly recorded. I hadnt noticed because it seems most zip utilities fix this, but it makes every file error for me unless its fixed. so about 2000 of my vars now have this error. I will have to find a way to correct it on my end but itll help tremendously if it could be fixed, or at minimum maybe reply with what compression library youre using (i know an old version of chilkat has this problem)
Thanks

I ran into the same situation with his tool. Fix that was easy and worked for me:
-Open var folder in Winrar
-Select all, repair (or alt+r). This will create copies looking like "rebuilt.myoriginalvarname.var"
-Download a tool called bulk renamer or similar. Select all, remove "rebuilt."
-Replace damaged vars with fixed ones

Honestly I still use his tool and follow this process from time to time as the benefits outweigh the nuissance

 

[cloudamus]

I keep getting prompted about missing frameworks for .Net. I already have .5 and even .6 installed.

Even if I hit yes, nothing happends. Same thing when running from the cmd.

 

[Case]

I have a utility that Ive done for myself to maintain my ridiculous count of files and give very quick parses and moves to find files that depend on an installation and move them around and Ive noticed that it looks like every file that has had its zip altered by your tool leaves the entry bytes borked. Im using system.io.compression.ziparchive to extract. To give you specifics on the issue, the two files below show what happens when I simply make a copy of the file, open in a utility which opens fine, save the meta.json, and then push it back down into the file. nothing in the files change, but the lengths get properly recorded. I hadnt noticed because it seems most zip utilities fix this, but it makes every file error for me unless its fixed. so about 2000 of my vars now have this error. I will have to find a way to correct it on my end but itll help tremendously if it could be fixed, or at minimum maybe reply with what compression library youre using (i know an old version of chilkat has this problem)
Thanks

I noticed there was a "VamUpdater.pdb" in my installation folder, so got curious & tried having a look at the "Vam_Updater.dll" with ILSpy - this is what I get:

// C:\3x_Games_C\Virt-A-Mate bare\Venkman.VamUpdater.7\VamUpdater.dll
// VamUpdater, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null
// Global type: <Module>
// Entry point: VamUpdater.Program.Main
// Architecture: AnyCPU (64-bit preferred)
// Runtime: v4.0.30319
// This assembly was compiled using the /deterministic option.
// Hash algorithm: SHA1
// Debug info: Loaded from portable PDB: C:\3x_Games_C\Virt-A-Mate bare\Venkman.VamUpdater.7\VamUpdater.pdb

using System.Diagnostics;
using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.Versioning;

[assembly: CompilationRelaxations(8)]
[assembly: RuntimeCompatibility(WrapNonExceptionThrows = true)]
[assembly: Debuggable(/*Could not decode attribute arguments.*/)]
[assembly: TargetFramework(".NETCoreApp,Version=v6.0", FrameworkDisplayName = "")]
[assembly: AssemblyCompany("Venkman")]
[assembly: AssemblyConfiguration("Release")]
[assembly: AssemblyFileVersion("3.0.0")]
[assembly: AssemblyInformationalVersion("1.0.0")]
[assembly: AssemblyProduct("VamUpdater")]
[assembly: AssemblyTitle("VamUpdater")]
[assembly: TargetPlatform("Windows7.0")]
[assembly: SupportedOSPlatform("Windows7.0")]
[assembly: AssemblyVersion("3.0.0.0")]

Wasn't Vam_Updater supposed to run with Net 5.0? (Ok, I'm using VS2022, could simply be that - I have no idea what I'm doing, I'm just a little ape punching buttons).

And "Windows 7.0"?

---

Question: How would I go about figuring out which compression library Vam_Updater is using? I'm seeing

// Ionic.Zip, Version=1.9.1.8, Culture=neutral, PublicKeyToken=edbe51ad942a3f5c (unresolved)

Is that the "the thingy"? Where do I have to poke & what does the thingy I'm poking for look like?