VaM 1.x Most of the .latest vars are missing

[Faco2000]

Hi all !
First I want to say that I'm fully aware of how to add vars on VAM and I'm checking missing vars and updates almost every time I use VAM.
When I look to my missing dependencies, I see that many of them are not with a numbered version (like 14mhz.Plugin-AutoFlutterTongue.5.var) but with .latest instead (like 14mhz.Plugin-AutoFlutterTongue.latest.var).
Even if I actually have the latest version of the var in my addonpackage folder, these ;latest vars are missing and they cannot be found anywhere.

Any clue about how to solve this ?

Many thanks !

 

[AshAuryn]

Could you screenshot an example, or provide a link to a resource that has these missing dependencies?

 

[Faco2000]

Waow ! I'm always impressed when a VAM superstar answers to one of my post Sorry for the late answer, I was in holidays...

Here are some screenshots but I don't know how to know which var calls a specific missign dependency. I now have a huge VAM folder where I put everything I have and I use VAM Thin Twin to launch light VAMs. Herebelow are screenshots obtained while launching the usual VAM.exe without using thin twin.

Many thanks for taking time to answer my question

 

[AshAuryn]

Currently looking into this. Strangely -- I'm not sure why you have an "AshAuryn.Sexpressions.1" dependency. That var never existed. I don't even have "version 1".

I tested this with "latest" and it worked fine, and correctly found "Sexpressions.5". However, when trying specifically "Sexpressions.1" I got "NOT ON HUB" which should not happen.

Will update here when I've done some more debugging.

 

[SlimerJSpud]

It might be possible that one damaged var is corrupting the scanning process. That can cause all kinds of ills. Unfortunately, there's no simple way to track the bad one down. There's a divide and conquer approach where you start pulling vars out of AddonPackages in batches to someplace Vam won't see them, then scanning again. If you have a suspect, try opening it with 7Zip. If 7Zip throws an error, that var is corrupt. Delete it and redownload.

 

[Faco2000]

Hi, it seems that most of the missing dependencies with .latest are coming from KittyMocap vars. Herebelow an example for one of her latest releases :

I'll contact her to see if there something wrong on my side or if there is something to be done

 

[AshAuryn]

I consulted with MeshedVR and when you click "scan for missing packages" and one of your vars has a dependency with a specific version that isn't available on the Hub (in this case, Sexpressions.1) then it will return "NOT ON HUB". This was by design, and is not a bug.

However, "latest" should not have an issue, and in testing we found no issue finding some of these missing packages with the "latest" designation. So it's safe to assume there is something in that folder that is corrupting the scan.

@Faco2000 if you can reach out to me on Discord I can run a real-time test with you and we may be able to narrow down the possibilities.

@SlimerJSpud 's suggestion is what I would do first. Move everything out of your addonpackages folder except a few of the vars with missing dependencies that you've verified are actually on the Hub. Then start testing batches of files by rotating them in to your addonpackages folder one batch at a time (rotate, meaning put them in, run the scan, then take them back out) until you've narrowed down a batch that makes the scan stop working.

*EDIT* --- I did find a cause for this and will work on a solution. The issue is complicated --- but basically if you have any vars that request a very specific version (like Sexpressions.1) and it doesn't exist on the Hub, then all other vars that require "sexpressions.latest" will also show "not on hub" because of part of the scan that removes duplicates in certain circumstances. I don't want you to waste your time searching for corrupted vars because you won't find any. I'll update here if I find a way to solve this on my end.

 

[SlimerJSpud]

That's weird. KM465 calls for Sulwon.4 on it's dependency list but the page for Sulwon shows only v6 in the history tab. Is it possible for a creator to purge old versions from the history? On the web interface, if you click on download by Sulwon.4 in the dependency list, you get v6.

Looks - Sulwon - History

hub.virtamate.com

That would mean that the KM465 var is at fault on that resource. I have Sulwon 5 and 6 in my folder, but not 4.

In the case of Sexpressions.1, any var that calls that version is wrong, because you said there was no such version. It's not the "latest" things causing the error, it's one or more badly assembled vars calling a specific version that's MIA.

 

[SlimerJSpud]

Heads up on another scene that might have related issues. The Test Drive scene calls for some resources as ".minlatest" which also behaves differently in-game vs. on the web. The web interface downloads the latest version if I click on it. If I already have it, I'll know when it asks for permission to overwrite it. In-game, it doesn't know I already have the latest version, and thinks it needs to download it again. When I let it download it again, I get an error:

!> Error while trying to save file AddonPackages/AshAuryn.AshAuryn_Sexpressions_2_Point_0.4.var after download

 

[Faco2000]

@AshAuryn @SlimerJSpud Many thanks for taking care of it ! If I understand well it only has an impact on the scanning, not necessarily on the functioning of the vars calling these .latest dependencies ?

I've tried a smaller VAM instance with the same KM465 var and it's much better (only 1 missing dep vs >10) :

Is there a simple way to identify which var is calling for a wrong or outdated dependency like "sexpressions.1" ? If I could track such kind of mistakes on vars of creators I care about and who are still active, I will inform them and clean my Var folder until an update is performed.

 

[AshAuryn]

I just uploaded a fix for this to our server. You should scan for packages again. Many should now correctly show a download link if they are available.

 

[AshAuryn]

TLDR; Requests for non-existent versions were causing all other legitimate requests for "latest" or a correct version to also show "not on hub". i.e. one bad dependency and the whole scan was corrupted.

Our scan has a de-duplication section that removes duplicates so that you don't end up having to download the same var twice.

On the website, and in certain places in-game, if a version doesn't exist then the Hub will serve the latest version as an alternative. However, specifically in the "scan hub for missing packages", dependencies with specific versions (instead of "latest") are hard coded to return "not on hub" if they aren't available. The "specific version" thing was originally intended mainly for plugins, where downloading an alternative version would likely break the plugin. Today, there are all kinds of reasons a creator might use "specific" instead of "latest", but development has ended for VaM 1.x.

In any case, what was happening:

1. You scan for missing packages and VaM sends a request for something like AshAuryn.Sexpressions.1 (doesn't exist), AshAuryn.Sexpressions.2 (valid), AshAuryn.Sexpressions.5 (latest version), and AshAuryn.Sexpressions.latest (this would be v5 as well)
2. The API would try to find v1 first, and since it doesn't exist it would offer v5 instead. It would also offer v2, v5 (again since it was specifically requested), and v5 one more time (because latest was requested)
3. Then, so that you don't end up downloading v5 three separate times, it would remove the duplicates of version 5 (it would remove the request for "Sexpressions.5" and the one for "Sexpressions.latest") leaving only the "Hey you requested v1 and it doesn't exist so here's v5 instead"
4. Then, in-game, VaM's old code would say "Hey, v1 doesn't exist, so I'm going to say "not on hub" and delete this file", and since the duplicates were already removed, and vam has removed the file, everything that needs v5 now says "not on hub"

The solution was to track mismatches and make sure we're keeping the information about specific versions that don't exist, but also serving at least one "legitimate" copy of the latest version if there are other vars that also need it.

@Faco2000 -- For the ones that still say "not on hub" like the vars that are weirdly depending on "Sexpressions.1" which doesn't exist, you can try unzipping the var into a folder, editing the meta.json file, and changing the "1" to "latest". Re-zip it and make sure the file ends in .var and not .zip, then try to load the scene. This may or may not repair that var. For a more in-depth repair, you may need to run all of the scene and preset files through a text-editor search pattern and replace all instances of "Sexpressions.1" with "Sexpressions.latest". Obviously only try this if you really care about that scene and there's no replacement.

 

[AshAuryn]

Is there a simple way to identify which var is calling for a wrong or outdated dependency like "sexpressions.1" ? If I could track such kind of mistakes on vars of creators I care about and who are still active, I will inform them and clean my Var folder until an update is performed.

Well, to identify which dependency is not available, you can look at the list the scan provides.
If the dependency has a version number, and the license says "null", then it was a specific version that doesn't exist.

If the dependency says "latest" and "not on hub" then it's simply not on the hub, but the dependency is correct.

After you find all of these "null" references, finding which vars are requesting them would be a bit trickier. You'd probably need a script that could go through your addonpackages folder and unzip each file, read the meta.json, and let you know if it finds that problem dependency. That would be a simple script, but for a large folder it would take a very long time to run and would probably eat system resources while it's running because of the constant extraction and cleanup of large files.

 

[SlimerJSpud]

I use this Python script to scan existing vars for dependencies. You'd still need to look through the results to make sense of it all. The concept of a reverse lookup is completely missing in Vam. Another way is to go to the affected resource on the web and look at its Dependency Usage list. For some plugins, that will be a massive list, tho.

Script for listing all dependencies

For a while I've been thinking of making a script that can list all dependencies in all vars and presets, or list which vars depend on a specific var. After getting inspired by le_hibous pyhton script (https://hub.virtamate.com/threads/too-many-heap-sections.38525/#post-108718), I finally got...

hub.virtamate.com

 

[Faco2000]

Well, to identify which dependency is not available, you can look at the list the scan provides.
If the dependency has a version number, and the license says "null", then it was a specific version that doesn't exist.
View attachment 517974

If the dependency says "latest" and "not on hub" then it's simply not on the hub, but the dependency is correct.

After you find all of these "null" references, finding which vars are requesting them would be a bit trickier. You'd probably need a script that could go through your addonpackages folder and unzip each file, read the meta.json, and let you know if it finds that problem dependency. That would be a simple script, but for a large folder it would take a very long time to run and would probably eat system resources while it's running because of the constant extraction and cleanup of large files.

Thanks a lot but I don't think I'll go for such a huge journey

I put @JayJayWon in the loop as he may be interested as well and maybe the browserassist plugin has something to do with my issue...

I've found something interesting :
- on a small VAM instance, I've focused on one missing dependency "SPQR.foundation.latest" as shown by jayjaywon browser assist while filtering the ressource type "Var packages" and looking at each VAR package with missing dependencies

- a 1st strange thing was that "SPQR.foundation" was not seen as a missing referenced package by the native scanner of the home page (while going in the VAM Hub section)
- then I've searched for SPQR foundation in my var folder and I've seen 2 copies of the same var "SPQR.foundation.2"
- I removed 1 of them and it was solved. Jayjaywon browser assist tool was now showing an existing "SPQR.foundation.2" instead of a missing "SPQR.foundation.latest"

but unfortunately, the missing packages with .latest extension shown by the native scanner cannot be solved the same way... herebelow a screenshot of the missing packages of this small VAM instance, as you can see, the scanner even shows twice the same missing packages with .latest extension :

 

[Faco2000]

I use this Python script to scan existing vars for dependencies. You'd still need to look through the results to make sense of it all. The concept of a reverse lookup is completely missing in Vam. Another way is to go to the affected resource on the web and look at its Dependency Usage list. For some plugins, that will be a massive list, tho.

Script for listing all dependencies

For a while I've been thinking of making a script that can list all dependencies in all vars and presets, or list which vars depend on a specific var. After getting inspired by le_hibous pyhton script (https://hub.virtamate.com/threads/too-many-heap-sections.38525/#post-108718), I finally got...

hub.virtamate.com

Sounds exactly what I'm looking for, unfortunately, I have many skills but Python for me is only a snake... I don't know what to do with it

 

[SlimerJSpud]

To run a Python script, you need Python installed. Go to https://www.python.org/downloads/ and download the Windows version. Then you can use Windows Powershell to run the above script.

cd c:\games\vam (or wherever your Vam is)
python List_Deps.py -p . -n km465 (The -n argument is just some string to match vars to.)

 

[SlimerJSpud]

The script tells you what vars might depend on some other var:

List_Deps.py -p . -n cwfz
The following var is not used as a dependency in other vars:
huaQ.CWFZ_143.1.var


The following 1 iteration of 'cwfz' has other vars that depend on it:
huaQ.CWFZ_143.latest ->
Xspada.Nicoletta2@Pub.1.var