- Messages
- 20
- Reactions
- Points
- 3
For a while I've been thinking of making a script that can list all dependencies in all vars and presets, or list which vars depend on a specific var. After getting inspired by le_hibous pyhton script (https://hub.virtamate.com/threads/too-many-heap-sections.38525/#post-108718), I finally got down to making it.
The main functionality is the following:
python "<Path_to_file>\dependencyScanner.py" -p "<VaM folder path>" [-n "<name_of_var>"] [-o ["<output_file>"]]
name_of_var can also be a partial name, the script will simply list every var that contains the part in its name.
A second function is that when you leave out -n "<name_of_var>", it will list every var in the folder that isn't depended on by any other var.
It is now also possible to save the results to a file, using the -o flag. If no name is given to the flag, it defaults to "SearchVarOutput.txt"
In case you have unzipped vars in the AddonPackages folder, it should catch those too.
I would like to make it into a plugin at some point, but that's a whole other matter.
Thoughts and feedback?
Example with -n ascorad
Edit: added the option of saving the result to a file
Edit2: should now also catch packages stored as folders with meta.json inside them
The main functionality is the following:
python "<Path_to_file>\dependencyScanner.py" -p "<VaM folder path>" [-n "<name_of_var>"] [-o ["<output_file>"]]
name_of_var can also be a partial name, the script will simply list every var that contains the part in its name.
A second function is that when you leave out -n "<name_of_var>", it will list every var in the folder that isn't depended on by any other var.
It is now also possible to save the results to a file, using the -o flag. If no name is given to the flag, it defaults to "SearchVarOutput.txt"
In case you have unzipped vars in the AddonPackages folder, it should catch those too.
I would like to make it into a plugin at some point, but that's a whole other matter.
Thoughts and feedback?
Example with -n ascorad
Code:
The following 7 vars are not used as a dependency in other vars:
ascorad.Adela_Scene.1
ascorad.Amia_Scene.1
ascorad.Ariadne_-_Moving_Day.7
ascorad.Enjoying_Evey.2
ascorad.Haley&Tess_Heads_Will_Roll_v2.5
ascorad.Haley_Armageddon_5.29
ascorad.Katia.2
The following 8 iterations of 'ascorad' has other vars that depend on it:
ascorad.Amia.latest ->
ascorad.Amia_Scene.1.var
ascorad.Ariadne_Look.latest ->
ascorad.Ariadne_-_Moving_Day.7.var
ascorad.asco_Expressions.latest ->
ascorad.Amia_Scene.1.var
ascorad.Ariadne_-_Moving_Day.7.var
ascorad.Enjoying_Evey.2.var
ascorad.Katia_Scene.2.var
ascorad.ascorad_Haley_v5_2.latest ->
coll69.Haley_At_Your_Service.12.var
ascorad.Breast_Morphs.latest ->
VamTimbo.SexyDance_5-17-23.5.var
ascorad.Livie.latest ->
ascorad.Vivie_&_Livie_Scene.4.var
ascorad.Pella.latest ->
ascorad.Vivie_Scene.4.var
ascorad.Preset_Adela.latest ->
ascorad.Adela_Scene.1.var
Python:
#!/usr/bin/python3
from zipfile import ZipFile
from glob import glob
from os.path import join, split, splitext, exists, basename, isdir, dirname
import sys
import argparse
from pathlib import Path
from json import load
import re
def dpPrint(text: str, toFile: str = ''):
print(text, flush=True)
if toFile:
try:
with open(toFile, 'a', encoding='utf-8') as f:
f.write(text + '\n')
except Exception as error:
print(f"ERROR, failed to write to {toFile}: {error}", file=sys.stderr, flush=True)
return
def check_name_variation(varName, dependenciesList):
parts = varName.split('.')
if parts[-1].isdigit() or parts[-1] == 'latest':
baseName = '.'.join(parts[:-1])
else:
baseName = varName
return varName in dependenciesList or f'{baseName}.latest' in dependenciesList
def getDependencies(varFile: str) -> tuple[Exception|None, set]:
try:
with ZipFile(varFile) as myzip:
with myzip.open('meta.json') as metaJson:
data = load(metaJson)
myzip.close()
if 'dependencies' not in data:
return None, set()
return None, set(data['dependencies'].keys())
except Exception as error:
return error, set()
# Get all dependencies listed in presets
def getPresetDependencies(customPath: str) -> dict:
allDependencies = {}
presets = sorted(glob(join(customPath, 'Custom', '**/*.vap'), recursive=True))
for vap in presets:
try:
with open(vap, 'r', encoding='utf-8') as f:
pattern = r'"([^"]+):/[^"]*"'
occurences = set(re.findall(pattern, f.read()))
except Exception as error:
dpPrint(f"ERROR, failed to read: {vap}: {error}", args.output)
continue
parts = Path(vap).parts
presetName = '/'.join(parts[parts.index("Custom")+1:])
dependenciesDict = {key: {presetName} for key in occurences}
for key, val in dependenciesDict.items():
if key in allDependencies:
allDependencies[key].update(val)
else:
allDependencies[key] = val
return allDependencies
# Collect all vars, a complete list of all dependencies, and which vars uses which dependency
def getAllVars(directoryPath: str) -> tuple[set, dict]:
allDependencies = {}
vars = glob(join(directoryPath, 'AddonPackages', '**/*.var'), recursive=True)
allVarList = {splitext(basename(var))[0] for var in vars}
for varFilename in vars:
if isdir(varFilename):
error, dependencies = getMetaDependencies(varFilename)
else:
error, dependencies = getDependencies(varFilename)
if (isinstance(error, Exception)):
dpPrint(f"ERROR: {varFilename}: {error}", args.output)
continue
if (len(dependencies) == 0):
continue
dependenciesDict = {key: {split(varFilename)[1]} for key in dependencies}
for key, val in dependenciesDict.items():
if key in allDependencies:
allDependencies[key].update(val)
else:
allDependencies[key] = val
return allVarList, allDependencies
# Get all dependencies listed in meta.json stored directly in folders
def getMetaDependencies(folder: str) -> tuple[Exception|None, set]:
filename = join(folder, 'meta.json')
try:
with open(filename, 'r', encoding='utf-8') as metaJson:
data = load(metaJson)
if 'dependencies' not in data:
return None, set()
return None, set(data['dependencies'].keys())
except Exception as error:
return error, set()
argParser = argparse.ArgumentParser(prog='DependencyScanner.py', description='Searches for dependencies inside all vars in a folder and if a name is given, lists whether there a vars that depend on it. If no name is given, it will instead list all vars in the folder which isn\'t a dependency of any other var.')
argParser.add_argument("-p", "--path",
type=Path,
default='.',
help="Path your VaM folder")
argParser.add_argument("-n", "--name",
type=str,
default='',
help="Name of a var file. Can be a partial name")
argParser.add_argument("-o", "--output",
type=Path,
nargs='?',
default=None,
const='.\\SearchVarOutput.txt',
help="File to save the results in (default: '.\\SearchVarOutput.txt')")
args = argParser.parse_args()
# Check the given arguments are valid
if not exists(args.path):
print(f"The given path doesn't exist: {args.path}")
exit(1)
if args.output:
if isdir(args.output):
print(f"The given output is a folder, not a file: {args.output}")
exit(1)
if not isdir(dirname(args.output)) and not basename(args.output) != '':
print(f"The given output file folder path doesn't exist: {args.output}")
exit(1)
open(args.output, 'w', encoding='utf-8').close()
parts = args.name.split('.')
if parts[-1] == 'var':
args.name = '.'.join(parts[:-1])
# Get a list of all vars and a dict of all dependencies from vars
allVarList, allVarDependencies = getAllVars(args.path)
allDepVars = list(allVarDependencies.keys())
# Get a dict of all dependencies from presets
allPresetDependencies = getPresetDependencies(args.path)
allDepVarsPresets = list(allPresetDependencies.keys())
# Combine all dependencies
allDependencies = allVarDependencies | allPresetDependencies
# Get a list of which vars in the folder that are not used as a dependency
noDependencyVar = [var for var in allVarList if not check_name_variation(var, allDepVars)]
noDependencyPreset = [var for var in allVarList if not check_name_variation(var, allDepVarsPresets)]
noDependency = sorted(list(set(noDependencyVar) & set(noDependencyPreset)), key=str.lower)
# If no name is given, list all vars that aren't used as a dependency
if args.name == '':
dpPrint(f"{len(noDependency)} vars are not used as a dependency:", args.output)
for var in noDependency:
dpPrint("\t" + var, args.output)
if args.output:
print(f"\nResults saved to: {args.output}")
exit(0)
# List all vars with the given name that aren't used as a dependency
foundVars = [var for var in noDependency if args.name.lower() in var.lower()]
if len(foundVars) > 0:
dpPrint(f"The following {str(len(foundVars)) + ' vars are' if len(foundVars) > 1 else 'var is'} not used as a dependency in other vars:", args.output)
for var in foundVars:
dpPrint("\t" + var, args.output)
# List all vars with the given name that depend on other vars
foundDepVars = sorted([var for var in allDependencies.keys() if args.name.lower() in var.lower()], key=str.lower)
if len(foundDepVars) > 0:
dpPrint(f"\nThe following {len(foundDepVars)} iteration{'s' if len(foundDepVars) > 1 else ''} of '" + args.name + "' has other vars that depend on it:", args.output)
# print all dependencies where the name is a key, or the name is part of the key
for key in foundDepVars:
if args.name.lower() in key.lower():
if args.name != key:
dpPrint(f"{key} ->", args.output)
for var in sorted(allDependencies[key], key=str.lower):
dpPrint("\t" + var, args.output)
if args.output:
print(f"\nResults saved to: {args.output}")Edit: added the option of saving the result to a file
Edit2: should now also catch packages stored as folders with meta.json inside them
Last edited:
Thanks!
