I gave up using vars as the system is broken and encourages massive duplication. Now I just unpack them. saved a lot of disc space and my load times are dramatically reduced.
Sad but true... and this is even the at least worse methode, compared to the old sharing solutions.
If it is only for the very slow file browser when opened from within VaM: Do create subfolders right from the start, so you don't break your own scenes when doing it later. The more files (with preview thumbnails) you have in a folder, the longer VaM will need to open it (says the man with > 200 FaceGen face textures in one single folder) .
If it is for the overall VaM starting time: do a spring clean and remove unneeded stuff to a spare drive, use a SSD if possible, check your VaM caching, do advanced and a little critical things like unpacking VAR files... or just live with the slow starting speed, get yourself a cold beverage in the meantime and be happy that we have so much VaM content now.
I don't expect this to be fixed until the future VaM 2.x will be ready.
Yeah I've already accepted the slow startup at first is just how it's going to be for now. The file browser is just so insanely slow, I can't do anything while I'm in VR. I considered unpacking the VARs as well, but scenes link directly to them.
It's a mess and TToby is right. I just need to back up and clean everything out.
I gave up using vars as the system is broken and encourages massive duplication. Now I just unpack them. saved a lot of disc space and my load times are dramatically reduced.
I don't know if unpacking will speed up the starting time, but I don't see how it'll save disk space nor fix duplication.
VAR are made to avoid duplication. Sure, some are poorly made, but at worst it references unneeded var (it's not duplicate content: it doesn't add anything except a few text lines in a JSON).
And if they are repacking stuff that should be referenced in VAR (which should not be done): well it's just like the old way before the VAR system. This creates duplicates, but unpacking them won't solve the issue.
And if you want to make a VAR some day, you'll repack the stuff you've extracted: very bad practice, as we just said.
If it is only for the very slow file browser when opened from within VaM: Do create subfolders right from the start, so you don't break your own scenes when doing it later. The more files (with preview thumbnails) you have in a folder, the longer VaM will need to open it (says the man with > 200 FaceGen face textures in one single folder) .
.re 'create subfolders' - do you mean subfolders in the AddonPackages folder or in VaM's 'regular/legacy' directories? (Save/scenes/... Custom/Person/Appearance/... etc).
Bcs I've structured my AddonPackages folder to death (10 main categories like Plugins, Clothing, Scenes-looks, Scenes-not-looks - and within each of those, I sorted everything by creator name), and it still takes ages to access stuff with VaM's filebrowser.
Interestingly, VaM ignores sub directories in AddonPackages. The only benefit to that is scenes don't have to keep up with a specific location.
Logically, the file browser should see a benefit to not stuffing a bunch of textures in one folder - having to render the images and process the items at once. But what I've found is it's usually faster to keep all of my textures for a character in the same folder vs. navigating to different folders...
@Case In the Addon Packages folder, you have the chance to create subfolder at any time, as VaM ignores the folder structure in that single specific folder. This will be good for sorting and finding things and maybe therefore a little bit for browsing speed, too.
In all the other folders, you have to think about creating subfolders first before building your own scenes, or you might break them once you have used the content. In folders with preview thumbnails, browsing will be strongly delayed by the processing speed of those previews. This is especially true for the folders where you will save your own stuff... scenes, looks, presets, textures, aso. This is the cause why meshed has build in (last version) that automatic page creation once you have reached a certain number of entries.
My example with the FaceGen face textures is a very extreme one, because I usually convert all my good FaceGen face textures to be used with the same set of my own few standard body textures. Opening that folder will take up to 2 minutes once in a session, now. For mentaly healthy users, the problematic folders might be most likely the preset folders and the scenes folder. Not now, but maybe later if you have collected a big pile of personal stuff. So, timely thinking about a subfolder structure might not be a bad idea.
@Case In the Addon Packages folder, you have the chance to create subfolder at any time, as VaM ignores the folder structure in that single specific folder. This will be good for sorting and finding things and maybe therefore a little bit for browsing speed, too.
In all the other folders, you have to think about creating subfolders first before building your own scenes, or you might break them once you have used the content
Thank you for the detailed reply (I'm aware of the issue of potentially breaking references. By myself, I call it "VaM lock-in" (hat-tip to Jaron Lanier)).
Some more questions, if I may:
My point is more: Have you noticed a measureable gain in terms of filebrowser speed when you structured your non-addonfolder resources (structuring = subfolder structure as detailed in your above posts)? I'm asking bcs I've long pondered whether my non-var resources might be the problem, and how to address it (Again: Assume that I'm aware of the issue of potentially breaking references).
And if so, how did you compare? You self-deprecatingly refer to your mental health - yet even so, I have a hard time believing anyone would go to the substantial trouble of creating two different (non-trivial) VaM installations that differ only in their directory structure. Yet that would be precisely what is necessary to make a quantifiable comparison - a fresh install will be faster due to lower number of resources alone? (if I'm not being an idiot ,,, again ...)
For mentaly healthy users, the problematic folders might be most likely the preset folders and the scenes folder. Not now, but maybe later if you have collected a big pile of personal stuff.
Let's say I've collected a big pile of personal stuff a long time ago ...
I agree the scenes folder may have a large influence - I'm less sure about the preset folders (I would rather have pointed to the textures folder as a potential issue).
My intuition is that filebrowsing speed may also strongly depend on what VaM updates and when - if it scans all subdirectories (which I suspect it does in many instances), then a nice orderly subfolder structure won't help. I don't have hard evidence, but my impression is that VaM tends to be a bit OCD-ish about resources, and that it tends to re-scan them ... basically any chance it gets, no matter whether the action makes sense or not.
I wondering whether another (possible) drawback of an elaborate filestructure is the trade-off between scanning speed and directory-tree depth. Every successive level of directories also means one more click, and one more scan. A large folder without any subfolders may take long time to scan, but in the end, this might take up less of your time than clicking through four layers of directories?
But this is all just speculation on my part - it depends on whether VaM scans subfolders and when (and I think it does, every chance it gets).
I thnk we have to define two different things:
Startup or loading speed vs browsing speed.
First - the VaM startup speed. This is definitely depending on how many stuff you have. A blanc and newly installed VaM will start up faster. I am not completely sure which files (and therefore which folders) are critical. AFAIK addon packages and morphs are being scanned at start, but what else? I know that MeshedVR has tried out many things to make this startup files scanning faster over multiple VaM versions.
I don't think we can speed up the startup with just organizing, but with sorting stuff out and keep VaM as clean as possible.
If other people are speaking about unpacking the addonpackages, this is because to spare the very little time for auto-processing the zip files.
Second - browsing speed. If you will open up a file browser in VaM, it takes some time to process the thumbnail images and to list up stuff. The more entries, the slower it gets. At this point a good folder structure will help a lot. Not too deep strictures, so you don't have to do multiple clicks to find your stuff. You have to find a balance.
You most probably won't see a difference with only twenty-something entries, but definitely with 200 and more files.
At this point the relatively newly added VaM function will jump in and it creates automatically some pages if you have reached a certain number of entries. You can browse through the pages (if there are some) at the top of the browser menu. Till now, I have seen that function only in my looks save folder, where I had up to 200 files or more. I don't have seen this behavior in my large texture folders. Nevertheless, I wouldn't advice you to wait till this function jumps in, because it will still keep a big number of files to process on each of those pages.
A big issue with creating a subfolder structure is IMHO, that you can't do a search over subfolders!!
Personally I see the browsing speed more critical than the loading speed, because in VR your framerate is going down drastically when the file browser is working hard and it starts to stutter. If VaM starts up slow, I just can do other things in the meantime.
To say it clearly, this is only some nitpicking. You still can run VaM very good without doing any of those actions. Yes, it is slow sometimes, but I know several other tools and games, that are even slower.
What is your load time and Addonpackage size? Recently I cleaned mine of unused .var and got it down to 77gb my start up time is 17sec in VR, I upgraded to a Samsung 980 m.2 so I think that helped. I can browse my 1.2k scenes fine while already being in a scene too but if I scroll down super fast the last thumbnails take a sec to load
* Afaics, packages are not scanned after startup UNLESS you manually trigger a hard reset (possibly also when you trigger a rescan in the packagemanager menu, but that feature had a bug a few versions back & I didn't check whether it was fixed). You can test for yourself by Alt+Tab-ing out of VaM after startup & adding a .var - filebrowser won't show it until you manually trigger a rescan.
* Not sure exactly under what conditions morph .var packages are checked after startup - as you said, Meshed did quite some work on morph management (it's by far the most mature part of VaM's resource management), and there may be some conditions when VaM will rescan packages explicitely marked as morph packs without user input.
* I've noticed that VaM only flags missing morph deltas of .var packages in the errormessages (the red screen) the first time I load a scene containing a person atom. Note: Not the scene that is missing a morph delta - loading any scene containing a person atom will trigger the errormessage. This may indicate that VaM checks the integrity of certain classes of resources only when they're needed - but still, it appears to check all the var packages that contain the respective class of resource.
* Agreed that speeding up the startup loading time is a lesser concern - btw. according to my log file, VaM needs about 20 seconds to scan my 7.400 addon packages on startup (RyZen 3800x, 570 chipset mainbord, SATA 1TB m.2 SSD drive). Given Williamwow123's reporting (post above) that their rig takes only three seconds less to scan a significantly smaller installation on a slightly faster drive, I agree that trying to get startup time down is probably neither feasible nor particularly important.
If you will open up a file browser in VaM, it takes some time to process the thumbnail images and to list up stuff.
<...>
At this point the relatively newly added VaM function will jump in and it creates automatically some pages if you have reached a certain number of entries. You can browse through the pages (if there are some) at the top of the browser menu. Till now, I have seen that function only in my looks save folder, where I had up to 200 files or more.
1) What is this 'automatic page creation function' you mentioned? Could I impose on you to make some screenshot to illustrate? I've gone through the last four changelogs and couldn't find anything similar.
2) What presets are we talking here - the Legacy .json presets (Save/appearance , Save/General ...) or the newer .vap presets? I haven't noticed anything like you describe browsing the 'new' appearance preset files (of which I have more than 1000?)
3) Not to nitpick, but I think it'd also be good to specify whether we're talking about browsing .var resources OR browsing unpacked resources (i.e. files in saves or custom subfolder).
Please note that I'm not doubting you, and that I'm only asking because I'm curious whether & how I could speed up my VaM by working on my non-var resources.
I am limited to Gen 3 m.2 as i am using a i9 10850k I bought the nvme 980 PRO but it is incompatible with 10th series and only work with Intel's 11th, the standard 980 has a read 2500 write 2600. The pro has 7k Read speed but sadly I had to return it
@Case
Here is a screenshot... it is funny that I now can see the "Limit" slider clearly. Never saw it before... lol.
Default seems to be 450, so you will only see this function in action if you have more than 450 files in a single folder.
With 1722 in my looks-folder, I have 4 "pages" to leaf through.
Why is this all dynamically generated instead of a local indexed DB that can be refreshed if you make manual additions vs the hub? Seems a pretty odd approach for an application that is designed for continuous creation.
I am limited to Gen 3 m.2 as i am using a i9 10850k I bought the nvme 980 PRO but it is incompatible with 10th series and only work with Intel's 11th, the standard 980 has a read 2500 write 2600. The pro has 7k Read speed but sadly I had to return it
This seems to be only a function of minor importance and nothing special, but it was added in one of the later versions to solve exactly those issue with the slow file browser. But it is still way too slow with 450 files per page (even on a G4 m.2)... So I would suggest to sort out things in folders like I said before, or, after I have realized that "limit slider", I would suggest to set this slider down for lazy people (like me).
According to the German words of wisedom "Everybody is useful... at least as a bad example!",
I am a little bit proud that MeshedVR added this function after a short chat with me on Discord about the slow browser plus the stuttering in VR, and after I told him I had that much files in some of my folders. His reaction was something like "O.M.G."
This seems to be only a function of minor importance and nothing special, but it was added in one of the later versions to solve exactly those issue with the slow file browser. But it is still way too slow with 450 files per page (even on a G4 m.2)... So I would suggest to sort out things in folders like I said before, or, after I have realized that "limit slider", I would suggest to set this slider down for lazy people (like me).
Thanks, I already did that, and the filebrowser 'feels' a little more reactive now. Emphasis on 'feels' - I haven't taken measurements.
I'm wondering whether the 'paging' only affects what is shown in the filebrowser window, or also the scanning process itself - i.e. if VaM only scans the first/second/third etc. 450 vars once they are requested.
Same with the 'hide' option - does it hide vars only from me, or does it also scan hidden packs less often?
If the latter, hiding some resources might be a good way to speed up filebrowsing?
According to the German words of wisedom "Everybody is useful... at least as a bad example!",
I am a little bit proud that MeshedVR added this function after a short chat with me on Discord about the slow browser plus the stuttering in VR, and after I told him I had that much files in some of my folders. His reaction was something like "O.M.G."
Thanks for raising the issue with MeshedVR - although I have to say I'm a little shocked that he wouldn't have a better intuition about the kind of installations his users are running. These problems are far from rare and complaints have been far from rare. Note I'm not accusing Meshed of not caring about his userbase - quite the opposite - but if the dev team is mostly experiencing VaM by running quasivirgin installations of VaM on monster-rigs, they may not have a good intuition about many of problems that are quite common with more ordinary installations & hardware. A lot of the issues we're raising here are issues of 'scaling badly' (sorry, don't know a better term).
p.s.: Wasn't it rather "Nobody is entirely useless - they can still serve as a chilling example"?
I'm wondering whether the 'paging' only affects what is shown in the filebrowser window, or also the scanning process itself - i.e. if VaM only scans the first/second/third etc. 450 vars once they are requested.
Hi, this is why I wanted to draw a line between "VaM startup speed" and "in-game browsing speed". IMHO only the content with a certain file formate in only some certain folders will be scanned at the VaM startup process once in a session. This will slow down the startup even more the more content you have. If you will open a scene, VaM will load all the stuff referenced in that scene and maybe (!) will scan some files, too. This is an other reason of VaM being somewhat slow.
An IMHO completely other thing is the VaM in-game browsing. If you open up the file browser menu, VaM "scans"the folders you are about to open, too, but that is the typical scanning of a file browser when goin through the content of the related folder, and has IMHO only a little (or nothing) to do with the startup scanning process.
Which files are being scanned at the VaM startup is a technical question, that maybe can only be answered by MeshedVR. I don't think this will have anything to do with the file browser filters you will set and is primary done only one single time per session at startup. AFAIK this are not all files, but only some certain files in some certain folders: for instance, in the morph-folders not the morphs as such (.dsf/.vmb) are being scanned any more, but those additional auto-generated small .vmi files. This was added to speed up the slow startup.
This is quite common in game/app development and nothing you can blame them for (with exeptions). I was into this harsh buisiness a decade ago, and you are most of the time working on and thinking about issues that arose from your own development process. You can't even think about all the silly ideas some users will have "out in the wild" and how they are bringing your poor programm at it's limits. This is the cause most big companies have their expensive testing departments, and that is the cause why small developers are doing some open beta-testing with the users being the testers. VaM is officially in its beta-state. Therefore it is important to report the bugs and issues we have. But due to VaM 1.x is not being supported anymore, let's focus on testing the future VaM 2.x out to the maxx
.
