VaM's cache: SSD vs HDD vs none - results of rudimentary testing

[glodomorrapl]

I've noticed my VaM's cache nearing the 100 GB size threshold, and I started wondering if trading this much storage was worth the loading time benefits. So with nothing better to spend my free time on, I tested it myself.

Initially I wanted to just test "cache on vs off", but when I was searching through forums trying to find out what others have found out about VaM's caching, I have also found several claims that HDDs are just as fast for this purpose as SSDs are. I decided to expand this test with some cache-on-HDD measurements, since I was curious.

The test done on my main and only VaM instance, game version 1.22.0.3, with 1313 VARs taking 59,5 GB of my storage space. The game is stored on an SSD.

Hardware used for the test: Ryzen 7 5800X3D CPU, RTX 3070 8GB GPU, 32GB RAM (2x 16 GB 3600 MHz sticks running on their XMP profile), 2TB NVMe Gen3 SSD with DRAM cache and a 2TB 5400RPM consumer-grade HDD.

Some more notes about the test:

• Before measurements are taken for each of the cache configuration/location, I completely power off my PC, boot it back up, close all unnecessary programs, and give it about 3 minutes to settle. This is done to wipe all sorts of caches outside of VaM that would skew the results, whether that's Windows's own cache, or drives' own. Spoiler alert: this is why HDD results might surprise you (not clickbait).
• Measurements are done by recording the testing process with a phone (instead of using something like OBS, to avoid unnecessary load on the PC) and measuring the time between the mouse click initiating the action and the last frame where any loading of sorts is visible (eg. when the red hourglass disappears). If CUAs are involved, CUAManager's loading window is also considered, although "post-waiting" is not counted as "still loading". Also because recorded videos were 60 fps, a measurement error up to 16,67 ms should be assumed for each result here, and because my video player apparently struggles to reliably show milliseconds, add another 3 ms of error on top of that.
• Percentage difference is always shown compared to the SSD cache.
• For the "cached" test, a fresh cache with just the resources used in this test is used (initially I wanted to use my current 90 GB cache, but it turned out using the "Change Location" button in VaM also purges your cache before changing its location, so for the sake of consistency I redid the test with the same, slim cache). For the "no cache" test, the cache has been completely disabled.
• I was wondering whether or not to restart VaM after every test to purge whatever assets it had already loaded in memory, but decided against it in favor of simulating a situation that I consider to be closer to reality for more VaM users (that is, jumping between scenes without restarting the game, unless I'm the weird one for not doing that).
• This is a rudimentary test, as title states. Differences in use-case, software/hardware configuration and testing methodology/thoroughness might yield different results. Take anything you see here at face value. These results are good enough for my purpose, I'm posting them here in case someone here finds them interesting. Hell, maybe these results might motivate someone to make a better test, learning from my mistakes.

---

Test 1: Game start time. Measured from double-clicking the game's shortcut to a fully responsive menu.

SSD cache	HDD cache (diff against SSD cache)	No cache (diff against SSD cache)
23.585 s	35.627 s (+12.042 s / +51.1%)	21.057 s (-2.528s / -10.7%)

I guess VaM's cache needs to be processed before the program launches, and having the cache on a hard drive shows a major increase in the boot-up time.

Test 2: Loading female appearances into the default scene.
Cycle between:
- SupaRioAmateur's Kaine (no clothes, CUA hair)
- Zin's Tifa (12 clothing items [full set for the look], 7 CUA colliders [CUA hair replaced with non-CUA hair])
- dnaddr's Belle (6 clothing items, no CUAs)
The cycle was immediately repeated once. Additionally, the time required to open the appearances menu has also been noted.

	SSD cache	HDD cache (diff against SSD cache)	No cache (diff against SSD cache)
First load of appearance preset #1	3.193 s	9.762 s (+6.569 s / +205.7%)	21.539 s (+18.346 s / +574.6%)
First load of appearance preset #2	3.326 s	6.987 s (+3.661 s / +110.1%)	14.410 s (+11.084 s / +333.3%)
First load of appearance preset #3	3.443 s	8.124 s (+4.681 s / +136.0%)	28.865 s (+25.422 s / +738.4%)
Second load of appearance preset #1	1.480 s	1.779 s (+0.299 s / +20.2%)	3.964 s (+2.484 s / +167.8%)
Second load of appearance preset #2	1.514 s	0.987 s (-0.527 s / -34.8%) *	7.035 s (+5.521 s / +364.7%)
Second load of appearance preset #3 **	0.749 s	0.698 s (-2.528 s / -10.7%)	0.633 s (-0.116 s / -15.5%)
Time to open the appearance selector menu	0.316 s	0.325 s (+0.009 s / +2.8%)	0.316 s (0 s / 0 %)

* ??? idk you tell me, that's how fast it loaded, this one's an anomaly of sorts
** second loads of preset #3 were so immediate there was no loading bar/red hourglass to go off of, so by "loaded" I assumed the very first frame the loaded look was visible on.

Test 3: Loading some scenes.
A set of test scenes loaded one after another:
- "light" scene: KittyMocap's "KM308 Evening sex"
- "moderate" scene: Zinigo's "First Date"
- "heavy" scene: TGC's "Virtual Sweetheart Reloaded", the CG-Part only scene
For the "heavy" scene, the "loaded" condition happens when the last freeze goes away, since I guess that scene still has some stuff to load even after VaM claims it's done loading.

	SSD cache	HDD cache	No cache
"Light" scene	26.030 s	34.024 s (+7.994 s / +30.7%)	55.957 s (+29.927 s / +115.0%)
"Moderate" scene	30.566 s	44.840 s (+14.274 s / +46.7%)	58.725 s (+28.159 s / +92.1%)
"Heavy" scene	106.306 s	103.407 s (-2.899 s / -2.7%) *	116.827 s (+10.521 s / +9.9%)

* oh look, another anomaly!

---

Summary​

• tl;dr of the results:
  Cache has no impact on appearance preset menu launch time.
  Loading looks for the first time since launching VaM takes about 2.5x as long with an HDD cache, and about 6.5x longer without any cache.
  Loading looks for the second time since launching VaM takes roughly the same amount of time with an HDD cache, and about 2.7x longer without any cache.
  Loading light/moderate scenes takes about 1.4x longer with an HDD cache, and about 2x longer without any cache.
  Loading heavier scenes with a lot of atoms/non-material stuff (such as "Virtual Sweetheart Reloaded") severely shrinks the cache's impact on loading times, with this scene seemingly taking nearly the same amount of time to load on an HDD cache and just 1.1x longer without any cache. <-- due to the testing methodology that didn't involve game/system restarts between loaded scenes, this is somewhat speculative - the true impact on loading times of heavy scenes could be larger if you load such a scene immediately after game start; see below for details.
  Should you use VaM cache? Unless you're very patient, absolutely. Is cache on HDD as fast as cache on SSD? Absolutely not, but it's still far better than without any cache, so if you're short on SSD space, it'll do.
• HDD results: So why was an HDD cache so much slower in my tests? Well, my drive is 5400 RPM, so a faster 7200 RPM drive would yield better results, though I wouldn't suspect it to close the gap towards the SSD results in a meaningful fashion. But there's one more thing. As mentioned, "caches outside of VaM, whether that's Windows's own cache, or drives' own" exist, and their existence is something I was reminded of when I was loading appearance presets with an HDD cache (when doing a test run of the tests before noting down any results, without PC reboots in-between the cache location changes) and noticed how an entire appearance preset was loaded without me hearing the drive at all. Loading another preset and checking HDD usage in Task Manager confirmed my suspicion - some other non-VaM cache is severely improving loading times, to the point they become indistinguishable compared to having VaM's cache on an SSD drive. If you change your cache location in VaM, load some appearance presets to fill up that cache, restart VaM and try to load the same presets again, you will likely cause this situation to occur. Because of that, the results of HDD cache you see here are essentially the worst case scenario, and depending on how exactly you use VaM, your experience might be slightly, or severely better.
• "Heavy" scene results: As mentioned in the second last note above the test results, I did not restart VaM between the tests - I only restarted the whole computer between cache location changes. Because of that, VaM kept retaining more and more resources loaded in memory with each appearance preset and scene being loaded. Even though the second loaded scene has shown a significant difference between SSD and HDD caches similar to the first scene, the "heavy" scene was the very last test, which means in theory it may have had the highest amount of resources already loaded in memory. The low loading time difference in this scene may have been caused by that, or it may have been caused by tons of non-texture stuff being loaded, or perhaps a mixture of both. This is a potential major flaw of this testing procedure that renders the measurements of that scene inconclusive.
• Anomalies: The system/background application load has been under control for the entire duration of the test, so if anyone more knowledgeable about VaM than me has an explanation as to why they happened, I'm all ears. If I wasn't so lazy, I'd probably re-do the tests several times to reduce the impact of these anomalies.
• Video footage of the tests being run: I still have them. I might share them. But my upload speeds blow, so I'll probably do it only if someone persuades me it's a good idea to do so.
• Cache size: That's on topic of the laziness - yeah I'm too lazy to re-do the test with, say, a 100 GB cache. Sorry.
• Charts: They suck because they were made in LibreOffice. I will not elaborate further.

This whole thing is probably pretty far from a perfect benchmark, and pretty far from the best use of my time, but here, enjoy it anyway.

 

[DJ]

Interesting. I'm the guy who started, and has been pushing the idea that HD vs SSD caches performed about the same.

Something to keep in mind is that the Vam cache ONLY caches textures. What happens is that when you load a texture, Vam converts it to a GPU-ready format, then saves it to the cache. So the next time you load it, rather than convert it again, it just loads the GPU ready version from the cache. So the first time you load a texture after clearing the cache, there's no speed improvement, as there's no cached version to read. It's only subsequent loads that speed up.

When I did my testing (which found essentially no difference with the cache on a 7200 rpm HD and a M.2 drive), I tested by setting up a ridiculous clothing preset consisting of the first 3 pages of my clothing list, then recording load times for that preset. That way i was looking at 30 or 40 second load times, making the timing easier. When relying on me manually starting and stopping the stopwatch on my phone, i had little confidence I could accurately time events lasting only a second or two. With a clear cache, i loaded the preset to get all the items into the cache, then restarted Vam and timed how fast it would load the second time. Then I changed the cache location and repeated the procedure. I restarted Vam before every test. I'd be interested in what result you get using that test method.

Also note that Vam keeps unused textures in memory. And Vam also uses windows virtual memory system, which is as section of your C drive assigned to be treated as memory. At the very least, you should be purging the memory is user preferences between tests, or items you think are loading from the cache will actually be loading from RAM. I'm not sure, though, if purging memory does anything to windows virtual memory system. The fact you weren't restarting between tests makes me doubt your measurements. You may have been loading textures from RAM, not from the cache, in some of your tests.

You are testing things I never did, but I wonder if there are other factors creeping in when you get into scene loading. Or maybe you're uncovering real performance issues that my more limited testing regime never detected. I honestly don't know.

My interpretation was that the CPU is doing enough background stuff while loading textures that the load time from disk isn't the actual bottleneck. If you add up the size of each texture, then calculate how long it would take to load that amount of data from disk, it's a small fraction of the actual load times in Vam.

 

[glodomorrapl]

oh yeah, i do remember your name from searching the forums, thank you for taking your time to read this it's also how i found out what exactly is VaM's cache exactly caching, just as you said. it's also why both the caches i was testing on already had resources from tested looks/scenes already in them, that much i did understand.

With a clear cache, i loaded the preset to get all the items into the cache, then restarted Vam and timed how fast it would load the second time. Then I changed the cache location and repeated the procedure. I restarted Vam before every test. I'd be interested in what result you get using that test method.

yeah, that sounds exactly like what happened to me - load stuff to fill up my cache > VaM restart > second load, that's when i noticed VaM wasn't actually reading its own cache from the HDD and the difference in loading speed was essentially none, hence my full PC restarts in-between cache location changes.

Also note that Vam keeps unused textures in memory. And Vam also uses windows virtual memory system, which is as section of your C drive assigned to be treated as memory. At the very least, you should be purging the memory is user preferences between tests, or items you think are loading from the cache will actually be loading from RAM. I'm not sure, though, if purging memory does anything to windows virtual memory system. The fact you weren't restarting between tests makes me doubt your measurements. You may have been loading textures from RAM, not from the cache, in some of your tests.

i'm aware the scenes i was loading may have a bunch of shared materials that were already loaded into memory from previously loaded scenes, hence reducing the loading times. but i opted out for a more realistic scenario of subsequently loading scenes without restarting anything. and yet, that still showed substantial differences between ssd/hdd cache. well, at least until i loaded the VSR scene and seen little difference - you may be right that this result could have been severely impacted by all the resources already in memory from the previous two scenes. i'll note this in the OP.
let's just say i was focused more on real usage scenarios than stressing the cache as much as possible.

You are testing things I never did, but I wonder if there are other factors creeping in when you get into scene loading. Or maybe you're uncovering real performance issues that my more limited testing regime never detected. I honestly don't know.

My interpretation was that the CPU is doing enough background stuff while loading textures that the load time from disk isn't the actual bottleneck. If you add up the size of each texture, then calculate how long it would take to load that amount of data from disk, it's a small fraction of the actual load times in Vam.

from these anomalous two measurements i got, i also feel like there's some sort of unknown factor(s) in VaM that are throwing off the numbers. well, let's hope this incomprehensive test motivates someone smarter than me to get to the bottom of this

 

[RiftWind]

Definatly windows cache of some sorts - i used to run VAM on low memory and needed to close and reopen the program several times in a session. Noticed it took nearly 10 minutes to load the first time, but closing and opening it multiple times afterwards it loaded in seconds.
I recently aquired an ssd to see if my load times would improve, it still takes the same time to load off the ssd as my hdd, which confused me a little bit.
Right now in the process of trying to thin out my addons folder to get it loading quicker, but theres no delete option in the program so its basically loading it up, making lots of notes, then quitting and finding the files manually :/

 

[glodomorrapl]

Definatly windows cache of some sorts - i used to run VAM on low memory and needed to close and reopen the program several times in a session. Noticed it took nearly 10 minutes to load the first time, but closing and opening it multiple times afterwards it loaded in seconds.
I recently aquired an ssd to see if my load times would improve, it still takes the same time to load off the ssd as my hdd, which confused me a little bit.
Right now in the process of trying to thin out my addons folder to get it loading quicker, but theres no delete option in the program so its basically loading it up, making lots of notes, then quitting and finding the files manually :/

10 minutes? may i ask how large your addon folder is?
i would reboot, open task manager (go to the performance tab, stay on cpu, right click on the graph and go to "Change graph to" > "Logical processors"), and then open the game. observe cpu usage of each core, if any of the cores are nearing 100%, maybe you're cpu limited now.
i'd also keep your eye on disk graphs. if you just plopped an ssd into your pc and didn't move windows onto it, i'd wager windows is still using your hdd for virtual memory (or prioritizes it), which could explain the lackluster improvements in loading times. hell, even if you moved/reinstalled windows onto your ssd, i'd still check for this.
as for cleaning up VaM, i know that with BrowserAssist you can remove VARs directly from in-game if they contain scenes, plugins or any kind of preset. i wonder if there's a plugin that could help with all packages in general?

 

[RiftWind]

Addon packages folder is on hdd, 165gb in size, containing 4259 items. Cache file is now on the sdd. I tried copying VAM to the sdd to see if improvements would happen but alot of the plugins wont work correctly (i was going install it to the sdd but wanted a quick copy n paste to see because lazyness).
I pretty much only run VAM in vr mode - using fpsvr to check memory usage / temp spikes etc. The system is responsive while its loading, but displays "VAM not responding" during the entire loading process. Running an I7 7700 though - i read alot of single core threads re VR gaming and programs like Vam, but so far the only issue i have is thermal throttling due to crappy cooling. I dont think that would change loading times though?

My query is that if vam takes along time to load due to scanning the addons folder, why not have it so it loads WITHOUT scanning the folder, and when you load assets in, they get loaded on demand or on browse? Surely the only reason for scanning the whole folder would be to have it all in memory at once for access, and as no one has that much memory it makes no sense that it loads all the items one by one, then allows you access to the main menu to load your selected scenes?

What is it that it does when its loading the first time?

 

[glodomorrapl]

Addon packages folder is on hdd, 165gb in size, containing 4259 items. Cache file is now on the sdd. I tried copying VAM to the sdd to see if improvements would happen but alot of the plugins wont work correctly (i was going install it to the sdd but wanted a quick copy n paste to see because lazyness).
I pretty much only run VAM in vr mode - using fpsvr to check memory usage / temp spikes etc. The system is responsive while its loading, but displays "VAM not responding" during the entire loading process. Running an I7 7700 though - i read alot of single core threads re VR gaming and programs like Vam, but so far the only issue i have is thermal throttling due to crappy cooling. I dont think that would change loading times though?

My query is that if vam takes along time to load due to scanning the addons folder, why not have it so it loads WITHOUT scanning the folder, and when you load assets in, they get loaded on demand or on browse? Surely the only reason for scanning the whole folder would be to have it all in memory at once for access, and as no one has that much memory it makes no sense that it loads all the items one by one, then allows you access to the main menu to load your selected scenes?

What is it that it does when its loading the first time?

the "not responding" thing is normal, it does that for me too even if my addons folder is like the third of your size.

thermal throttling leads to lower clocks, and that directly leads to lower performance. but whether it would impact the startup time would probably depend on whether or not your cpu is actually the limiting factor. in my case, starting up VaM pegs one of my cpu cores to 80-90% and reads my SSD at about 80 MB/s tops (far below what it's capable of), so i would say it's very possible cpu matters a good bit. also your cooling really must be crappy, your cpu is supposed to draw maybe like 60 W under load, that's really not that much. even one of the most popular yet very reasonably priced coolers like thermalright PA120 can cool cpus that are 3x as hungry as yours.

i don't think it's loading addons completely into memory lol, probably just kind of indexes them so scene/preset/plugin browsers immediately know what to display without forcing the user to wait. but whether that process could be done more efficiently is a whole other story. maybe it would help a lot if the addon processing at startup was multithreaded? but that's the deal with many porn games - they just kind of focus on pushing out content first, optimization comes later. especially when they're funded via patreon, by people who are well-off enough to subscribe, so maybe they're also well-off enough to have a fairly powerful PC... i digress

 

[kretos]

(...) i wonder if there's a plugin that could help with all packages in general?

Of course there is. Var browser. It detach all vars from VAM (so VAM is empty), and use it's own mechanism to inject var and its depedencies on launch of the scene. Little overhead in play because if you want to change sth in the scene you have to first load it manually (ie. clothes, hair, etc).
I have over 17000 vars right now (about 1TB of data) and run VAM without no problem.

 

[glodomorrapl]

Of course there is. Var browser. It detach all vars from VAM (so VAM is empty), and use it's own mechanism to inject var and its depedencies on launch of the scene. Little overhead in play because if you want to change sth in the scene you have to first load it manually (ie. clothes, hair, etc).
I have over 17000 vars right now (about 1TB of data) and run VAM without no problem.

jesus, 1 TB?? when my VAM instance hit 200 gigs, it was enough to prompt me to reinstall VAM and pick out packages that i'm actually going to use while carefully ignoring dependencies that i didn't actually need. i can't imagine my VAM instance approaching anywhere near a thousand gigs lmao
just checked out the plugin, it's an interesting idea. having to do another step when modifying scenes sounds like a dealbreaker to me (especially since i almost always load up custom looks), but if the alternative was to wait like 5 minutes for the game to boot up, i suppose it's a compromise i'd take...

 

[Sally Whitemane]

Except for that anomaly - exactly the results I'd expect.

Any idea what the hardware DRAM cache capacity is on your SDD and HDD?
Initially I thought if that cache is bigger on the HDD that could explain the anomaly and why it could 'burst read' better in some cases.
Then I looked up typical models in shops and it seems a 2TB HDD usually has 256 MB DRAM cache and a NVMe SDD much more. Exactly the opposite.

 

[glodomorrapl]

Except for that anomaly - exactly the results I'd expect.

Any idea what the hardware DRAM cache capacity is on your SDD and HDD?
Initially I thought if that cache is bigger on the HDD that could explain the anomaly and why it could 'burst read' better in some cases.
Then I looked up typical models in shops and it seems a 2TB HDD usually has 256 MB DRAM cache and a NVMe SDD much more. Exactly the opposite.

i guess my HDD is on the lower end, apparently it only has 64 MB of cache. my SSD seems to have a full 1 GB of it.

 

[kretos]

2TB HDD usually has 256 MB DRAM cache and a NVMe SDD much more

Because usually SSD real speed is much below specs, and few gigs of fast cache helps to hide it

 

[mikei9]

Has anyone here tried a gen5 nvme? Was curious if there were any noticable differences in loading times.

 

[DJ]

Has anyone here tried a gen5 nvme? Was curious if there were any noticable differences in loading times.

For any given texture, if you calculate how long it should take to load from disk, and compare that to the actual Vam load time, you'll find Vam takes waaaay longer to load the texture than what's needed to move the actual data. I'm not sure what it's doing, but it seems clear that disk transfer speed is NOT the bottleneck limiting texture load times. So I wouldn't expect much, if any, difference.

For Vam startup time, though, maybe. You'd have to run some tests.

 

[mikei9]

For any given texture, if you calculate how long it should take to load from disk, and compare that to the actual Vam load time, you'll find Vam takes waaaay longer to load the texture than what's needed to move the actual data. I'm not sure what it's doing, but it seems clear that disk transfer speed is NOT the bottleneck limiting texture load times. So I wouldn't expect much, if any, difference.

For Vam startup time, though, maybe. You'd have to run some tests.

Thank you, appreciate the clarification!