Plugins Benchmark

Why? What is this?

As a side project, I was looking at PC components for upgrading my old machine. That made me browse through various forum threads on the Hub discussing FPS in VaM. Everyone is posting random FPS values, but barely anybody shares which exact settings and scene was used. But those differences can make huge difference, if they are not identical comparing performance is really pointless. Being half a meter further back or using a different FoV may give you 10 more FPS. Especially resolution makes a big difference. It would be really helpful when deciding whether you want to spend some extra 100€ on that faster RAM module, different CPU, etc. Does it make sense to sink money in an RTX3080....or is an 3070 enough?

So, to help solve this problem, I'm presenting you the MacGruber Benchmark! Its a series of scripted scenes that measure performance of different aspects in VaM. In the end you are a presented with a nice result screen which shows you an overall average FPS value for quick comparison, detailed results and time measurements for each individual scene, the VaM settings that were used and of course some basic system stats like CPU, graphics card, VR headset used, etc. It does not know about things like overclocking or undervolting, though.

The plugin does also ensure that unknowing users can't accidentally produce invalid results. There are various integrity checks, like checking all dependencies are present, all plugins were allowed to run, the scenes weren't modified, etc. In VR the plugin does also lock your head movement to make sure everyone sees exactly the same. Note that this may cause more motion sickness than you are used to, see section below how to reduce it.




"Pay if you liked it!"
Producing a complex tool like this one requires time, currently about 130-150 hours of work went into this project over a period of 3.5 months. This benchmark is FREE to use for everyone, because the more data we have, the more useful it will be for the community. HOWEVER, the clear expectation is that if this is reasonably useful for you, you leave a matching pledge on my Patreon site. This obviously goes especially for commercial creators. Let's see whether this concept works or is abused.


How do I run it?

• Unlike other scenes, this scene requires to be run from its VAR package. Don't unpack, don't modify it in anyway or it will refuse to work. This is intentional to make sure that you don't accidentally disable important parts of the scenes, e.g. by not downloading the dependencies, not allowing its plugins to run, removing characters, etc.
• The easiest way is to navigate to this resource page using the "Hub Browse" window within VaM and press "Download All". Then just open the scene normally. Look for this icon in VaM's main menu bar:

Where do I share the results and discuss?

• The result screen has a button that creates a screenshot of your results and puts it in your Saves\PluginData\Benchmark folder for easy sharing.
• Also I created a discussion thread here: https://hub.virtamate.com/threads/benchmark-result-discussion.13131/

Can I compare results between different benchmark versions?
Each new version of this Benchmark could make changes that may affect performance. So, be careful when comparing results. As of March 2023 (up to version 4) the only significant change was introduced with Version 3 since new scenes where added and the older scenes changed slightly. Therefore v1 results can be compared with v2. Also v3 with v4. However, be careful when comparing v3 or v4 with earlier v1/v2 results!

What do the different scenes do?
There are 8 different scenes, total duration is a bit more than 11 minutes, depending on loading time between scenes:

• Warm-up: The first scene is simply indented to warm up your machine, since temperature may affect performance. It runs twice as testing showed that 1 minute was not enough to produce reliable results.
• Baseline 1: This is a mocap scene with simple clothing and hair. Its intended so you can compare against the following two scenes that are identical except for one thing each.
• ClothSim: Identical to Baseline 1, except that it is using nice clothing with simulation enabled.
• Mirror: Identical to Baseline 1, except that there is a mirror placed in the background.
• Baseline 2: Another mocap scene with simple clothing and hair. This one being closer to the character than Baseline 1. Its intended so you can compare against the following scene that is identical except for one thing.
• HairSim: Identical to Baseline 2, except that it is using nice long hair with simulation enabled.
• Baseline 3: This is a hybrid mocap and manual animation scene with 3 characters. There is a lot of idle motion and relative animation going on.
• Simpler Physics: Identical to Baseline 3, except that "Advanced Colliders" and "Soft Body Physics" are turned off for each of the 3 characters. This should bring some noticeable improvement on low-end systems, but do almost nothing for higher-end CPUs.

Performance FAQ

My system seems to be performing slower than expected compared to the results posted by others that have similar systems. What should I check?

• Try running some generic gaming benchmark that has an online result database, for example 3DMark with its TimeSpy benchmark. 3DMark provides you with a nice graphic showing you the score distribution curve of systems that have the same CPU and GPU model. So you can easily tell whether you are on that curve where you expect to be. In my case I expected to be slightly ahead of the bulk because of CL14 memory, mildly factory-overclocked GPU, CPU undervolt and good air cooling. If your system performs way outside of this range, there may be something wrong, of course.

• Maybe your XMP memory profile is not enabled in BIOS? For example mainboards for the popular AMD CPU's operate by factory-default only at 2133 MHz memory frequency. If you have a 3200 or 3600 MHz RAM module, you probably want to activate that. Note that the default is there for a reason, of course, as speeds over 3200 MHz are usually not officially supported by CPUs.
• Temperature overheating may be a factor, check the following section.
• You might have defined various settings overrides at GPU driver level. For example Dynamic Super Resolution (DSR) or similar things. You may want to try resetting settings in NVIDIA Control Panel (or the AMD equivalent) to default.

How does temperature affect my system?

• Both CPUs and GPUs may automatically reduce their clock-rate to when reaching a certain maximum temperature. Therefore running the benchmark with an already warmed up system may result in slightly lower performance results. However, the benchmark scene intentionally runs a warmup scene two times to make sure your system is properly warmed up. This scene does not contribute to the final benchmark results, which reduces the impact of temperature throttling.
• You may want to monitor your temperatures (and fan speeds) during benchmarks using some software like HWMonitor. Note that HWMonitor requires admin rights to run properly, otherwise you only get very few sensor readings.
• Note that Ryzen 5th generation CPUs get regularly very hot. Even with good cooling, on stock settings these routinely run into 90°C under continuous load and have to throttle down. According to AMD this is normal. Undervolting your CPU can be a solution, about 10°C lower temps seem to be common for Ryzen 5th generation, but keep in mind that, just like overclocking, it voids your AMD warranty. Especially make sure to test your system properly....and then test again. Not every calculation error of your CPU leads to an obvious crash, so it can be hard to notice. You do not want to work with a CPU that thinks 1+1 is 3 in 1/10000 of cases. Some software like Prime95 that actually verifies the calculation results are actually correct may help.

Might other applications affect my results?

• Yes. As with any benchmark, you should close applications you can easily close. Especially browsers often like to use way more resources for "adverts in the background" than you might think.

I'm not getting exactly the same result each time I run it?

• Even with identical settings and accounting for temperature, you will get slightly different results each time. There are just too many factors the benchmark can't account for. For example things your operating system does in the background, VR tracking running, other applications, etc. However, with my testing I got results within 0.5-1.0 fps each time I was running.

What does VSync do?

• Vertical synchronization makes your graphics card wait for the next refresh of your display. If you have a 60Hz display or 90Hz headset, the game will try to reach exactly this framerate. This produces a much smoother image as frames won't be skipped or shown twice. With some displays it can also happen that the display switches to the next frame in the middle of the refresh cycle. This can cause very visible artifacts called tearing.
• Since VSync does sync the framerate to exactly your display refresh rate, a benchmark becomes pointless...you measure display refresh rate, not game performance. Therefore, unless you want to see if your system can keep the stable framerate needed, VSync should be turned off for any benchmark.
• Note that Oculus apparently does not allow VSync to be properly turned off for some reason. It appears when getting near particular framerates like 45 and 90, VSync turns on, while you can have framerates in-between 60 and 89.

What is Asynchronous Spacewarp or Motion Smoothing?

• By default most (or all) VR headsets try to reduce motion sickness by introducing additional interpolated images when your machine can't keep up with your headsets native refresh rate. Oculus calls this Asynchronous Spacewarp (ASW) while SteamVR uses the term Motion Smoothing (MS). While both implementations are a bit different, basically the goal for a 90Hz headset is to stablize the game framerate at 45Hz (or 30Hz), while your eyes still get the 90Hz with additional interpolated frames inbetween. This is usually a good thing, so normally, if your system can't keep stable 90fps all the time, you would want it enabled. However, for a Benchmark it makes it impossible to get useful measurements, apparent differences between machines would be very small and mostly just random noise as you can only end up with 30, 45 or 90fps. Find instructions how to turn this off below.

How to turn off Oculus Asynchronous Spacewarp?

1. Start the Oculus software.
2. Download OculusASW-Helper.1.zip and extract it somewhere.
3. Run OculusDisableASW.bat from above ZIP. Note that this assumes you installed Oculus software to its default location C:\Program Files\Oculus. Possibly you need to change the path in the script.
4. Launch VaM.
5. Run some scene that has at least a single character and check the framerate in the User Preferences menu. Make sure ASW is turned off and you are not locked to particular framerates like 45 or 90fps.
6. Run the benchmark
7. Note that ASW turns back on automatically when you restart the Oculus software. You will have to disable it again.

How to turn off SteamVR Motion Smoothing?

1. Go to SteamVR settings, section "General"
2. Make sure "Advanced Settings" are shown
3. Set "Motion Smoothing" to "Off"
4. Close settings
5. Launch VaM.
6. Run some scene that has at least a single character and check the framerate in the User Preferences menu. Make sure ASW is turned off and you are not locked to particular framerates like 45 or 90fps.
7. Run the benchmark

What does min1% and max1% mean in this benchmarks result screen?

• It's the 1% quantile. That means 1% of the frames were better/worse than this value.
• Measuring the actual min/max frame during the entire run would only provide very random results that are just useless noise.

Reducing Motion Sickness in VR

• One of the things on top of the list a VR game should NOT do is to lock head rotation in VR. However, for a Benchmark its important everyone renders exactly the same or as close as possible, otherwise results are not comparable. Hence camera movement has to be locked.
• When running via SteamVR I had some strong wobble from the head tracking and benchmark trying to counteract each other. At least for my hardware, running with Oculus is smoother.
• I recommend to run this benchmark in seated position and trying to move your head as little as possible to reduce motion sickness. Note that motion sickness may also hit you later with some delay when taking of your headset and getting up. Just to be save, you probably should not run this benchmark when you need to drive a car (or similar) in the next few hours.
• Of course you can prevent/reduce motion sickness by keeping your eyes closed during most of the benchmark, only peeking a quick look now and then. Each of the scenes plays some music and there is a ding-ding elevator sound at the end of the benchmark telling you when its save to open your eyes. There are 9 scenes, total duration is a bit more than 11 minutes, depending on loading time between scenes.

Credits

• MacGruber: Scene setup, special purpose plugins as well as Life, LogicBricks and Essentials plugin collections
• @Acid Bubbles: Timeline and Glance plugins
• @ReignMocap: Friendly permission to extract mocap from his scenes for this purpose
• @VL_13: Friendly permission to use "Top_SK" and "Leggings_S" clothing items for this purpose
• @prestigitis: Friendly permission to use "Sleeveless Trapeze Dress" clothing items for this purpose
• @NoStage3: "Short Pixie" and "Long Side 3" hair
• @kemenate: "Male Hairstyles Pack 01" hair
• @VamTastic: Penis_Grip morph
• @RenVR: "Janie" skin (VaM build-in)
• @Jackaroo: "Lee" skin (VaM build-in)
• MeshedVR / VaM build-in assets
• Alex-Productions: Music
• David Law: Skyline of Hong Kong