Benchmark

Plugins Benchmark

Download [27.04 MB]
Support me at...
Overview Updates (4) Dependencies (13) Reviews (19) History Discussion
Lione said:
Does my renderscale settings affect the benchmark or is the benchmark fully native ?
No matter what i use the Benchmark will use enforced right
Click to expand...
Resolution scale affects resolution, so it has a huge impact. For a scale of 1.5x your poor graphics card has to crunch 1.5x1.5 = 2.25x as many pixels, so it needs to be more than 2x faster to keep up. For the official benchmark the resolution scale slider in VaM is forced to 1.0x. However, you can set resolution scale also externally like in SteamVR settings, and my VaM plugin can't override a system wide setting of course. Therefore on the result screen the effective resolution scale and actual internally used resolution are displayed.
The benchmark is not modifying how VaM renders, its just changing VaM settings. Because we care about how VaM renders, not some obscure benchmark.
 
Thx again for the detailed background.

So in my screenshot i do not have to care that my renderscale is in red displayed 1.2xxxx
and i do not have to create another test with 1.0 default.
I was actually thinking that my 1.2 may cause a different benchmark result compared to others who use 1.0 ;)

Confusiness solved.
 
Hi, great plugin - I have been waiting for a decent VAM benchmark to be developed for a long time :D

I did the original benchmarks on the old reddit forum here a few years ago, but it was basically just asking people to run the old built in CPU/GPU benchmarks and no way to control what people did after that! Hopefully this modern benchmark with modern scenes and features will be a lot more useful.

One thing though, I was expecting the benchmark to change the VAM window to some fixed size so its comparable for everyone but it didn't do that - just stayed at the resolution I choose when I started up with...

START "VaM" VaM.exe -show-screen-selector -vrmode None

...option. I picked 1920 x 1200 in that.

When I got results the resolution was in red which obviously means its not official.

Some questions:
What is the official resolution that the desktop version is supposed to run at?
Do you not force a resolution when benchmark is being run?
How do I enter my name in results (I only get Anonymous)?
 
Last edited:
MacGruber updated Benchmark with a new update entry:

Updated Performance FAQ

Changelog:
  • New section: "My system seems to be performing slower than expected ..."
  • Added more info to "How does temperature affect my system?"
  • New section "How to turn off SteamVR Motion Smoothing?", thanks to @VaMDeV for providing info and screenshot.
  • New section "What does min1% and max1% mean in this benchmarks result screen?"

Check it out: https://hub.virtamate.com/resources/benchmark.11336/
Click to expand...

Read the rest of this update entry...
 
Why do you have to download and run some batch file to disable Asynchronous Spacewarp?

Can't we use the Oculus Debug tool like we always did?

C:\Program Files\Oculus\Support\oculus-diagnostics\OculusDebugTool.exe

This is also permanent as far as I know

1634991392607.png
 
I have an optimization to come up with: could you integrate a check like with the "official settings" so that everyone has to have a "normal" desktop resolution scale?
(1920x1080 / 2560x1440 / 3840x2160)

It would be even better if everyone has to have one of those three resolutions, the people with 16:10 or whatever as well. I mean everyone is testing with a new windowed resolution which makes it, at least in my eyes, a little obsolete. I mean we want a 1:1 comparison which is "hard enough" with plenty of VR resolutions, but desktop is totally different. 16:9 is the standard those days and those three resolutions are used, I would guess, from more than 90-95% of the people.

//EDIT
I have an even better idea for all resolutions, may it be desktop or VR: could you integrate a formula at the end that calculates FPS per dot resulting in a benchmark score? That would be it for everyone and every device!

1920x1080 = base resolution = 1 (2073600) every pixel count is divided by 2073600 for a standard

Desktop:
SYSTEM 1: 1920x1080 = 2073600 dots ( : 2073600 = 1) // avg FPS = 100 // 100 * 1 = 100
SYSTEM 2: 2560x1440 = 3686400 dots ( : 2073600 = 1,77) // avg FPS = 70 // 70 * 1,77 = 123,9
Meaning: SYSTEM 2 is ~24% faster than SYSTEM 1! Booyakasha!!! :D

VR:
SYSTEM 3: 2688x1600 (Oculus VR) = 4300800 dots ( : 2073600 = 2,07) // avg FPS = 60 // 60 * 2,07 = 124,2
SYSTEM 4: 4936x2740 (Valve Index) = 13524640 dots ( : 2073600 = 6,52) // avg FPS = 40 // 40 * 6,52 = 260
Meaning: SYSTEM 4 is around 2x faster than SYSTEM 3. I made it! 🤓
Meaning 2: SYSTEM 2 and SYSTEM 3 are nearly the same in speed.

The numbers at the end are the benchmark score related to 100 FPS in 1920x1080. I hope this makes sense.
Of course this only compares results. No matter if someone is in the CPU or GPU limit.
 
Last edited:
I launched the benchmark again due to some system changes and tweaking and found out, after launching it with lower resolution at 1920x1080 and switching it back now to 2560x1440, the benchmark still runs on 1920x1080. Did I make some mistakes or why does the benchmark not recognize my desktop resolution correctly?

btw: No reaction to my last post is a little sad. I put a lot of effort in it :cry:
 
MacGruber updated Benchmark with a new update entry:

Version 3

Changelog
  • New scene with in two variants, featuring 3 characters having fun. Something like this was asked for a lot as it should make a more realistic test, as many VaM scenes have more than one character. As usual the variant is identical except for one thing, in this case "Advanced Colliders" and "Soft Body Physics" are turned off for all 3 characters. On my old machine (i7-4770K) that would have given a noticeable performance boost...
Click to expand...

Read the rest of this update entry...
 
I've been playing around with "resizable bar" to see if it has any impact on load times in Vam (very preliminary tests say no, but I have more to do), and it made me realize that load times for scenes, clothing, textures, etc are an aspect of performance that Benchmark doesn't consider. That might be a feature worth adding in any future version of the plugin.
 
DJ said:
I've been playing around with "resizable bar" to see if it has any impact on load times in Vam (very preliminary tests say no, but I have more to do), and it made me realize that load times for scenes, clothing, textures, etc are an aspect of performance that Benchmark doesn't consider. That might be a feature worth adding in any future version of the plugin.
Click to expand...
The idea of the benchmark was a standardized scene, so you can compare results. So it tries to explicitly wait for the scene to be fully loaded, before starting the animation and time measurement. However, load times depend heavily on things like how many VARs you got installed, how many textures you already got cached, etc. You can't really "standardize" that, it would require that a only particular set of VARs is being installed.
 
Anyone else having trouble running the benchmark in the latest update 1.22? I just get a dark empty room for each test. Tried on a clean install and my regular install. And I'm still having issues with it recognizing an official test no matter where I load the benchmark from.
 
MacGruber updated Benchmark with a new update entry:

Version 4

Changelog
  • Hashes updated for VaM 1.22.0.1 to account for changes in how scenes are saved.
  • Additional integrity checks:
    • New VaM version check. Ensuring you are not using VaM 1.22.0.0, which broke the Timeline plugin. Make sure to update to VaM 1.22.0.1 or newer.
    • VaM's new DepthTexture setting (which requires a VaM restart to properly reset)
Notes
  • Results from v3 with VaM 1.20 and...
Click to expand...

Read the rest of this update entry...
 
have this error when benchmarking
Scene loaded as it says in the manual
I know my PC is more than powerful enough, but I wanted to test it anyway
NVIDIA GeForce RTX 3060 12Gb
AMD Ryzen Premium Vr Edition7 3700X 8 core processor 3.60 GHz
32.0 GB Ram

Failed to compile MacGruber.Benchmark.4:/Custom/Scripts/MacGruber/Benchmark/Benchmark.cslist. Error:
!> [CS1061]: Type SuperController' does not contain a definition for generateDepthTexture' and no extension method generateDepthTexture' of type SuperController' could be found. Missing an assembly reference? in <Unknown> at [614, 35].
 
Nostradamus said:
Failed to compile MacGruber.Benchmark.4:/Custom/Scripts/MacGruber/Benchmark/Benchmark.cslist. Error:
!> [CS1061]: Type SuperController' does not contain a definition for generateDepthTexture' and no extension method generateDepthTexture' of type SuperController' could be found. Missing an assembly reference? in <Unknown> at [614, 35].
Click to expand...
You seem to be running VaM 1.21 or older. Use the old v3 of the Benchmark or update to VaM 1.22.0.1.
 
Hey MacGruber, thank you for the current update! Unfortunately, the new version doesn't do anything on my end, without any messages or errors.

- VaM version: 1.22.0.1
- Plugins are "Always Enabled" for the Benchmark scene
- The scene loads fine, no errors, no messages
- I click on "Apply Official UserPreferences"
- When using VaM in Desktop mode, clicking on "Run Official Benchmark", the text disappears, but nothing else happens (even after minutes)
- When using VaM in VR mode, after clicking "Run Official Benchmark", an additional text appears about Motion Smoothing, and after confirming MS is off, again, nothing happens

I'm pretty sure I'm missing something very obvious here, I'm just not sure, what. Any ideas? The previous version of your Benchmark worked fine on VaM 1.20 (haven't run any benchmarks on VaM 1.21).

Thanks a lot in advance.
 
@theMickey That seems pretty weird. Things to check:
  1. VaM version is indeed "1.22.0.1"
  2. Make sure you install the Benckmark package via the Hub-Browse window within VaM, also installing all it's dependencies. Hitting "Download All" should just work. No other sources. No modifications, no unpacking, etc.
  3. Make sure Benchmark version is indeed "4"
If that doesn't help, try this:
  1. Check that you allowed ALL the plugins, not just the Benchmark itself. The scene runs plugins from its own package, but also Life, LogicBricks, Essentials, Timeline, SpawnPoint and Glance. (However, there should be measures in place that it would tell you if this were an issue...)
  2. Remove all session plugins (also clearing your session plugin defaults, if you have them). Restart VaM.
  3. Use VaM's default-default scene. Maybe you have some custom default scene that loads plugins that don't properly clean up after themselves. So, reset that and restart VaM.
 
@MacGruber Thank you for your fast response, and your checklist.

> 1. VaM version is indeeed "1.22.0.1"
It is.
1677761773734.png

> 2. Make sure you install the Benchmark package via the Hub-Browser...
> 3. Make sure Benchmark version is indeed "4"
I did, and it is version 4. None of the packages has been modified or unpacked.
1677761852010.png 1677762037853.png

> 1. Check that you allowed ALL the plugins ...
According to the above screenshot, that should be the case, right? Life, LogicBricks, Essentials, Timeline and Glance are 100% enabled, as I can run other scenes without any problem which contain those.

> 2. Remove all session plugins ...
I don't have any.
1677762225228.png

> 3. Use VaM's default-default scene...
I indeed did use a "custom" default scene, so I changed it back to the original one, and then restarted VaM.

Still: your benchmark does nothing after clicking "Run Official Benchmark" -- the text on the "TV" disappears, but nothing else happens. Again: no messages, no errors. (Oh, and btw: "Freeze Motion/Sound" is "off" of course.)

// EDIT: I just did a "fresh" install of VaM by copying the Updater to an empty directory and downloading VaM. I then downloaded your Benchmark through the Hub -- and it works. So something in my other instance seems to be off, but that's the same instance for which your benchmark v3 worked on 1.20, and after updating VaM and your benchmark it stopped working. I'm curious to find out what's causing this...

// EDIT 2: I found the culprit!
Ok, this is one is definitely on me. After spending hours on troubleshooting (the usual: deleting the VaM Cache, deleting and reinstalling the Benchmark from VaM Hub, copying all my AddonPackages incl. the AddonPackagesFilePrefs and AddonPackagesUserPrefs from the "old" to the "fresh" instance, also copying my Custom folder and finally my Saves folder from the "old" to the "fresh" instance -- after every single step I had no success on running the benchmark in my "old" instance, while it was working fine in the "fresh" instance), I found the root cause of the Benchmark not working on my "old" instance:

I used a program to compare all the files and its contents between my "old" and the "fresh" instance -- and there was no difference!
1677790370068.png 1677790387077.png

...but then it dawned on me: I usually "hide" files that I don't have to interact with, like all the ...Benchmark.bat files from MeshedVR, which I never use. And because I also never edit my VaM preferences by manually editing prefs.json, I hid that file as well! And this was my mistake! If prefs.json is hidden and has something different from smoothpass = 4 in it, the Benchmark just doesn't work.

Long story short: Do not hide prefs.json from your OS, so the Benchmark can run with smoothpass = 4 as intended.
 
Last edited:
Hi im stuck on loading charater during benchmark.
- VaM version: 1.22.0.1
- No plugin activated during benchmark.
I waited for 5 minutes but i can't proceed on janie load.

1681733505672.png
 
flyleaf said:
Hi im stuck on loading charater during benchmark.
- VaM version: 1.22.0.1
- No plugin activated during benchmark.
I waited for 5 minutes but i can't proceed on janie load.
Click to expand...
Things to check:
  1. Did you get ANY error messages when starting VaM or when loading the scene?
  2. Make sure you install the Benckmark package via the Hub-Browse window within VaM, also installing all it's dependencies. Hitting "Download All" should just work. No other sources. No modifications, no unpacking, etc.
  3. Make sure Benchmark version is indeed "4"
  4. Check that you allowed ALL the plugins, not just the Benchmark itself. The scene runs plugins from its own package, but also Life, LogicBricks, Essentials, Timeline, SpawnPoint and Glance. (However, there should be measures in place that it would tell you if this were an issue...)
  5. Remove all session plugins (also clearing your session plugin defaults, if you have them). Restart VaM.
  6. Use VaM's default-default scene. Maybe you have some custom default scene that loads plugins that don't properly clean up after themselves. So, reset that and restart VaM.
 
Same Error like theMickey, start Scene click on Start Benchmark but nothing happens Text is display off an nothing start, SmoothPasses check is on 4 any idea to get it start?
 
DumpyElephant said:
Same Error like theMickey, start Scene click on Start Benchmark but nothing happens Text is display off an nothing start, SmoothPasses check is on 4 any idea to get it start?
Click to expand...
Have you read the last "spoiler" in my post to see why it didn't work for me and how I fixed it? Maybe you did something similar.
 
Same issue, followed all instructions, including 'spoiler' and still doesn't work. Stays stalled on Character load Janie.

Is on a brand new VAM installation. My previous VAM install worked fine. There's some odd issue here and seems like it's not just affecting me
 
@VamPire @DumpyElephant @Jimmichoochoo
I'm still waiting for someone to confirm that all the "Things to check" I listed above don't help:

Plugins - Benchmark

Does my renderscale settings affect the benchmark or is the benchmark fully native ? No matter what i use the Benchmark will use enforced right Resolution scale affects resolution, so it has a huge impact. For a scale of 1.5x your poor graphics card has to crunch 1.5x1.5 = 2.25x as many pixels...
hub.virtamate.com hub.virtamate.com

Also check for the thing @theMickey said.
 
You must log in or register to reply here.

Similar threads

VaM_LO
Scenes [Scene] Benchmark
Replies
1
Views
671
meshedvr
meshedvr
Back
Top Bottom