50-100$ commission for "Benchmark Scene"

MacGruber

Invaluable member
Developer
Wiki Contributor
Featured Contributor
Messages
1,607
Reactions
3,169
Points
143
As a side project, I was looking at PC components for upgrading my old machine. That made me browse through various forum threads on the Hub discussing FPS in VaM. Everyone is posting random FPS values, but barely anybody shares which exact settings and scene was used. But those differences can make huge difference, if they are not identical comparing performance is really pointless. But it would be really helpful when deciding whether you want to spend some extra 100€ on that faster RAM module, different CPU, etc.

So therefore, as intermediate project before continuing with more tutorials, I want to build a proper "official benchmark" scene. This scene would use a number of yet to write plugins to enforce that you use particular graphics & physics settings, ensure that your machine is properly warmed up before starting to make time measurements, as a warm system might throttle down power usage and be slower. Of course it should write the results, used resolution and so on to a nice text file so you can easily post the data in forums and other places. Essentially it should function in a similar way than build-in benchmarks of popular games, except its a scene you launch.

I don't have the capacity at the moment to build another high-end quality scene, while at the same time writing those needed plugins, so I'm looking at using an existing scene made somebody else. This could be paying a creator to "free" an existing paid scene or even be a commission. Depending on delivered quality and what needs to be done, something around 50-100$ via Patreon would be reasonable. That's not much, but I mean its mostly free advertising your content and lots of people will download this. Possibly we can convince @meshedvr to donate to the cause as well, but of course that's not for me to decide.

The requirements as I see them are the following:
  • Generally well-build quality scene, representative of VaM.
  • Two fully animated characters that touch each other. That means several controls being animated as well as several morphs, at least facial expressions and breathing. Intention is to get the physics engine to do some work.
  • About 1-2 minute of animation with some camera movement including medium and close-up distances. Ideally in a way that it is watchable in VR without getting motion-sick, so slow and continuous movement. The scene will run multiple times during the benchmark, possibly going through different quality settings, etc.
  • I guess sound, clothing, hair and nice environment / prop assets are expected as well.
  • Content should be reasonably "mainstream". We want as many people as possible to run this. Maybe it does not even need to contain any nudity.
  • Ideally the scene should not contain anything random, so exactly the opposite of what I'm usually doing in my scenes ;) The reason is that the scene of course needs to behave identical every time you run it...so you get the same reproducible performance results.
  • The scene should only need a small number of dependencies. ALL of them need to be available for free as hub-hosted VAR package. The scene itself, after adding my benchmark plugins, will be uploaded to the Hub for free as well. We want as little hurdle to run this on your machine as possible.
  • No warnings or error messages.
  • No license violations, obviously.
If you have suggestions which scene would fit this description, want to suggest your own scene or negotiate commission conditions, please contact me. Discord or Hub preferred. Please do not waste my time by not having read/understood above requirements. If you think you can cobble something together in 2 hours, don't bother either, you can't.

Important: Do not start doing major work before contacting me first! Having 5 different people working on things without any coordination does not help anyone.

Cheers,
MacGruber
 
Glad to see that Mac Gruber has accepted that challenge!
I would like to add some more aspects:

- Those scene should ideally stress every technically aspect of VaM in a good mix. Not only how good your PC is at rendering, but also not only how good it is with physics, aso.
- It should be well balanced between stressing the PC and, at the same time, it should ideally reflecting every-day usage. No need for measuring artificially issues that normally won't ever occurs.

@MacGruber I like this project. Unfortunately I can't be of any valuable help, except maybe morphs or graphics. But If you will need some people for testing or for giving annoying advices, don't hesitate to contact me.
 
Last edited:
Well, ideally a benchmark should cover like 10-20 different scenes testing different aspects, but that's just not realistic. First in building it and second in people actually running it. I agree that testing artificial issues is pointless. So, yeah, my intention is to build a reasonable every-day scene, hence two characters. Of course there is no harm in that scene looking cutting edge. Anyway, that scene should probably run 3 times. The first one to warm up the machine, second run with CPU focused settings and the third with settings eliminating the CPU bottleneck, e.g. turning off advanced colliders or reducing physics rate to test the GPU. Something like that, so you see how your machine in different scenarios.
 
Sounds good. Maybe it should use some of the build-in stuff to reduce download size? I think I remember there was those beach house environment, that was quite a bit performance heavy on my old PC, for instance...
Question, are you able to switch through the build-in quality settings by code? This would maybe help a bit to avoid people posting nonsens numbers. Though, it certainly could not avoid every "error", like different SteamVR SS settings, or display driver settings. Not to forget the fact that we would like to use that scene for trying out different quality settings...
If not too much hassle, optionally the output text file could include not only FPS, but also latency, frame drops, aso, values that are important for VR.
 
Cool, didn't know this is even possible.
 
Can't help with the plugin, but I can contribute to the rest.

I think it should use only built-in stuff (except for the dedicated plugin(s)). As Ttoby said, there are enough to play with (environments, sim clothes, props...). And it would avoid any issue with versioning.
Imagine using an asset that is updated later on to add more details... Would screw the result consistency.
 
Even though I follow you here and are a Patron of yours I only noticed the Discord posting in #screenshots. Guess it's the same for most others too.

One thing that isn't controlable by any plugin are the SteamVR settings. Since a few versions it's possible to configure the FOV as well as the render resolution. If you just decrease the FOV the smaller viewable area is still rendered at the full resolution, resulting in a sharper image. AND if you decrease render resolution at the same time you can lower your FOV / render resolution and get better performance (but keeping the same visual quality).

And regarding the scene I'd advise two scenes, one which stresses only the GPU and one that only stresses the CPU. The reason is that you cannot control it well enough with settings alone. Bottlenecking the CPU even with physics disabled is easily possible.

Also, the scene should not be too complex in terms of (soft body) physics because they like to glitch out and eat much more CPU power than necessary.

I may be able to create the scenes, depening on what you want the scenes to include.
 
Was wondering how is this different than benchmark scripts VAM provides but that prototype video helped demonstrate. Would love to up that pot so if there is a way let me know. Been trying to find a better way to benchmark Windows vs Linux (via proton) and this would help.
 
For anyone interested in the project status, I just posted a paywalled post here on my Patreon. However, short version here:

I'm currently still working on the technical side of things. Once that settled, I will obviously need a number of short scenes to actually run the benchmark on. For that reason I put out above commission to have someone build those scenes for me. As probably most people, I have a life and a real job, so there is only so much time that can be put into VaM without going insane. Therefore the main intention from my side here was for me to save time, not cooperate for cooperation's sake.

While I got a couple of responses, sadly there was nobody among them I trusted to have fully understood the scope and complexity of what is needed. Maybe there would have been someone among them with the needed skills and time, but without some released scenes on Hub/Patreon with similar quality/complexity to show for, there is no way for me to know. There would have been a high risk of having to iterate back and forth 10-20 times to get what is needed, or failing altogether, if that someone was in way over their head. That might have meant having to spend more time discussing and explaining than it would take for me to do the work myself. Remember, the goal was to save time. Also its frustrating, for both sides. So I will probably do it myself, which means it will take a while. @an3k and @khaal, I hope you understand and don't take this the wrong way.

However, I got permission from ReignMocap to use some of his mocap recordings for free. Also AcidBubbles allowed me to use part of a scene. In both cases there would still be lots of work required from my side to get it to a state that is usable for me. But its a start. I have not yet made a final decision on what I'm going to use, I might also end up using parts of one of my own scenes. Benefit of doing the work myself is of course that I can do it in my own time, not having to wait on anybody....and not wasting time because I wasn't able to explain what I needed in the first place :)
 
Sorry to interrupt your guys conversation, this is off topic. I have notice that some scenes run exceptionally well with amazing FPS for myself. The image clarity is amazing, others not so much. Which specific settings should I use to optimize scene?
 
While I got a couple of responses, sadly there was nobody among them I trusted to have fully understood the scope and complexity of what is needed. Maybe there would have been someone among them with the needed skills and time, but without some released scenes on Hub/Patreon with similar quality/complexity to show for, there is no way for me to know. There would have been a high risk of having to iterate back and forth 10-20 times to get what is needed, or failing altogether, if that someone was in way over their head. That might have meant having to spend more time discussing and explaining than it would take for me to do the work myself. Remember, the goal was to save time. Also its frustrating, for both sides. So I will probably do it myself, which means it will take a while. @an3k and @khaal, I hope you understand and don't take this the wrong way.

Well, the problem is that you weren't very specific in what the scenes should contain (on a technical level). You know very well that based on the type of physics used either the CPU or the GPU has work to do. That's why I asked.
I totally understand that you don't want "to iterate back and forth 10-20 times" but since you weren't able to at least answer ... well ... all I can do is wish you good luck in finding some highly creative guy who also "fully understands the scope and complexity of what is needed".
 
I think this will be a lot of work put into a project soon irrelevant (VaM 2.0). To be honest: I would wait.
But I'm definitely into it. Looks great so far. AVG fps is not working as it should, but I'm sure you know.
 
I think this will be a lot of work put into a project soon irrelevant (VaM 2.0). To be honest: I would wait.
But I'm definitely into it. Looks great so far. AVG fps is not working as it should, but I'm sure you know.
Man, this aged well, Lol.
 
Back
Top Bottom