Switched RTX 2070 for RTX 3080 Gaming Z Trio, now VAM crashes in VR with Rift S on heavy scenes :(

[pinosante]

I am at a loss. Desktop works fine, it seems. Other GPU stuff works good (like CUDA applications / heavy GPU machine learning tasks). Other VR games work fine.

Simple (small) VR scenes also work. But big VR scenes, crash VAM. And it crashes my Rift S so hard, that I have to reboot to actually be able to use the Rift S again.

I checked with GPU-Z and it's not that the GPU memory is overloading.

Any tips on how to problem solve this? I'm kinda depressed that the sole reason for spending $1000, seems to be in vain .

 

[QuestionableReality]

I think it would helpothers (incl me) to know the rest of your system.
Im running a 3080 too and rarely have any issues.
Most performance struggles for me route from having 203213 browser tabs up in the background while running VR or similiar

 

[pinosante]

I think it would helpothers (incl me) to know the rest of your system.
Im running a 3080 too and rarely have any issues.
Most performance struggles for me route from having 203213 browser tabs up in the background while running VR or similiar

Fair enough.

• Cpu: Ryzen 9 5900X.
• Mobo: msi mag b550 tomahawk.
• Ssd: samsung 980 pro 1 tb.
• Mem: Corsair Vengeance LPX CMK32GX4M2E3200C16 x2 (64 Gb).
• PSU: Corsair RMx Series RM750x.

The crashing on heavy files happens on a clean startup with no other stuff happening, when the GPU temp and everything is really low key. So turn on computer, run VAM heavy file, crash. But on simpler filesno problem. Other games, apps, no problem anyway.

Hope this helps. Maybe it’s the PSU? That’s my worry. That it crashes on peak current draw or something.

It happens both when I run in Oculus only, or through SteamVR.

 

[Sally Whitemane]

From my experience I'd say it's not the PSU. 750 Watts from a brand like Corsair Gold certified should be good enough for a 3080 - unless you have some A LOT extra stuff like 8 more spinning hard drives connected to it.
The graphic cards manufacturers like to over estimated the recommended PSU because they assume the worst case with users using cheap no name PSUs.
If the PSU fails to provide enough power is should turn off or reset. How does your crash looks like - does it just crash the VaM-process? Freeze everything?

If you want to be sure about the PSU you could create a synthetic load with running the Prime95 stress-test on the CPU and FurMark on the GPU at the same time - if the PSU withstands that torture for a few minutes it's very safe to assume it's good enough.

Updated to the latest BIOS on the mainboard?
My best guess would be a some kind of software/driver issues or unstable RAM (that can happen unnoticed over years) - maybe run a memtest from a USB drive during a night (takes very long to pass 64 GB) to check for errors - at least that could be crossed off the list then.

 

[pinosante]

From my experience I'd say it's not the PSU. 750 Watts from a brand like Corsair Gold certified should be good enough for a 3080 - unless you have some A LOT extra stuff like 8 more spinning hard drives connected to it.
The graphic cards manufacturers like to over estimated the recommended PSU because they assume the worst case with users using cheap no name PSUs.
If the PSU fails to provide enough power is should turn off or reset. How does your crash looks like - does it just crash the VaM-process? Freeze everything?

If you want to be sure about the PSU you could create a synthetic load with running the Prime95 stress-test on the CPU and FurMark on the GPU at the same time - if the PSU withstands that torture for a few minutes it's very safe to assume it's good enough.

Updated to the latest BIOS on the mainboard?
My best guess would be a some kind of software/driver issues or unstable RAM (that can happen unnoticed over years) - maybe run a memtest from a USB drive during a night (takes very long to pass 64 GB) to check for errors - at least that could be crossed off the list then.

Thanks for giving me these pointers!

At this point I'm pretty confident to say, that only VAM gives me these crashes with my new GPU. And only with heavy load (a scene I made with gazillions of mp3's and animations, and atoms, and...). Regular scenes all work with the GPU.

What the crash looks like is this:
* the scene file will load
* I see the VAM loading scene
* Atoms are being loaded
* It takes pretty long
* I see white vertical bars in the distance
* I hear static noise
* The progress bar is like half way, I see part of the scene forming (some atoms being displayed)
* My VR screen goes blank
* My Rift S goes from white led, to orange/red led, and the VR screen is black and unpowered
* My Windows 10 gives me a Rift S message: Rift S is not connected to the displaypot
* When I look at my computer monitor, I actually see the scene I'm loading
* Sometimes, I can actually move around with my headset, and see the scene moving around (although the Rift S screen is black)
* Rift S goes completely black, no leds, no powered screen
* No way to reconnect (by reloaing oculus software and/or replugging the cables)
* Rest of system is ok
* I need to restart my computer to be able to connect with the Rift again.

So based on this "report" and the fact that until now, all other VR applications and/or GPU stuff is working fine, this might be something to do with the Rift S / Oculus and VAM. But I'm at a loss what it might be.

I'll do the memory check you mentioned and the stresstest as well, just to be sure, but I'm puzzled how the Rtx 2070 was able to load everything, and the Rtx 3080 just crashes on the same (heavy load) scene.

Edit: I just did the stress test and all went well. So I'm glad to say that it's not the PSU. I'll run the mem test over night. And I'll look at an update for the BIOS.

I have one more question: do you really think updating the BIOS and checking memory will work? I mean, why would that give problems now with the 3080 where it didn't give problems with the 2070? (Honest question).

 

[Sally Whitemane]

I really doubt it, just an idea - maybe it's a VaM configuration issue. I mean that some setting is still set specific for your old GPU. I'm pretty sure you did this already - did you reset everything, clear cache, etc.?
But then I guess it should be crashing in desktop mode too.
I'm out of ideas and I have no clue about VR hardware :/
Checked manufacturer site of GPU if there is a new BIOS to flash for that card? But I doubt that is the problem too - GPU BIOS problem are rare and usually only happen in the early product life ... 3000-series cards have been out for a long time.

Nope, I have nothing more - sorry.

 

[HolySchmidt]

How many powercables run from your PSU to your GPU? I want to know exactly how many cables are connected to your PSU, not to your GPU.
Two or three? If you have any splitter-powercables (Y-cable) connected to your GPU, this is very probable your problem. Since your GPU has three connectors, you have to have three seperate cables.

I think you don't have any problems with other games because they are not as demanding as VaM is, especially this one scene. This can easily be found out with MSI Afterburner + RivaTuner and ingame statistics.

If you are at maybe 98% CPU load + 98% GPU load, it could be also your PSU. BeQuiet or Seasonic would be my advice. But in general, 750W should be safe.

I have a 3080ti FTW3, a 12700K and a BeQuiet Straight Power 10 750W with three(!) PCIe connections. No problems at all.

Edit
If I'm right your PSU only has two PCIe connection-ports available.
The Trio Z has a max power consumption of 370W.

 

[pinosante]

How many powercables run from your PSU to your GPU? I want to know exactly how many cables are connected to your PSU, not to your GPU.
Two or three? If you have any splitter-powercables (Y-cable) connected to your GPU, this is very probable your problem. Since your GPU has three connectors, you have to have three seperate cables.

I think I have two, so one of them, is split to fit into the second and third.

I'll open up my case and use three separate cables. I'm curious though, why would the stress test be fine (which stressed my system to 99% on CPU / memory and GPU without any problems...

 

[pinosante]

I think I have two, so one of them, is split to fit into the second and third.

I'll open up my case and use three separate cables. I'm curious though, why would the stress test be fine (which stressed my system to 99% on CPU / memory and GPU without any problems...

I have the cables done like on this picture:

https://i.redd.it/qfwh3kboeyl51.png

Far right. But you are saying that it's even better to use three separate cables? Can't hurt to try I guess...

 

[pinosante]

Ghe... and this is what's shown on the asus support page:

 

[HolySchmidt]

From what I know, 3 cables are recommended if you have 3 ports on your GPU. The question is: can you use 3 or does your PSU only have 2 ports?

 

[Sally Whitemane]

What ASUS shows there is the solution for a card that could have higher power limit than yours. So that may not apply to your card.

As long as the GPU draws from two cables it's safe and nothing is going to melt - here are some numbers:

• PCI-Express slot provides 75 Watts
• One 6+2 connector provides 150 Watts. Cables can handle more I think.
• Even with only two connected via two cables that is already 375 Watts.

MSI RTX 3080 Gaming Z Trio has power limit of 380 Watts. Nothing will melt with 5 Watts "missing".
The third connector is only really required for overclocking then - one could probably run the card without the third 6+2 if it allows that.

How does the VR headset get it's power? That I do not know.

 

[pinosante]

From what I know, 3 cables are recommended if you have 3 ports on your GPU. The question is: can you use 3 or does your PSU only have 2 ports?

Only 2...

 

[pinosante]

Ok, so interesting update: I decided to try MacGruber's Benchmark and see what happens. VAM in desktop just crashes after the sim clothing test. So warm up 1, warm up 2, clothing all work fine. But once it loads the next test, VAM just becomes grey and freezes... There is definitely something not going right with the 3080 under load. .

 

[pinosante]

Could one of you, who also have a 3080, maybe make a screenshot of the nvidia settings? Or check if they have anything different from mine?

I really appreciate you thinking with me. I'm at a total loss.

 

[HolySchmidt]

What you can try is: undervolt the card to 0.750V or a little more (0.760V) at 1700MHz. Just for testing power consumption. This should be equal to around 270W max maybe.
My guess still: 2 cables is not enough.

It would be helpful if you could make a log-file with MSI Afterburner to see when exactly your picture is freezing/crashing and what happens hardware-sided.
I mean we have nothing: no temps, no power consumption, no clock-speeds, no voltage or whatever.

 

[pinosante]

What you can try is: undervolt the card to 0.750V or a little more (0.760V) at 1700MHz. Just for testing power consumption. This should be equal to around 270W max maybe.
My guess still: 2 cables is not enough.

It would be helpful if you could make a log-file with MSI Afterburner to see when exactly your picture is freezing/crashing and what happens hardware-sided.
I mean we have nothing: no temps, no power consumption, no clock-speeds, no voltage or whatever.

I will!

 

[Sally Whitemane]

Two cables with two 6+2 connector provide 300 Watt, probably more with a third connector, I only know the connector limits, but not cable limits, plus 75 Watts from PCI Express slot => 375 Watts.

RTX 3090 Founder Edition here - but all those settings are the same.

In MSI Afterburner you can limit the cards total power draw with a simple % slider.
You could just lower that to something like 80% and then hit apply at the bottom to test if it still crashes then.
So that 80% would mean a power limit of 304 Watts for your specific card. (0.8 * 380 Watts card power limit)

Tried VaM_Upater.exe > Options > Sync / Repair All to make sure all VaM files are okay?

 

[HolySchmidt]

Right, for this purpose it will be fine.

 

[pinosante]

Update: I retried the Benchmark with the latest Nvidia Drivers (I've been trying older drivers and newer drivers) and with the newest drivers at least I was able to pass the Benchmark in both Desktop + VR. Loading my custom scene still crashes. I'll download afterburner and try your lower power settings. I'll also try to fix VAM with the updater, didn't know that was possible.

 

[pinosante]

Ok, so I tried limiting my power to 75% with Afterburner, but that still didn't prevent VAM crashing. I have attached the afterburner log of the crash. The crash happens around the 23:06:19 mark I think. Roughly, could be 20 seconds earlier or later. Any idea what's going on? To me it looks like VAM exactly crashes at the moment the used video memory goes past 7 Gb. It is also capped at 7 Gb... After the crash I kept VAM on for 20 seconds, before I turned off VAM and the memory usage went back to 0.

So something with my GPU memory being flooded? (Or faulty?)

(But I have also done some machine learning tasks with CUDA, where 9 Gb of the 10 Gb was used, without problems... so why would it make VaM crash?)

 

[Sally Whitemane]

I see it the same way as you already diagnosed yourself.
This hints towards faulty VRAM. But RAM usage is high at that time too.
Since the 3080 is the new part it should be the VRAM by that logic. You do not run out of memory - so it should not have crashed.

Memory is always very hard to diagnose. I've had a server randomly crash and it did not show any error in a mem-test. Then I ran it longer and it took 5 passes to show errors. The memory was bad, but in like 99.99% of the time it worked.
In your machine learning tasks or games the faulty VRAM bits may randomly be used by irrelevant data that will not lead to a crash - like in a texture for a game a pixel has a different color now and your would never notice. Testing memory without specialized software is just a waste of time.

Then it also depends on the physical (not virtual) memory address whether a software will crash.
If the memory IC with addr. 0x0000.0000.0000.1000 is bad and used all the time it will crash software instantly. It's the first 4 KB used.
If the memory IC with addr. 0x0000.0002.4000.0000 is bad it can only crash if more than 9 GB was used.

I'd run a looong RAM test at night now to make sure this can be crossed of the list of potential problems.

To test the VRAM I know there is a leaked Nvidia tool floating around on the internet to test VRAM. It is hard to find. But I'm not sure it will work with the 3000-series. I've seen that used by people who do GPU repairs to diagnose bad VRAM.

 

[wolverine333]

I run a Rift S as well, I'm wondering what kind of crashing you're experiencing, screen freezing in headset with loud static? Or does VaM close automatically? I've had some issues in the past and may be able to help. I'm running 16 core, 3070 with 64gb ram, M.2 etc.

 

[wolverine333]

sorry just saw on part of the thread you mentioned static. I had the same problem and it was a nightmare. My headset would freeze the image in the headset and make a loud white static noise and go black. I would have to do the same thing you mentioned and restart system. The problem is the rift cables faulty, I got a mint Rift S in the box with cable controllers etc for 150$ on offerup. I knew that was the issue because I used the new cable on my old headset and haven't had an issue since. It's been well over 3 months now. I hope that may help you.

 

[pinosante]

sorry just saw on part of the thread you mentioned static. I had the same problem and it was a nightmare. My headset would freeze the image in the headset and make a loud white static noise and go black. I would have to do the same thing you mentioned and restart system. The problem is the rift cables faulty, I got a mint Rift S in the box with cable controllers etc for 150$ on offerup. I knew that was the issue because I used the new cable on my old headset and haven't had an issue since. It's been well over 3 months now. I hope that may help you.

Hmm is that cable something I can replace? And did this happen for all vam related stuff? In my case it is only for heavy duty scenes…? Was that your experience? Really curious what happened in your case!