Guides How to use Passthrough in VAM (VR/AR)

[bd-energy]

@thinoreos hey, dev of alxr here, just want to clear-up some misunderstandings and mistakes in the guide.

First if you don't need to use a green chroma-key please use the builds from here they are much newer and have various fixes/improvements. There are server builds there too that you must use. Since v0.11 I've intentionally made the clients not compatible with the offical v18.2.3 server build and provide custom builds for a number of reasons.

Mainline ALVR does not support passthrough (modes) currently, this a feature i implementated in alxr, the clients are very different. The "greenscreen" versions is the alxr client with the masking shader modified.

In your guide, for the supported headsets your referencing from the wrong place, the list is here (ignore the rest of the info there I need to update it). List of headsets that have passthrough support are:

Quest 2
Quest Pro,
Vive Focus 3
XR Elite
Pico Neo 3
Pico 4

Hololens 2
Magic Leap 2

The devices before the last 2 require explicit support through OpenXR extensions. The last 2 plus any other (AR) device with an additive enviromental blend mode in core spec OpenXR gets it for "free" because this mode always treats pure black pixels as transparent without any additional changes. I need do a bit more work on core spec blend modes.

I have no clue if YVR 1/2 is supported by alxr currently, there certainly isn't full controller bindings for it.

Some wired headsets like Valve index are technically possible (I have linux/windows clients) but the ones that only have an OpenXR runtime via SteamVR, to do iti will be very really convoluted and wont be worth the time.

The places where it says alvr apk please change it to alxr as that might cause confusion, and again please update your guide for the black chroma-key case to use the client & server builds from nightly repo.

 

[bd-energy]

Something I forgot to mention, I've merged in my facial & eye tracking branches to main branch but to use that requires a plugin since these can't be supported by OpenVR.

 

[thinoreos]

@thinoreos hey, dev of alxr here, just want to clear-up some misunderstandings and mistakes in the guide.

First if you don't need to use a green chroma-key please use the builds from here they are much newer and have various fixes/improvements. There are server builds there too that you must use. Since v0.11 I've intentionally made the clients not compatible with the offical v18.2.3 server build and provide custom builds for a number of reasons.

Mainline ALVR does not support passthrough (modes) currently, this a feature i implementated in alxr, the clients are very different. The "greenscreen" versions is the alxr client with the masking shader modified.

In your guide, for the supported headsets your referencing from the wrong place, the list is here (ignore the rest of the info there I need to update it). List of headsets that have passthrough support are:

Quest 2
Quest Pro,
Vive Focus 3
XR Elite
Pico Neo 3
Pico 4

Hololens 2
Magic Leap 2

The devices before the last 2 require explicit support through OpenXR extensions. The last 2 plus any other (AR) device with an additive enviromental blend mode in core spec OpenXR gets it for "free" because this mode always treats pure black pixels as transparent without any additional changes. I need do a bit more work on core spec blend modes.

I have no clue if YVR 1/2 is supported by alxr currently, there certainly isn't full controller bindings for it.

Some wired headsets like Valve index are technically possible (I have linux/windows clients) but the ones that only have an OpenXR runtime via SteamVR, to do iti will be very really convoluted and wont be worth the time.

The places where it says alvr apk please change it to alxr as that might cause confusion, and again please update your guide for the black chroma-key case to use the client & server builds from nightly repo.

Thank you for all the info & constant updates, updated as so!

 

[thinoreos]

thinoreos updated How to use Passthrough in VAM (VR Mixed Reality) with a new update entry:

1.5

Now using latest version of ALXR's nightly builds

Read the rest of this update entry...

 

[VAMIC]

I followed these instructions and got the APK installed on my quest 2 via Sidequest along with the server installed on my PC. When I run the app from my Quest (by going to Apps from Unknown Sources) I get a greenish screen but nothing else, no menu commands etc. So when I go back to my PC and try to run VAM nothing shows up or populates in this green screen. Also, I get no responses to pressing any buttons. I've been doing it wirelessly. Is it required to do this when connected via wire to a PC? This part isn't clear. Also, what is supposed to happen when I boot up ALXR?

 

[fatpatfitzgerald]

I'm also having issues with this newer ALXR version. I've performed a clean install on server and client multiple times (both newer animetiddyenthusiast and korejan versions), and once I launch ALXR from the Quest 2, my Quest screen glitches out and CPU usage shoots up to 100%, showing VRServer at 95+%.

No amount of tweaking seems to fix this, and even the new nightly is having issues. I'm going to try it with an earlier v0.11 nightly, but so far the only version I can get to 'work' for the Quest 2 is the 18.2.3 server and previous green or black ALXR v0.10 versions.

Edit: v0.11.0+nightly.2023.08.05 seems to work fine. Not sure what's going on with the v0.12s
Edit2: The last v0.11 server seems to work with the updated animetiddyenthusiast 'White screen' modded apk client. Super odd. I'll try the v0.12 server again later.

 

[thinoreos]

I followed these instructions and got the APK installed on my quest 2 via Sidequest along with the server installed on my PC. When I run the app from my Quest (by going to Apps from Unknown Sources) I get a greenish screen but nothing else, no menu commands etc. So when I go back to my PC and try to run VAM nothing shows up or populates in this green screen. Also, I get no responses to pressing any buttons. I've been doing it wirelessly. Is it required to do this when connected via wire to a PC? This part isn't clear. Also, what is supposed to happen when I boot up ALXR?

The greenish screen is the loading screen of the ALXR apk, that part on your Quest is correct. It seems like you did not set up & launch ALVR launcher on your PC? (Step 4)

Also noted will make it clearer that it's wireless & not wired!

 

[thinoreos]

I'm also having issues with this newer ALXR version. I've performed a clean install on server and client multiple times (both newer animetiddyenthusiast and korejan versions), and once I launch ALXR from the Quest 2, my Quest screen glitches out and CPU usage shoots up to 100%, showing VRServer at 95+%.

No amount of tweaking seems to fix this, and even the new nightly is having issues. I'm going to try it with an earlier v0.11 nightly, but so far the only version I can get to 'work' for the Quest 2 is the 18.2.3 server and previous green or black ALXR v0.10 versions.

Edit: v0.11.0+nightly.2023.08.05 seems to work fine. Not sure what's going on with the v0.12s
Edit2: The last v0.11 server seems to work with the updated animetiddyenthusiast 'White screen' modded apk client. Super odd. I'll try the v0.12 server again later.

I think this might be a server thing on ALVR's nightly part, but I am not the dev & could be wrong about this

 

[bd-energy]

I'm also having issues with this newer ALXR version. I've performed a clean install on server and client multiple times (both newer animetiddyenthusiast and korejan versions), and once I launch ALXR from the Quest 2, my Quest screen glitches out and CPU usage shoots up to 100%, showing VRServer at 95+%.

No amount of tweaking seems to fix this, and even the new nightly is having issues. I'm going to try it with an earlier v0.11 nightly, but so far the only version I can get to 'work' for the Quest 2 is the 18.2.3 server and previous green or black ALXR v0.10 versions.

Edit: v0.11.0+nightly.2023.08.05 seems to work fine. Not sure what's going on with the v0.12s
Edit2: The last v0.11 server seems to work with the updated animetiddyenthusiast 'White screen' modded apk client. Super odd. I'll try the v0.12 server again later.

@fatpatfitzgerald let me know what you're setup is, are using an AMD gpu? from the nightly repo you can use any version of the server from 0.11 onwards. Try the version of the server that works for you with 0.12 clients and let me know.

 

[Saint66]

Hi, for me everything is running as expected (using ALXR since a long time).
However, I have an issue after updating to the 18.5 server. Its also working fine, no problems with using ALXR with it.

But now with 18.5, everytime when I start SteamVR, the ALVR server also starts automatically. Some times I want to launch VAM with OpenVR through VirtualDesktop, but the automatic start of the server wont let me launch it in VR (it starts VAM in desktop mode). I start it via a batchfile like this: "START "VaM" VaM.exe -vrmode OpenVR". But the server also starts when I just open SteamVR via shortcut.

So, two questions:
1. can I prevent the start of the ALVR server when launching SteamVR? I found nothing in ALVR's settings
2. would your updated client work with oolder versions, like 18.4 or 18.2.3?

This is running me crazy, never had that with older versions

Ok, you can disable it in SteamVR settings, but then the server is not starting at all of course.
So, I have to switch it on/off situational, no biggie:

 

[bd-energy]

Hi, for me everything is running as expected (using ALXR since a long time).
However, I have an issue after updating to the 18.5 server. Its also working fine, no problems with using ALXR with it.

But now with 18.5, everytime when I start SteamVR, the ALVR server also starts automatically. Some times I want to launch VAM with OpenVR through VirtualDesktop, but the automatic start of the server wont let me launch it in VR (it starts VAM in desktop mode). I start it via a batchfile like this: "START "VaM" VaM.exe -vrmode OpenVR". But the server also starts when I just open SteamVR via shortcut.

So, two questions:
1. can I prevent the start of the ALVR server when launching SteamVR? I found nothing in ALVR's settings
2. would your updated client work with oolder versions, like 18.4 or 18.2.3?

This is running me crazy, never had that with older versions

Ok, you can disable it in SteamVR settings, but then the server is not starting at all of course.
So, I have to switch it on/off situational, no biggie:
View attachment 281602

@Saint66 If you unregister the driver it should disable the auto start and only load the driver when you run the server launcher.

 

[bd-energy]

sorry I missed the second question, 18.2.3 isn't supported anymore mainly because of the UDP bug fix(es) cherry-picked into the newer versions, it needs to be applied to both the client and server but it break compatibility with v18.2.3. The offical 18.2.3 build is also a very dated binary and just plain not good. It was never catered for an OpenXR based client, everything pre-v20 was based around a client using Oculus SDK which alxr-client never was. Together with the UDP fix and the bad experience it can be I intentionally made it not be able to connect to that version of the server.

For 0.11 clients onwards you're free to use which ever server versions are in the nightly builds.

 

[Saint66]

@Saint66 If you unregister the driver it should disable the auto start and only load the driver when you run the server launcher.

View attachment 281605

Yeah I tried that, although I had installed ALVR instead of the zip-option, so my drivers was like that -> " C:\Program Files\ALVR "
Anyway, removing it leads to Server not starting anymore at all, it just shows a splash screen and closes itself again.
Reinstall needed.

 

[thinoreos]

Hi, for me everything is running as expected (using ALXR since a long time).
However, I have an issue after updating to the 18.5 server. Its also working fine, no problems with using ALXR with it.

But now with 18.5, everytime when I start SteamVR, the ALVR server also starts automatically. Some times I want to launch VAM with OpenVR through VirtualDesktop, but the automatic start of the server wont let me launch it in VR (it starts VAM in desktop mode). I start it via a batchfile like this: "START "VaM" VaM.exe -vrmode OpenVR". But the server also starts when I just open SteamVR via shortcut.

So, two questions:
1. can I prevent the start of the ALVR server when launching SteamVR? I found nothing in ALVR's settings
2. would your updated client work with oolder versions, like 18.4 or 18.2.3?

This is running me crazy, never had that with older versions

Ok, you can disable it in SteamVR settings, but then the server is not starting at all of course.
So, I have to switch it on/off situational, no biggie:
View attachment 281602

Since you're using Quest, set the BAT file like this: Launching VAM without SteamVR

 

[Saint66]

Since you're using Quest, set the BAT file like this: Launching VAM without SteamVR

Yeah thx, I have that for standard use as well, but afaik it uses then the Oculus runtime. For some cases I need OpenVR runtime used
But this is not the main problem, already found a temporary solution.

My real problem is that the server starts whenever I start SteamVR. De-registering the driver to avoid that breaks the server for me, it just shows a short loading screen and shutdown itself then.

 

[bd-energy]

Yeah I tried that, although I had installed ALVR instead of the zip-option, so my drivers was like that -> " C:\Program Files\ALVR "
Anyway, removing it leads to Server not starting anymore at all, it just shows a splash screen and closes itself again.
Reinstall needed.

Did you leave the driver blocked in the "manage steamvr add-ons" dialog after unregistering it? that would happen if so.

 

[Saint66]

Did you leave the driver blocked in the "manage steamvr add-ons" dialog after unregistering it? that would happen if so.

No, I didn't.
This is how it looks right after deregistering the driver:

But when I close ALVR and Steam and restart SteamVR, the option is gone:

ALVR does not start anymore then

 

[bd-energy]

No, I didn't.
This is how it looks right after deregistering the driver:
View attachment 281894

But when I close ALVR and Steam and restart SteamVR, the option is gone:
View attachment 281895

ALVR does not start anymore then

Yeah that is normal because it's not registered driver, when it's not loaded it wont show in the add-ons dialog. If you're not already using it, try the non-installer version.

 

[Saint66]

Yeah that is normal because it's not registered driver, when it's not loaded it wont show in the add-ons dialog. If you're not already using it, try the non-installer version.

Yeah I already switched. So I gave up and doing things manually for each purpose, no biggie. And thx

 

[javoom]

Thanks for this! But is there any way to use this passthrough method with ALXR ALVR on a remote cloud based machine like Shadow? Currently using Virtual Desktop but does ALXR only work locally like Air Link?

 

[bd-energy]

Thanks for this! But is there any way to use this passthrough method with ALXR ALVR on a remote cloud based machine like Shadow? Currently using Virtual Desktop but does ALXR only work locally like Air Link?

I knew of some shadow users who have in the past, i don't remember what they did exactly, you can try reading this

 

[thinoreos]

thinoreos updated How to use Passthrough in VAM (VR Mixed Reality) with a new update entry:

1.6

Updated to ALXR 0.13.0

Read the rest of this update entry...

 

[TRSD]

This is extremely cool. I almost got it working.

Can anyone else confirm the latest version (v0.13.0+nightly.2023.09.07) working on a Quest 2? I got everything set up. Even seen the effect working in VAM. But controller inputs aren't being forwarded to SteamVR/VAM, no inputs are being registered in SteamVR's controller debug. They track properly.

EDIT: This works. EDIT: But not for VAM. Fixed SteamVR's inputs though.

 

[Mel]

I have just followed your guide, and downloaded the new ALVR for green screen, it works ,but my controllers lose tracking and I cannot select anything, is there another settingI need to alter. TIA

 

[thinoreos]

I have just followed your guide, and downloaded the new ALVR for green screen, it works ,but my controllers lose tracking and I cannot select anything, is there another settingI need to alter. TIA

So the last version I tested that works was the 2023.09.07 version, ALXR released a new version (09.08) right after. I have not tested that version, so it's a possibility something went off in that