Guides How to use Passthrough in VAM (VR/AR)

[Battletoad08]

Try possessing a male with skin the same color as your passthrough.

That will just make him invisible? Im talking about when you are in passthrough your hands and anything else is always behind the female character. like if you put your hand infront of her face, it still goes behind her head.

Any tips on the quest 3 hand tracking? it looses tracking so easy even with all the lights on.

 

[thinoreos]

That will just make him invisible? Im talking about when you are in passthrough your hands and anything else is always behind the female character. like if you put your hand infront of her face, it still goes behind her head.

Any tips on the quest 3 hand tracking? it looses tracking so easy even with all the lights on.

What do you mean hands are behind the female character? Something is wrong with your tracking, it shouldn't be behind her head. Take a video.

You might need to clean the front cameras of your Quest, tracking should be very good with sufficient lights

 

[VRCube]

That will just make him invisible? Im talking about when you are in passthrough your hands and anything else is always behind the female character. like if you put your hand infront of her face, it still goes behind her head.

Any tips on the quest 3 hand tracking? it looses tracking so easy even with all the lights on.

I haven't tried it, but if the VR arms you're posessing are the passthrough color, they will create a passthrough window so you can see your physical hands. It would be a kind of jerry rigged dynamic occlusion.

There's no way to have your hands in front of the virtual content without dynamic occlusion. It's theoretically possible to implement in VAM but it has not been done.

Depth API for Unity | Oculus Developers

Depth API exposes depth maps representing the per-frame sensed environment depth. It may be used for a variety of depth-based visual effects. Occlusion is a mixed reality (MR) software feature supported by this API. It enables occluding (hiding, masking) virtual objects behind real world...

developer.oculus.com

 

[Battletoad08]

I haven't tried it, but if the VR arms you're posessing are the passthrough color, they will create a passthrough window so you can see your physical hands. It would be a kind of jerry rigged dynamic occlusion.

There's no way to have your hands in front of the virtual content without dynamic occlusion. It's theoretically possible to implement in VAM but it has not been done.

Depth API for Unity | Oculus Developers

Depth API exposes depth maps representing the per-frame sensed environment depth. It may be used for a variety of depth-based visual effects. Occlusion is a mixed reality (MR) software feature supported by this API. It enables occluding (hiding, masking) virtual objects behind real world...

developer.oculus.com

Yea thats what I mean, that would be crazy immersive. Who can implement that?

 

[thinoreos]

Yea thats what I mean, that would be crazy immersive. Who can implement that?

Oh you mean Occlusion, VAM 2 could ?

 

[marlowInc]

How do I make this work in SteamVR launch mode? I got the hand tracking to work now but the background color plugin + background session plugin both stopped functioning. The passthrough function is working, as I can see through anything blue, but the plugin to make the entire background blue isn't.

 

[Battletoad08]

How do I make this work in SteamVR launch mode? I got the hand tracking to work now but the background color plugin + background session plugin both stopped functioning. The passthrough function is working, as I can see through anything blue, but the plugin to make the entire background blue isn't.

the hand tracking from virtual desktop sucks for me

 

[thinoreos]

the hand tracking from virtual desktop sucks for me

Again, you should clean your lenses and have proper lighting... Ours work very well as you can see in the videos

 

[thinoreos]

How do I make this work in SteamVR launch mode? I got the hand tracking to work now but the background color plugin + background session plugin both stopped functioning. The passthrough function is working, as I can see through anything blue, but the plugin to make the entire background blue isn't.

You mean Sally's color background? Can you explain more as in why it isn't working? Try resetting the plugin, or going into passthrough first THEN doing hand tracking

 

[Nameless Vagabond]

passthrough is possible on index, you can enable the camera in steamvr settings and after a restart from there enable 3d room view and translucent passthrough room view style, in your vr menu there is an eye icon to then display your camera, it will overlay the camera on your vr, simply load into vam and as long as there is a pure black background your room will be visible in the scene space.

this is janky, and your room isn't entirely visable, nor is it actually ar passthrough, however it "works" if you're thinking all is lost on index, hopefully someone can implement a better system because the index is very capable of fully showing your room (albeit in a laggy way if you want depth) but layering anything on top of it is but a dream

Thanks for bringing this up, I had no idea that this was even an option on the index. It's just as janky as you described, and the girl I brought into my room seemed to be a bit more translucent than what your video showed, but at least it gave me a taste of what AR passthrough is like... which is very nice!

I can't wait until a meta alternative comes around that can do passthrough like the quest 3 can

 

[trety]

Does anyone have 'generic' android build with custom color passthrough?
Since i didn't like Q3 and returned it, i was looking for some alternatives, and found HTC Elite XR headset. It has lower res cameras than Q3, but still does the job. I got it working with newest ALVR [18.11] server and ALXR client with 'black' passthrough, using 'v0.32.0+nightly.2024.04.23', but it kills most of my models -_-
Unfortunately Pico and Quest apk doesn't work on this headset, they shown as '2d' apps and crashing.

 

[osiris23]

Does anyone have 'generic' android build with custom color passthrough?
Since i didn't like Q3 and returned it, i was looking for some alternatives, and found HTC Elite XR headset. It has lower res cameras than Q3, but still does the job. I got it working with newest ALVR [18.11] server and ALXR client with 'black' passthrough, using 'v0.32.0+nightly.2024.04.23', but it kills most of my models -_-
Unfortunately Pico and Quest apk doesn't work on this headset, they shown as '2d' apps and crashing.

I don't know of any but I'm surprised about the Q3. I purchased mine back in Dec and it was solid. What didn't you like about it?

 

[trety]

I don't know of any but I'm surprised about the Q3. I purchased mine back in Dec and it was solid. What didn't you like about it?

It sucks wired - and my VR usage is a bit specific - i can stay all night long wearing HMD.
It was also super heavy on my face [with standard strap].

I have pretty weird internet connection - i'm living in my company house most of the year, and i have no access to house router config. I tried to connect my own as a bridge years back, but failed completely.
So as for Quest, i wouldn't even be able to have it both - connected to internet and my PC at the same time wirelessly.

If it would be pure PCVR headset i would definatelly stick to it - lenses, passtrough, even the controllers tracking were awesome.

 

[thinoreos]

Does anyone have 'generic' android build with custom color passthrough?
Since i didn't like Q3 and returned it, i was looking for some alternatives, and found HTC Elite XR headset. It has lower res cameras than Q3, but still does the job. I got it working with newest ALVR [18.11] server and ALXR client with 'black' passthrough, using 'v0.32.0+nightly.2024.04.23', but it kills most of my models -_-
Unfortunately Pico and Quest apk doesn't work on this headset, they shown as '2d' apps and crashing.

Yeah that's running into generic APK territory, not sure if it will work in any way or form, kinda surprised you got ALXR going... If you can get that running, I think your best bet is to switch all your models & hair into bright colors ?

Regarding the Quest 3, the standard strap sucks. You defo need a battery strap to balance it out so it sits very comfortably on your face, something Meta should tell users

 

[moonunit46]

Try possessing a male with skin the same color as your passthrough.

Do you know of any plugins that could streamline the process? I've made a male with a custom texture that's the same color as the chroma key but it's a pretty imperfect implementation so far

 

[trety]

I contacted ALXR dev in my case - 'custom color' passthrough for generic android HMDs...
He just shared in my thread non public, dev, version of ALXR with passthrough color and method sliders.
It works on Elite XR!

Custom color passthrough · Issue #106 · korejan/ALVR

Hi, recently i bought HTC Vive Elite hmd, mostly for passthrough features and it turned out a way more complicated than i expected. I had tried ALXR and it works great, except... it uses 'black' co...

github.com

 

[VRCube]

Do you know of any plugins that could streamline the process? I've made a male with a custom texture that's the same color as the chroma key but it's a pretty imperfect implementation so far

It didn't work for me either when I tried it. The shadows and highlights on the hands just mess it up completely.

 

[thinoreos]

I contacted ALXR dev in my case - 'custom color' passthrough for generic android HMDs...
He just shared in my thread non public, dev, version of ALXR with passthrough color and method sliders.
It works on Elite XR!

Custom color passthrough · Issue #106 · korejan/ALVR

Hi, recently i bought HTC Vive Elite hmd, mostly for passthrough features and it turned out a way more complicated than i expected. I had tried ALXR and it works great, except... it uses 'black' co...

github.com

Interesting! How is it going so far with your Elite XR with this? Can you post some videos? Interested to see

 

[rollingrick]

When I launch vam with virtual desktop, my quest 3 controllers disappear, and I'm forced to use mouse, though i cant see mouse pointer in my vr headset. Also the scene seem super directional. I can't move characters, it just stops me. Trying to get them out of my walls...

 

[thinoreos]

When I launch vam with virtual desktop, my quest 3 controllers disappear, and I'm forced to use mouse, though i cant see mouse pointer in my vr headset. Also the scene seem super directional. I can't move characters, it just stops me. Trying to get them out of my walls...

Is this your first time using Virtual Desktop with VAM? Try watching the tutorials on Virtual Desktop with SteamVR.... sounds like you're launching in desktop mode

 

[trety]

Interesting! How is it going so far with your Elite XR with this? Can you post some videos? Interested to see

Sorry for late reply...
Elite Xr has lower res front cameras than q3 i guess, and after further testing [hmd in general] i wouldn't reccomend it for anyone lol
I was tempted with passthrough in general, but it's damn impossible to keep that thing charged long enough to have fun with it - i just spend about one hour in VaM passthrough mode and it drained almost 60% of the battery while being connected with my pc via usb >_<
Without a battery, in 'glasses mode' it's like 10min and it's dead ...
Either way, since i didn't like quest 3, mostly for the weight and being forced to buy VD for it. I'm gonna keep XR for a while till we will get something better, other than q3.

Here is short MMD motion i recorded in passthrough

 

[thinoreos]

Sorry for late reply...
Elite Xr has lower res front cameras than q3 i guess, and after further testing [hmd in general] i wouldn't reccomend it for anyone lol
I was tempted with passthrough in general, but it's damn impossible to keep that thing charged long enough to have fun with it - i just spend about one hour in VaM passthrough mode and it drained almost 60% of the battery while being connected with my pc via usb >_<
Without a battery, in 'glasses mode' it's like 10min and it's dead ...
Either way, since i didn't like quest 3, mostly for the weight and being forced to buy VD for it. I'm gonna keep XR for a while till we will get something better, other than q3.

Here is short MMD motion i recorded in passthrough
View attachment 373571

Interesting! I knew the battery was small, but 10 minutes?? Damn... Looks decent in the video though

 

[rollingrick]

Is this your first time using Virtual Desktop with VAM? Try watching the tutorials on Virtual Desktop with SteamVR.... sounds like you're launching in desktop mode

I thought the same thing, but switching to vr mode didn't help. But you are correct. My first time. I'll try looking for the tutorials

 

[mrdicklick]

It does not work for me with pico4 and alxr.

If you start alxr it starts steamvr automatically.
But you also have to start the pico streaming assistant for connection between PC and pico via usb. And the streaming assistant wants also to start the steamvr.

And that's the problem, because steamvr was already started by alxr and you can't make a connection between Pico and streaming assistant / steamvr.

Anybody here with pico4 got it running?

 

[thinoreos]

It does not work for me with pico4 and alxr.

If you start alxr it starts steamvr automatically.
But you also have to start the pico streaming assistant for connection between PC and pico via usb. And the streaming assistant wants also to start the steamvr.

And that's the problem, because steamvr was already started by alxr and you can't make a connection between Pico and streaming assistant / steamvr.

Anybody here with pico4 got it running?

I know this is probably not the answer you want to hear, but Virtual Desktop is easier to set up with Pico 4 for this. ALXR whenever you have bugs you will have to debug with them on their discord or github