Plugins + Scripts PassthroughKey

[CraSh.it]

CraSh.it submitted a new resource:

PassthroughKey - Clean Virtual Desktop chroma-key passthrough for VR

PassthroughKey
Clean Virtual Desktop chroma-key passthrough for VaM – by CraSh.it

Put your VaM characters into your real room on Quest, with a clean key: no holes in the character, no fringing on the background, and a VaM menu that stays solid.

What it does
Virtual Desktop's PC passthrough works by chroma key: it cuts out every pixel close to one chosen colour. The catch is that the colour has to be exact, and anything near it gets cut too...

Read more about this resource...

 

[CraSh.it]

PassthroughKey – User Manual
Version 2.0​

What PassthroughKey does

PassthroughKey puts your VaM characters into your real room, using Virtual Desktop's passthrough on a Quest headset.

Virtual Desktop shows your room wherever the picture has one exact background colour (the "key"). PassthroughKey takes care of that for you:

• Clean background. The background is always the exact colour Virtual Desktop removes, even with lighting, bloom or other effects in the scene.
• No holes in the character. Dark hair, shadows and clothes stay solid, even with a black key. Fine hair strands are kept too.
• Solid VaM menu. The menu and its text stay visible instead of being cut out.
• Room hidden automatically. Floors, walls, props, sky, fog and light effects are hidden, people stay. A list lets you bring back anything you want, per scene.
• Floor shadows. Characters cast shadows on your real floor, and their body, hair and clothes rest on it.
• Light matching. Pick what lights your room, point at your real lights with the controller (or draw your window), and the scene lighting matches your room – even the sun shining through your window.
• Room anchor. After a headset sleep or a scene load, the character goes back to the same spot in your room.
• Profiles. Six slots to save your setup; one of them loads when VaM starts.

You need: VaM, a Quest headset (tested on Quest 3) and Virtual Desktop. Quest Link and Air Link are not supported. They have no chroma-key passthrough.

Installing and first start

1. Download the plugin (put the .var file into your VaM/AddonPackages folder) and start VaM.
2. Open the main menu, go to Session Plugins and add PassthroughKey from the package. A session plugin stays loaded when you change scenes.
3. Open the plugin's window. The status box on the main page should say active | shader OK.
4. Optional, recommended: in Session Plugin Presets choose Change User Defaults, so PassthroughKey loads every time VaM starts.

Your settings are not saved with scenes. Save them in a profile (see Profiles) so they come back next time.

Setting up Virtual Desktop

In the headset, open the Virtual Desktop menu and go to Settings > Streaming > VR Passthrough. Turn passthrough on and set:

Setting	Value
Key colour (R G B)	The three numbers shown in the plugin's Virtual Desktop box
Similarity	The same value as the plugin's VD Similarity % (default 5%)
Smoothness	The same value as the plugin's VD Smoothness % (default 5%, needed for floor shadows)
Sharpening	0%
Gamma (Streaming tab)	1.0. Lower values darken the image after the plugin, so dark hair, lashes and clothes turn see-through. For a darker look use the plugin's Character gamma instead.
Snapdragon Game Super Resolution	Off

Important: type the numbers from the plugin's Virtual Desktop box, not the colour values from the colour picker. Virtual Desktop sees colours slightly differently, and the plugin already works out the numbers it needs. For black, white and pure colours both are the same.


Main page

The main page shows how everything is running, the values for Virtual Desktop, and the way into the six settings pages.

Status box (left).

• PassthroughKey v2.0: the version you have.
• active / off and shader OK: whether the plugin is working ("off (desktop mode)" means it is paused because VaM is not in VR). If it says "bundle not found" or "shader missing", see Troubleshooting.
• Key: the key colour in use.
• Environment: how many scene objects are hidden.
• Post effects: how many effects (bloom, grain and so on) are paused.
• Lights: whether light matching is set up and applied.

Required Virtual Desktop settings (right). Everything to enter in Virtual Desktop: the key colour as R G B, Similarity %, Smoothness %, Sharpening 0%, Gamma 1.0 and Super Resolution off. If you change Smoothness in the plugin, the new value shows in red until you set it in VD too.

Page buttons. Six buttons open the settings pages: Key & Virtual Desktop, Scene handling, Room & Shadows, Profiles, Light matching, Advanced. Every page has a Back button at the top right.

Key pass active. Turns the plugin on or off. Off puts the scene back exactly as it was.

Disabled in desktop mode. On by default. The plugin only works while VaM is in VR; on the monitor you see the normal scene. It switches by itself between VR and desktop mode, as long as Key pass active is ticked.

Quick guide. The five setup steps in short: load as a Session plugin, use the defaults and match Virtual Desktop, calibrate the floor (Room & Shadows), set up light matching, save a startup profile.

Reset all settings to default. Click once, then again within 3 seconds to confirm. All settings go back to default; captured lights and the room anchor are kept.


Key & Virtual Desktop

This page sets the background colour that Virtual Desktop removes, and how safely the character is protected from being cut.

Key colour (left).

• Key preset: Black (default and recommended), Dark grey, Mid grey, Blue, Green, Magenta, or Custom.
• Colour picker: pick any colour. Changing it switches the preset to Custom. When you type a number, press Enter or wait 2 seconds and it is applied.
• Pick a colour that isn't in your character's hair or clothes. Black works well for most scenes, because the plugin protects dark areas.
• The help text below sums up the sliders on the right.

Virtual Desktop keyer (right).

• Virtual Desktop box: the same Required Virtual Desktop settings as on the main page. It updates as you change the colour or the sliders.
• VD Similarity %: how much Virtual Desktop cuts around the key colour. Set the same value here and in Virtual Desktop. 5 is a good start.
• VD Smoothness %: how softly Virtual Desktop fades the edge of the cut. Set the same value here and in VD. 5 (default) is needed for floor shadows; 0 if you don't use them.
• VD colour curve: how Virtual Desktop reads colours. It was measured at 0.89; leave it there.
• Extra safety margin: how far the character's colours are kept from the key colour. The default is 0.1. Lower keeps dark colours truer but can let holes appear in dark areas and in the VaM menu. Raise it if you see holes.

Fine hair is automatic: hair strands thinner than a pixel are kept, and the thin dark edge they bring is brightened to match the character.


Scene handling

This page decides what of the VaM scene stays visible, and what happens to visual effects. The hidden atoms list fills the left side; Environment and Post effects are on the right.

Environment (right).

• Hide environment (keep people): while the plugin is on, everything except people, the VaM menu and in-scene buttons is hidden: rooms, floors, furniture, props, fog, volume light shafts, particles. Items a person holds or wears stay, and so do the edit handles of every atom. New items are picked up every 2 seconds. Turning it off (or the plugin) shows everything again.
• Hide skybox: hides the scene's sky so it can't cover your room. It also hides sky spheres that some plugins build themselves (for example PictureToSkybox).

Hidden objects (left). Every atom the plugin hides is listed here, 14 to a page, so you can bring back what you want to keep in this scene.

• Ticked = hidden. Untick an atom to show it again (a chair to sit on, light shafts for atmosphere). Tick it to hide it again.
• People are listed first and are shown by default. Tick a person to hide them – body, clothes, accessories, hair and anything attached to them.
• Plugin-made sky spheres appear at the end, marked "scene root, sky".
• The bar at the top shows the page and how many atoms are listed and hidden. < Prev. page and Next page > move through the list.
• Your choice is saved per scene and comes back whenever you load that scene. It is not part of the profiles.
• Shown atoms can block your room behind them, the same as in any normal scene.

Post effects (right). Effects like bloom, grain or colour grading (for example PostMagic) change the background colour, which stops Virtual Desktop from removing it.

• Suspend (exact key): pauses those effects while the plugin is on. Recommended.
• Keep, snap background: keeps the effects and pulls the background back to the key colour.
• Snap tolerance (Keep mode): how far off the background may drift and still be pulled back. Raise it if the background flickers; lower it if dark edges of the character disappear.

Light matching

This page lights the character like your real room, so it looks like it belongs there. The lights stay fixed to your real room, even when you move around in VaM. Work through the numbered steps.

Top.

• Status box (left): whether your lights are captured and applied, and what just happened.
• Character gamma (right, default 0.85): a darker or brighter look for the character and everything else that isn't the key. Below 1 is darker. Use this instead of Virtual Desktop's Gamma, which must stay at 1.0.

Main light (left).

1. Preset: what lights your room – Ceiling light (default), Window (sunny), Window (cloudy), Lamp, TV/Monitor. The preset sets the light type, brightness and colour, and its own look for the floor shadows and the room fill (those sliders change when you pick a preset).
2. Capture MAIN light: measure your real light with the controller (see Capturing a light below). Each preset remembers its own capture: switch preset and the light jumps back to where that preset was captured. A preset you haven't captured yet shows "Not captured yet".

• Brightness and Colour (K): how strong and how warm the light is. Lower K is warm and orange, higher is cool and blue. The step buttons move it by 1, 10, 100 or 1000.
• Cast shadow on atoms: whether this light casts shadows on the character.

Second light (right). Optional, for a second light source. Tick the box in its header, then pick a preset and capture it like the main light. Main and Second remember their captures separately, so two lights of the same kind work (for example two lamps).

Capturing a light. A green laser comes out of the controller; aim with it and press Select (A or X) to confirm each step. B or Y cancels the capture and keeps your previous one. While you capture, the VaM menu and VaM's own laser pointers are hidden and VaM switches to Play mode, so nothing gets selected by accident; all of it comes back when you finish. A short message low in your view tells you each step.

Preset	What you do
Ceiling light, Lamp, TV/Monitor	Point at the light, press Select. Take a big step to the side, point at the same light again (a yellow line shows your first aim), press Select.
Window (cloudy)	Touch one bottom corner of the window glass, press Select; touch the other bottom corner, press Select. Point at the top edge of the glass – the outline follows the laser – and press Select. The light comes in softly through your window.
Window (sunny)	The same three steps as cloudy, then point at the middle of the sunlit patch on the floor and press Select. The sun shines through your window: only the sunlit area is lit, with sharp shadows.

Touch the glass, not the frame or the wall. The sunlit patch can only be picked on the floor for now.

Room.

• 3. Room fill mode: Soft (ambient) lights the shaded side evenly; Directional (one side) adds a soft light from one side.
• Room fill brightness and Room fill colour (K): for the chosen fill mode.
• PT lights replace scene lights: turns off the scene's own lights while yours are in use. Some scenes set a very low camera exposure, which makes your lights look dull; it is set to 1 while your lights are in use and put back when you remove them.
• Remove PT lights, restore scene lights: removes the plugin's lights and turns the scene's lights back on.
• 4. Apply / update lights: creates or updates the lights in the scene. Brightness, colour and fill sliders work live afterwards. Applied lights come back by themselves after a VaM start (startup profile) and after every scene load – save your profile once after pressing Apply. Remove PT lights turns this off.

Room & Shadows

This page puts the character on your real floor, keeps it in place in your room, and adds floor shadows.

Room anchor (left).

• Set room anchor (select 2 fixed points): touch point A with a controller and press Select, then point B at least 40 cm away – for example two table corners. This also saves where the character stands and which way it faces. Set it again whenever you move the character.
• Re-align room (select the same points): after a headset sleep or a scene load, touch the same points (A first) and press Select on each. The character, the lights and the shadows go back to the same spot and direction in your room. Height is left to Calibrate floor. If the points don't match, nothing changes and you get "Re-align failed".

Floor match (left).

• Calibrate floor height (with controller): touch the floor with a controller and press Select (B or Y cancels). Do this once; the result is saved in profiles.
• Calibration offset (m): fine-tunes the calibrated floor.
• Align scene to real floor: moves the scene so its floor sits on your real floor.
• Scene height offset (m): raises (+) or lowers (−) the scene against your floor.
• Body collides with real floor and Hair & cloth collide with real floor: the character rests on your floor instead of sinking through.
• Hide everything below the real floor (off by default) and Floor cut depth (m): cut off anything that pokes through the floor.

Floor shadows (right). Tick the header to turn them on (default on). Needs Virtual Desktop Smoothness 5. A teal notice reminds you when you switch them on.

• Shadow quality: 1K, 2K (default) or 4K.
• Near / Far shadow strength: how dark the shadow is where the character touches the floor, and further away.
• Near/far blend (m) and Fade distance (m): how quickly near turns into far, and how far the shadow fades out.
• Softer with distance and Contact blur (px): how much the shadow blurs further away, and right at the contact point.
• Shadow reach (m): how far from the character shadows are drawn.

Profiles

VaM doesn't save session plugin settings with scenes, so save them here. A profile holds everything, including your captured lights, the floor calibration and the room anchor, so you can keep one per room or per look.

• Top: the result of your last save or load (left) and an overview of all six profiles and which one is the startup profile (right).
• Six profile cards, each with Save settings to this profile, Load this profile and Startup profile. Only one profile can be the startup profile; it loads automatically when VaM starts.

Profiles are stored in VaM/Saves/PluginData/PassthroughKey. Profiles from v1.0 don't work with v2.0; save them again.


Advanced

Pass tuning (left). The defaults are right for almost everyone.

• Clear to black (cleaner edges): gives cleaner character edges. Leave on.
• Use depth (off = key everything): tells the character apart from the background. Off removes everything; only for testing.
• Edge erode (px): trims the outline of the character by a few pixels.
• Despill: removes colour fringes on the edges. Only matters with pure blue or green keys.
• Keep key pass last on camera: keeps the plugin working after other plugins add camera effects.

Diagnostics (right). For checking problems or reporting bugs.

• Debug view (white = kept): shows what is kept (white) and what is removed (black).
• Test pattern (colour chart): a colour chart floating in front of you, to check which colours Virtual Desktop cuts out. The bottom row is a Virtual Desktop check: all solid = VD is set up correctly; see-through = check VD Gamma 1.0 and the other VD image settings. Turn it off when done.
• Floor shadows: show raw shadow (magenta): shows the floor shadow in magenta.
• Fine hair: show hair pixels (magenta): shows the pixels kept as fine hair. If the whole background turns magenta, something in the scene tints the background (see Troubleshooting).
• Report UI layers to log, Log cameras, Log visible scene objects and Log lighting: write technical details to VaM's log (cameras, whatever still covers your room, the scene's lights and exposure), useful in a bug report.

Headset messages and sounds

During a capture or calibration, short messages float in front of you. They stay level, follow your view with a soft lag, and sit low in your view while you aim with the laser.

Colour	Meaning
Green	What to do now; a red line under it means something to fix (too close, point higher…)
Teal	A step worked: Got it, Done!, Floor set, Room anchor set, Room re-aligned
Red	Failed or cancelled; your previous result is kept

Sounds: a soft tick after each step, a chime when a capture or calibration is done, a low tone when it fails or is cancelled.


Troubleshooting and tips

Problem	What to do
The background doesn't disappear	Check every value in Virtual Desktop matches the plugin's Required Virtual Desktop settings box (R G B, Similarity, Smoothness, Sharpening, Gamma, Super Resolution). Status should say active and shader OK.
Status says "bundle not found" or "shader missing"	Reinstall the package and restart VaM.
Holes in the character or in the VaM menu	Raise Extra safety margin in steps of 0.05, or lower Similarity in both places.
Dark hair, lashes or clothes are see-through (the real room shows through them), bright skin stays solid	Set Virtual Desktop Gamma (Streaming tab) to 1.0. Below 1 VD darkens the image after the plugin and pushes darks into the key, worst with the black key. For a darker look use Character gamma (Light matching). Advanced → Test pattern: its bottom row shows whether VD is set up right.
The whole background is black in one scene	Make sure Hide skybox is on. Advanced → Fine hair: show hair pixels (magenta): if the whole background turns magenta, something in the scene tints it – report the scene.
Dark colours look washed out	Lower Extra safety margin a little and check nothing gets holes.
Part of the scene is still visible	Check that Hide environment is on and that the atom is ticked in the Hidden objects list (Scene handling). Things attached to a person stay. Advanced → Log visible scene objects lists what still covers the room.
Something you want to keep is gone, or a person is invisible	Scene handling: untick it in the Hidden objects list (people: untick to show). The choice is saved for this scene.
A capture selects a node or the menu pops up	Press B or Y to cancel and start the capture again from the plugin; VaM is switched to Play mode during captures.
The background flickers with effects on	Set Post effects to Suspend.
Settings gone after a restart	Save a profile and tick Startup profile on its card.
The character floats or sinks	Room & Shadows: Calibrate floor height, tick Align scene to real floor, adjust Scene height offset.
The character is in the wrong place after a headset sleep or a scene load	Room & Shadows: Re-align room (select the same two points, A first).
No floor shadows	Tick Floor shadows (Room & Shadows page), set VD Smoothness to 5, and Apply the lights.

Credits

PassthroughKey v2.0 by CraSh.it.

Thanks to Sally Whitemane, whose Camera Background Color plugin inspired the creation of PassthroughKey.

 

[Battletoad08]

I tried this, I don't see the plus side of it, I still had to use the green from the camerabackground color plugin colors. I dont see the benefits? The light options of putting lights where your lights are in your room but you can do that manually.
Am I missing something?

 

[CraSh.it]

I tried this, I don't see the plus side of it, I still had to use the green from the camerabackground color plugin colors. I dont see the benefits? The light options of putting lights where your lights are in your room but you can do that manually.
Am I missing something?

Hi, thanks for trying it and for your feedback!
The main idea is that you don't need green any more. Leave the key on Black (the default) and copy the R G B and Similarity from the plugin's Virtual Desktop box into VD (Smoothness 0).
Normally black eats dark hair, shadows and clothes, and the VaM menu goes see-through. PassthroughKey protects all of that, so you get a black key with no holes and no green fringing on skin and hair.
The other bits are about not having to fiddle per scene: the room is hidden automatically in any scene (people stay), effects like bloom or PostMagic don't break the key, and settings live in profiles.
Lights: yes, you can place them by hand. Capture just does it in a few seconds by looking at your window or lamp, and the lights stay locked to your real room when you move around in VaM, which manual lights don't.
If black didn't key cleanly for you, tell me what you saw and I'll help sort it out!

 

[Battletoad08]

Hi, thanks for trying it and for your feedback!
The main idea is that you don't need green any more. Leave the key on Black (the default) and copy the R G B and Similarity from the plugin's Virtual Desktop box into VD (Smoothness 0).
Normally black eats dark hair, shadows and clothes, and the VaM menu goes see-through. PassthroughKey protects all of that, so you get a black key with no holes and no green fringing on skin and hair.
The other bits are about not having to fiddle per scene: the room is hidden automatically in any scene (people stay), effects like bloom or PostMagic don't break the key, and settings live in profiles.
Lights: yes, you can place them by hand. Capture just does it in a few seconds by looking at your window or lamp, and the lights stay locked to your real room when you move around in VaM, which manual lights don't.
If black didn't key cleanly for you, tell me what you saw and I'll help sort it out!

I tried the black, it cut out every part of the hair and some of the eyes and stuff. I tried some of the other colors in your drop down but all had a bright outline around the model. I adjusted the VD settings to match on every color. These were black scenes with just 1 girl and 1 plain light.

 

[CraSh.it]

I tried the black, it cut out every part of the hair and some of the eyes and stuff. I tried some of the other colors in your drop down but all had a bright outline around the model. I adjusted the VD settings to match on every color. These were black scenes with just 1 girl and 1 plain light.

That's not right. I'm pretty sure your settings are mismatched.
Can you share a screenshot of your VD passthrough config?

 

[cnorton76]

Great plugin! Works better than Sally's version and really makes the model "pop out" in MR/AR scenes. Great work!

The body is almost perfect, with very minimal haloing. The hair has no haloing, however, some of the fine strands are lost and the mask seems to be very aggressive. Being able to tone down the hair mask or better yet if we could separately control the body and hair mask and be able to fine tune them individually would make this plugin nearly perfect.

 

[CraSh.it]

Great plugin! Works better than Sally's version and really makes the model "pop out" in MR/AR scenes. Great work!

The body is almost perfect, with very minimal haloing. The hair has no haloing, however, some of the fine strands are lost and the mask seems to be very aggressive. Being able to tone down the hair mask or better yet if we could separately control the body and hair mask and be able to fine tune them individually would make this plugin nearly perfect

Thanks, great pointer!
I wouldn't even notice as my test model has a straight flat hairstyle.
The hair mask in v1.0 is indeed very aggressive.
I've been working on some major improvements over the last few days, and the Fine Strand Fix is already in. It looks so much better now.
On top of all that, there's also a massive perk coming your way soon.
Stay tuned!

 

[Battletoad08]

That's not right. I'm pretty sure your settings are mismatched.
Can you share a screenshot of your VD passthrough config?

Sorry it took so long. I use the black setting and the hair and eyes like pupils are see through. Edges of skin look great but anything dark/black and transparent.

 

[CraSh.it]

Sorry it took so long. I use the black setting and the hair and eyes like pupils are see through. Edges of skin look great but anything dark/black and transparent.

View attachment 638669

Can you try to reset all settings in the plugin? (Bottom of the main page) hit it twice.
Also does the VaM top menu look see-through in dark areas?

 

[CraSh.it]

Can you try to reset all settings in the plugin? (Bottom of the main page) hit it twice.
Also does the VaM top menu look see-through in dark areas?

Also are there any other plugins running alongside PassthroughKey?

 

[Battletoad08]

Can you try to reset all settings in the plugin? (Bottom of the main page) hit it twice.
Also does the VaM top menu look see-through in dark areas?

I dont have any session plugins that have to do with lighting or chrome keys or background camera or anything. I clicked reset twice and this is what I see.

 

[CraSh.it]

I dont have any session plugins that have to do with lighting or chrome keys or background camera or anything. I clicked reset twice and this

I dont have any session plugins that have to do with lighting or chrome keys or background camera or anything. I clicked reset twice and this is what I see.

View attachment 638965

That leave two possible culprits.

1. What does the plugin's status line say? It should be "active | shader OK".
2. Advanced → tick "Debug view (white = kept)". Is the character white?
3. In VD, turn off Sharpening and Snapdragon Super Resolution, and reset any colour adjustments (brightness/contrast/gamma/saturation). They change the dark colours before VD keys them.

That should pin it down

 

[Battletoad08]

That leave two possible culprits.

1. What does the plugin's status line say? It should be "active | shader OK".
2. Advanced → tick "Debug view (white = kept)". Is the character white?
3. In VD, turn off Sharpening and Snapdragon Super Resolution, and reset any colour adjustments (brightness/contrast/gamma/saturation). They change the dark colours before VD keys them.

That should pin it down

still didnt work

 

[CraSh.it]

Found it!
Your Gamma slider in Virtual Desktop is at 0.71. Set it to the default 1 and you're good to go

 

[Battletoad08]

Found it!
Your Gamma slider in Virtual Desktop is at 0.71. Set it to the default 1 and you're good to go

The gama fixed it. Kinda sucks tho, it washes out the game some at 1. I cant find any ingame brightness setting.

Thanks

 

[CraSh.it]

The gama fixed it. Kinda sucks tho, it washes out the game some at 1. I cant find any ingame brightness setting.

Thanks

No worries, I will add plugin gamma compensation to the next update.

 

[CraSh.it]

Hey gang
Here's a quick look at what's been happening since v1.0.
It’s not out yet - still testing and polishing, but here’s what’s coming:

• Real floor shadows. Characters will cast soft, natural shadows on your actual floor.
• Easier light matching. Pick what lights your room (ceiling light, sunny or cloudy window, lamp or TV), point at it, done. Each light type gets its own look.
• Remembers your lights. Capture each light once, and switching presets puts it right back where it belongs.
• Room anchor. Touch two fixed spots in your room. After the headset sleeps or you load a new scene, touch them again, and your character is back exactly where you left them.
• Stays on your floor. Body, hair and clothes will rest on your real floor instead of sinking through it.
• Fuller hair, cleaner edges. Fine strands that used to vanish are coming back, without the dark outline.
• Character gamma. A darker overall look with one slider, and the key stays clean.
• Built-in setup check. The test pattern will tell you at a glance if a Virtual Desktop setting is off.
• Six profiles, with one that loads at startup.
• Cleaner menus. Every page redesigned to full-fit (no scrolling).

Big thanks to everybody who gave feedback. Your reports kicked off some of these fixes and improvements. Keep it coming!

Releasing soon.
Stay tuned!

 

[CraSh.it]

CraSh.it updated PassthroughKey with a new update entry:

PassthroughKey 2.0

PassthroughKey 2.0
Your VaM characters in your real room

PassthroughKey puts your VaM characters into your real room using Virtual Desktop's passthrough on a Quest. It keeps the background exactly the colour Virtual Desktop removes, keeps the character and the VaM menu solid, and hides the scene around them.

What's new in 2.0

• Real floor shadows – characters cast soft shadows on your actual floor, dark...

Read the rest of this update entry...

 

[Meharock]

Hi! That's crazy how we started creating mostly the same thing at the same time and even released the first version the same day XD Good job with the VD! I should check your light capture, is it that good? Does character feel pasted well itno the room with the basic light capture?

 

[CraSh.it]

Hey

Hi! That's crazy how we started creating mostly the same thing at the same time and even released the first version the same day XD Good job with the VD! I should check your light capture, is it that good? Does character feel pasted well itno the room with the basic light capture?

Great minds think alike
Yeah, that evening our plugins got released right next to each other; I was like, WTF? Is this for real?
Then I realised we both have an original approach and that all the hours spent on this project were not completely wasted.
Well done to you too. Your UI looks well polished. I can see the passion there.
The native passthrough gives you so much more control from the developer's perspective, although it is locked to Meta users only.
That said, I'm sure we can share ideas to push this as far as it goes.
My next focus is the walls and flat surfaces for shadow reception and character interactions (got inspired by this other plugin I saw earlier ).

"Does the character feel pasted well into the room?"
Yes and no. The Quest's passthrough cameras are not exactly retinal resolution, so some kind of character "blend" could help the illusion.
It is on my list, but I didn't get the chance yet.
The lighting is tricky, and it's mostly down to the user to match their own room conditions.
I think the plugin provides enough sliders to do that just fine.
Give it a shot, let me know

 

[TheHookQC]

Is it compatible with dlss5 + Neural Rendering plugins? Bacause I can't see throught. Objects disappear correctly but background stays solid black. Debug sees colors as muted for VD. In VD passthrough values are ok (RGB to ) and similarity and smootness to 5%) ,gamma is set to 1.0 and supersampling is disabled as well as sharppening.

 

[CraSh.it]

Is it compatible with dlss5 + Neural Rendering plugins? Bacause I can't see throught. Objects disappear correctly but background stays solid black. Debug sees colors as muted for VD. In VD passthrough values are ok (RGB to ) and similarity and smootness to 5%) ,gamma is set to 1.0 and supersampling is disabled as well as sharppening.

Not yet. The DLSS Neural Rendering plugin adds its own processing step after PassthroughKey’s final pass.
It reworks every pixel, including the background, so the background is no longer the exact key colour and Virtual Desktop doesn’t cut it out. That’s why you get solid black.

Making the two work together is on my list for a future update. Until then:

1. Turn off the Neural Rendering / denoise step in the DLSS plugin, or the whole DLSS plugin, while you use passthrough.
2. Then switch PassthroughKey’s Key pass active off and on again, so it’s back at the end of the camera’s chain.
3. Check it with Advanced → Test pattern: if the bottom row is see-through, the background now keys.

If it’s still black, run Advanced → Log cameras and send me the log lines. They show what runs after PassthroughKey.

 

[TheHookQC]

Not yet. The DLSS Neural Rendering plugin adds its own processing step after PassthroughKey’s final pass.
It reworks every pixel, including the background, so the background is no longer the exact key colour and Virtual Desktop doesn’t cut it out. That’s why you get solid black.

Making the two work together is on my list for a future update. Until then:

1. Turn off the Neural Rendering / denoise step in the DLSS plugin, or the whole DLSS plugin, while you use passthrough.
2. Then switch PassthroughKey’s Key pass active off and on again, so it’s back at the end of the camera’s chain.
3. Check it with Advanced → Test pattern: if the bottom row is see-through, the background now keys.

If it’s still black, run Advanced → Log cameras and send me the log lines. They show what runs after PassthroughKey.

here's the camera log:

PassthroughKey: cameras (name / depth / clear / UI layer / stereo):
'ScreenCameraClear' depth -1 clear Color UI False eye None
'CenterEyeAnchor' depth 0 clear Color UI True eye Both <- protect pass
'MonitorRig' depth 2 clear Color UI True eye None
'ScreenUICamera' depth 10 clear Nothing UI True eye None
'PassthroughKey_UnderFloorCam' depth -101 clear Color UI False eye None (renders to texture)
'PassthroughKey_ShadowCam' depth -100 clear Color UI False eye None (renders to texture)
(disabled/manual) 'ClearUpscale_CharCamera' depth 2 clear Color UI False eye Both
(disabled/manual) 'ClearUpscale_BgCamera' depth 1 clear Color UI True eye Both
(disabled/manual) 'RightHandAnchor' depth 0 clear Nothing UI False eye None
(disabled/manual) 'LeftHandAnchor' depth 0 clear Nothing UI False eye None
(disabled/manual) 'RightEyeAnchor' depth 0 clear Color UI True eye Right
(disabled/manual) 'LeftEyeAnchor' depth 0 clear Color UI True eye Left
(disabled/manual) 'HiResScreenshotCamera' depth 0 clear Color UI True eye None
(disabled/manual) 'LoResScreenshotCamera' depth 0 clear Color UI True eye None
VR: True key pass: active
PassthroughKey: components on 'CenterEyeAnchor' (render order, hdr False, colour space Linear):
0. UnityEngine.Transform
1. UnityEngine.AudioListener (disabled)
2. UnityEngine.Camera
3. MeshVR.PerfMonCamera
4. MK.Glow.MKGlow (disabled)
5. Leap.Unity.LeapXRServiceProvider (disabled)
6. LeapMotionProviderSwitcher
7. PassthroughKeyOpaquePass
8. PassthroughKeyPass
9. VamDlssNr.NrCapture
PassthroughKey: RIG CHAIN (head camera up to the root; local pos / world y):
0. 'CenterEyeAnchor' local (-0.025, 1.739, -0.060) world y 1.540 scale 1
1. 'TrackingSpace' local (0.000, 0.000, 0.000) world y -0.199 scale 1
2. 'OVRRig' local (0.000, 0.000, 0.000) world y -0.199 scale 1
3. 'HeightOffset' local (0.000, -0.199, 0.000) world y -0.199 scale 1
4. '[CameraRig]' local (0.054, 0.000, 0.803) world y 0.000 scale 1
5. 'NavRig' local (0.000, 0.000, 0.000) world y 0.000 scale 1
6. 'WorldScaleAdjust' local (0.000, 0.000, 0.000) world y 0.000 scale 1
7. 'CoreControl' local (0.000, 0.000, 1.500) world y 0.000 scale 1
8. 'SceneAtoms' local (0.000, 0.000, 0.000) world y 0.000 scale 1
navigationRig '[CameraRig]' world y 0.000
PassthroughKey: TRACKED under 'HeightOffset' (world y / height above play-space origin):
'HUDAnchors' world y -0.199 above origin 0.000
'HUDAnchorDown' world y 0.681 above origin 0.880
'HUDAttachPoint' world y 1.186 above origin 1.384
'AnchorParent' world y 1.046 above origin 1.245
'HUDAnchorWorldUI' world y 0.601 above origin 0.800
'OVRRig' world y -0.199 above origin 0.000
'TrackingSpace' world y -0.199 above origin 0.000
'LeftEyeAnchor' world y 1.540 above origin 1.739
'CenterEyeAnchor' world y 1.540 above origin 1.739
'RightEyeAnchor' world y 1.540 above origin 1.739
'TrackerAnchor' world y -0.199 above origin 0.000
'RightHandAnchor' world y 1.277 above origin 1.476
'Hands' world y -0.199 above origin 0.000
'RightHandPhysical' world y -0.199 above origin 0.000
'MaleHand02R' world y -0.199 above origin 0.000
'MotionControlAlternateHands' world y -0.199 above origin 0.000
'MonitorRig' world y 1.401 above origin 1.600
'MonitorHUDAnchor' world y 0.037 above origin 0.236
'MonitorHUDAttach' world y 0.417 above origin 0.615
'CenterEye' world y 1.401 above origin 1.600
'HUDReferenceDown' world y 0.681 above origin 0.880
'Capsule' world y 1.413 above origin 1.612
'Sphere1' world y 1.403 above origin 1.601
'Sphere2' world y 1.403 above origin 1.601
'HeadPossessPoint' world y 1.401 above origin 1.600
'AltHUDAttachPoint' world y 1.331 above origin 1.530
'ThoughtBubble' world y 1.050 above origin 1.249
Scene height offset 0.000, floor match False

It seem NR still running after disabling in the plugin if I'm not mistaken

 

[CraSh.it]

here's the camera log:

PassthroughKey: cameras (name / depth / clear / UI layer / stereo):
'ScreenCameraClear' depth -1 clear Color UI False eye None
'CenterEyeAnchor' depth 0 clear Color UI True eye Both <- protect pass
'MonitorRig' depth 2 clear Color UI True eye None
'ScreenUICamera' depth 10 clear Nothing UI True eye None
'PassthroughKey_UnderFloorCam' depth -101 clear Color UI False eye None (renders to texture)
'PassthroughKey_ShadowCam' depth -100 clear Color UI False eye None (renders to texture)
(disabled/manual) 'ClearUpscale_CharCamera' depth 2 clear Color UI False eye Both
(disabled/manual) 'ClearUpscale_BgCamera' depth 1 clear Color UI True eye Both
(disabled/manual) 'RightHandAnchor' depth 0 clear Nothing UI False eye None
(disabled/manual) 'LeftHandAnchor' depth 0 clear Nothing UI False eye None
(disabled/manual) 'RightEyeAnchor' depth 0 clear Color UI True eye Right
(disabled/manual) 'LeftEyeAnchor' depth 0 clear Color UI True eye Left
(disabled/manual) 'HiResScreenshotCamera' depth 0 clear Color UI True eye None
(disabled/manual) 'LoResScreenshotCamera' depth 0 clear Color UI True eye None
VR: True key pass: active
PassthroughKey: components on 'CenterEyeAnchor' (render order, hdr False, colour space Linear):
0. UnityEngine.Transform
1. UnityEngine.AudioListener (disabled)
2. UnityEngine.Camera
3. MeshVR.PerfMonCamera
4. MK.Glow.MKGlow (disabled)
5. Leap.Unity.LeapXRServiceProvider (disabled)
6. LeapMotionProviderSwitcher
7. PassthroughKeyOpaquePass
8. PassthroughKeyPass
9. VamDlssNr.NrCapture
PassthroughKey: RIG CHAIN (head camera up to the root; local pos / world y):
0. 'CenterEyeAnchor' local (-0.025, 1.739, -0.060) world y 1.540 scale 1
1. 'TrackingSpace' local (0.000, 0.000, 0.000) world y -0.199 scale 1
2. 'OVRRig' local (0.000, 0.000, 0.000) world y -0.199 scale 1
3. 'HeightOffset' local (0.000, -0.199, 0.000) world y -0.199 scale 1
4. '[CameraRig]' local (0.054, 0.000, 0.803) world y 0.000 scale 1
5. 'NavRig' local (0.000, 0.000, 0.000) world y 0.000 scale 1
6. 'WorldScaleAdjust' local (0.000, 0.000, 0.000) world y 0.000 scale 1
7. 'CoreControl' local (0.000, 0.000, 1.500) world y 0.000 scale 1
8. 'SceneAtoms' local (0.000, 0.000, 0.000) world y 0.000 scale 1
navigationRig '[CameraRig]' world y 0.000
PassthroughKey: TRACKED under 'HeightOffset' (world y / height above play-space origin):
'HUDAnchors' world y -0.199 above origin 0.000
'HUDAnchorDown' world y 0.681 above origin 0.880
'HUDAttachPoint' world y 1.186 above origin 1.384
'AnchorParent' world y 1.046 above origin 1.245
'HUDAnchorWorldUI' world y 0.601 above origin 0.800
'OVRRig' world y -0.199 above origin 0.000
'TrackingSpace' world y -0.199 above origin 0.000
'LeftEyeAnchor' world y 1.540 above origin 1.739
'CenterEyeAnchor' world y 1.540 above origin 1.739
'RightEyeAnchor' world y 1.540 above origin 1.739
'TrackerAnchor' world y -0.199 above origin 0.000
'RightHandAnchor' world y 1.277 above origin 1.476
'Hands' world y -0.199 above origin 0.000
'RightHandPhysical' world y -0.199 above origin 0.000
'MaleHand02R' world y -0.199 above origin 0.000
'MotionControlAlternateHands' world y -0.199 above origin 0.000
'MonitorRig' world y 1.401 above origin 1.600
'MonitorHUDAnchor' world y 0.037 above origin 0.236
'MonitorHUDAttach' world y 0.417 above origin 0.615
'CenterEye' world y 1.401 above origin 1.600
'HUDReferenceDown' world y 0.681 above origin 0.880
'Capsule' world y 1.413 above origin 1.612
'Sphere1' world y 1.403 above origin 1.601
'Sphere2' world y 1.403 above origin 1.601
'HeadPossessPoint' world y 1.401 above origin 1.600
'AltHUDAttachPoint' world y 1.331 above origin 1.530
'ThoughtBubble' world y 1.050 above origin 1.249
Scene height offset 0.000, floor match False

It seem NR still running after disabling in the plugin if I'm not mistaken

Thanks, that log helps. Two things are still changing the image after PassthroughKey:

• VamDlssNr (NR) is still active on the VR camera and runs after PassthroughKey’s last pass. Turning it off in its menu apparently doesn’t remove it. Disable the plugin itself and restart VaM.
• ClearUpscale renders the character and background with its own cameras and builds the final image, which replaces the key background. Turn that off too.

With both off, the background should key.
Advanced > Test pattern: the bottom row should be see-through.

Making PassthroughKey work together with DLSS / NR upscalers is on my list.