Other ALXR Quest/Pico client v0.12.1 for server v18.5.0 with modified passthrough mask

[animetiddyenthusiast]

ah yeah that sounds about right

 

[insistent]

Maybe the new pico OS update is causing the crash? i Believe since the update most of my other homebrew app doesnt work anymore.

 

[atani]

I just tried the new version. The client works great but passthrough modes still show issues, unless I'm doing something wrong. The behaviour is the same whether I'm in VaM or just Steam VR.
Using left menu + B doesn't show any passthrough, just artifacts.
Using left menu + A works, it gives artifacts but they're either fewer or easier to miss.

I don't think I have any wildly off settings in the ALVR server, the problems only come with passthrough activated. I'll try to install a older version to compare.

Spoiler: Artifacts

I mean short lines with colours that pop in and out at random. These do not register in a screenshot and only occur when I activate any of the passthrough modes.

 

[atani]

Updating my answer regarding the latest version for the Pico 4.

Full passthrough does work. I was using the older recommendation of 220, 220, 240 and the spectrum on Sally's plugin wasn't showing any passthrough area to see my mistake. After reading your updated description to use 100, 129, 100, it showed a nice big passthroughed area in the spectrum and in VaM.
It's amazing what you did with the outline, it's barely visible ? , incredible work. This bring immersion to a whole new level.

Unfortunately the artifacts are there and there's high latency effects when using the menu or Scene Browser. There's a higher latency when in passthrough mode, seeing in the ALVR server graphs, but it's something like a 25% jump, not something dramatic compared to non-passthrough. I also heard the headset fan starting earlier than usual and battery use seemed higher than normal, perhaps there's something there making the headset struggle a lot more than in non-passthrough besides the use of the front camera?

Is there something I should see and tell you that would be meaningful for debugging? How's the result in your Quest, does it also have the artifacts I mention?

 

[Seraphim]

Updating my answer regarding the latest version for the Pico 4.

Full passthrough does work. I was using the older recommendation of 220, 220, 240 and the spectrum on Sally's plugin wasn't showing any passthrough area to see my mistake. After reading your updated description to use 100, 129, 100, it showed a nice big passthroughed area in the spectrum and in VaM.
It's amazing what you did with the outline, it's barely visible ? , incredible work. This bring immersion to a whole new level.

Unfortunately the artifacts are there and there's high latency effects when using the menu or Scene Browser. There's a higher latency when in passthrough mode, seeing in the ALVR server graphs, but it's something like a 25% jump, not something dramatic compared to non-passthrough. I also heard the headset fan starting earlier than usual and battery use seemed higher than normal, perhaps there's something there making the headset struggle a lot more than in non-passthrough besides the use of the front camera?

Is there something I should see and tell you that would be meaningful for debugging? How's the result in your Quest, does it also have the artifacts I mention?

What are your Video/Connection settings in alvr desktop? I dont think I see any artifacts except the thin jagged outer line, but am still getting random app shutdowns/disconnects. Us having the same headset, i would think it was the desktop settings, i dont know.

 

[atani]

What are your Video/Connection settings in alvr desktop? I dont think I see any artifacts except the thin jagged outer line, but am still getting random app shutdowns/disconnects. Us having the same headset, i would think it was the desktop settings, i dont know.

There's tons of options in the settings, especially in the advanced settings. I think almost all are in the default option, and considering there's no artifacts until I do the key combination to use passthrough I don't think the settings would be the cause?
I can also uninstall everything and start fresh. Are your settings in the server changed somewhere special?
I didn't experience shutdowns yet, so that's good.

 

[Seraphim]

There's tons of options in the settings, especially in the advanced settings. I think almost all are in the default option, and considering there's no artifacts until I do the key combination to use passthrough I don't think the settings would be the cause?
I can also uninstall everything and start fresh. Are your settings in the server changed somewhere special?
I didn't experience shutdowns yet, so that's good.

Weird. It seems to just have to do with whenever im trying to interact with the headset controls (adjusting volume, taking screenshots, etc) It will boot out of the app a few seconds after. Try that on yours. (Im on latest pico update 5.7.2 btw)

 

[animetiddyenthusiast]

There's tons of options in the settings, especially in the advanced settings. I think almost all are in the default option, and considering there's no artifacts until I do the key combination to use passthrough I don't think the settings would be the cause?
I can also uninstall everything and start fresh. Are your settings in the server changed somewhere special?
I didn't experience shutdowns yet, so that's good.

hmm I do get the artifacts even without passthrough on if my settings are too high but passthrough layers are pretty taxing on the system. They have identical specs but pico is higher res at 2160x2160 vs 1832x1920 per eye plus pico uses high res color camera for passthrough while quest just shows us the tracking camera feed. Passthrough pushes my GPU hard already so a machine with the same specs but higher res should struggle more. I've been setting stuff up to get performance metrics on my quest and will be trying to optimize stuff for everyone. I do think my latest shaders actually did reduce the occasional stutters I was getting from high GPU usage but I need to actually test and compare.

As for ALVR settings you definitely don't want default. I wish I had a writeup on here about server settings but I'm still not happy with mine. Default is bad though, very bad. I recommend lowering the video base resolution to reduce the decoder load on your headset. The other settings depend more on your router setup (wired is also possible, guide on alvr github) and PC

NGL these settings arent good so dont just copy them. I'm not happy with them but I don't think I've ever been happy with ALVR quality. Some important guidelines, use CBR not VBR, never use a maximum bitrate above 150 for HEVC or 250 for H264. You will get extreme lag in those scenarios.

Havent tested but I noticed this was checked and it seems like something we want OFF. We are getting rid of the background, we dont want solid colors to be the focus. But it could make things worse. I'll test it out.

I use steamvr tracking as oculus version is extremely jittery. tbh the controls are the worst part and I cannot get them right in ALVR, but I can at least say steamvr is much more consistent than oculus tracking

For connections settings I recommend TCP. I'm not sure what effect changing the bufferbytes would have so I havent touched it yet.

glitches and artifacts can occur due to packet loss. try this option if you want, but it could also make things worse

 

[animetiddyenthusiast]

Weird. It seems to just have to do with whenever im trying to interact with the headset controls (adjusting volume, taking screenshots, etc) It will boot out of the app a few seconds after. Try that on yours. (Im on latest pico update 5.7.2 btw)

hmm it must be some interaction with Pico OS. I'll see if I can find any info about that. I have noticed that I sometimes get a black screen and warning that ALXR is not responding, but I just hit okay and it lets me back into the app no issue. Only noticed it since I started this branch but I dont think I caused it unless its just bc I'm pushing the system harder. Anyway the shaders are decent enough that I'll focus on optimization stuff for a bit instead of just tweaking the shaders endlessly

Actually now that I think about I'm pretty sure I saw some code indicating android devices are forced into high perfromance mode when passthrough is enabled in ALXR. It might be better to disable that and allow the headset to manage itself.

Unfortunately the artifacts are there and there's high latency effects when using the menu or Scene Browser. There's a higher latency when in passthrough mode, seeing in the ALVR server graphs, but it's something like a 25% jump, not something dramatic compared to non-passthrough. I also heard the headset fan starting earlier than usual and battery use seemed higher than normal, perhaps there's something there making the headset struggle a lot more than in non-passthrough besides the use of the front camera?

Partially addressed this already with the server settings, but consider that any extra latency on your headset will increase latency in sending your tracked poses to the server and any latency from your PC increases the delay before you get to act as well. If you add VAM lag on top of that it becomes very bad, very very bad. Try to play on a fairly clean install and do hard resets every couple scenes to reduce the number of assets loaded in. I also highly recommend the bepinEx varbrowser plugin that loads in stuff only when you need it. It can still become an issue after a while bc it doesnt really unload, but it makes startup much better

Honestly I have my hands a bit full but I'd really like to write a patch to alter the asset loading both for general optimization and to have a way to only load relevant atoms and ignore all the CUAs and meshes we don't want in XR bc my current method of loading a scene and then deleting stuff with UIA or manually is inefficient and still hurts performance. I know there are some methods to directly load animations with timeline but idk how to sync stuff up or if thats the best option since you have to export stuff first. Anyone know much about that stuff?

 

[atani]

Weird. It seems to just have to do with whenever im trying to interact with the headset controls (adjusting volume, taking screenshots, etc) It will boot out of the app a few seconds after. Try that on yours. (Im on latest pico update 5.7.2 btw)

Right, I forgot I did try to use the volume buttons and the client crashed. Yes, I confirm the same behaviour. I rarely do that so forgot about it.

@animetiddyenthusiast
I kept my ALVR settings close to default assuming they were reasonable settings, guess I'll have to play with them even not understanding what most do. The extra effort on the Pico makes sense, hopefully addressing the server settings will improve the results.
But would still like to say that this is currently the best experience ever with passthrough, I'm quite excited to see what future improvements will do here, but damn, it's amazing what is available already in 2023.

 

[animetiddyenthusiast]

Right, I forgot I did try to use the volume buttons and the client crashed. Yes, I confirm the same behaviour. I rarely do that so forgot about it.

@animetiddyenthusiast
I kept my ALVR settings close to default assuming they were reasonable settings, guess I'll have to play with them even not understanding what most do. The extra effort on the Pico makes sense, hopefully addressing the server settings will improve the results.
But would still like to say that this is currently the best experience ever with passthrough, I'm quite excited to see what future improvements will do here, but damn, it's amazing what is available already in 2023.

default settings are horrendous holy crap dude. I still dont know how to fix the lag, tbh, but if you look at my settings as a reference point you can at least make it look better

 

[Cumflation_Magician]

Pico 4 version now works great! Thanks.
I am however retarded so I will describe some of the process for other troubled souls:
- first of all if you miss something refer to the overview and follow the additional guide linked inside (also here )
- Remove all environments and floors from a scene (usual process: edit mode, click arrow, find the relative assets and delete)
- Download this plugin to set the scene color background to R 100, G 120, B 100 (the values in the original description are wrong and I kept using them because, again, I'm retarded)
- Press the menu button on the left controller (the one with three lines) + B on right controller (I kept pressing menu + A which gives a half transparency mode)

 

[animetiddyenthusiast]

- Remove all environments and floors from a scene (usual process: edit mode, click arrow, find the relative assets and delete)
- Download this plugin to set the scene color background to R 100, G 120, B 100 (the values in the original description are wrong and I kept using them because, again, I'm retarded)
- Press the menu button on the left controller (the one with three lines) + B on right controller (I kept pressing menu + A which gives a half transparency mode)

The color doesnt have to be exact btw you can use the vertical HUE scrollbar and pick anywhere within green once you have it lined up in the right area. I find slightly yellow looks best in many scenes. And the transparenty mode is what I refer to as blend mode just FYI


UI Assist can remove all non-person atoms for quick cleanup and then you can have a button for spawning in lights/loading a subscene. Just don't use it in any scene with scripts attached to non-person atoms. Also be aware of cases where the animations are based on collisions. The safest way to prep a scene is to individually select atoms, go to materials/shader and hide the material that way you don't mess with collision. This is super tedious though so I was planning to write a script to automate that at some point just been busy and I know how to code in unity but never written any scripts for VAM. There is a more generalizable way to do it in any unity game by adjusting layers so I'm probably gonna investigate that idea first.

If anyone checking this thread happens to be good with VAM scripting and wants to help do a specific one first that would be cool too. Should be pretty simple. Basically something like this

public class cleanScene : MonoBehaviour //or whatever vam scripts inherit from
{
private List<GameObject> sceneObjects = new List<GameObject>();

public void Init()
{
    //make a button with VAM API for session plugin or just have it activate immediately upon adding to scene plugins
    this.sceneObjects = SceneManager.GetActiveScene().GetRootGameObjects();
    //I'm guessing there's a way to do this more directly in VAM
    cleanObjects();
    
}

public void cleanObjects()
{
    //generalizable and nondestructive method but would require setting up a hidden layer in another class if VAM doesnt have one to use
    int LayerHidden = LayerMask.NameToLayer("Hidden Layer");
    foreach (GameObject go in this.sceneObjects) {   
        if (!gameObject.TryGetComponent(Atom.Type("Person") ) ) //Random shot in the dark, probably a better way to check this in vam
            go.layer = LayerHidden; //if using layer method
        //otherwise
            foreach(Renderer re in go.GetComponentsInChildren<Renderer>()
                    {
                        r.enabled = false;
                        //sometimes it may be better to swap the shader or material out for a new one with alpha of 0 or change the render queue but this generally works
                    }
    }
}
    
    
}

something along those lines. feel free to finish if you're able to, otherwise I'll do it at some point.

 

[animetiddyenthusiast]

Ok haven't tested yet but this seems to delete walls, floors, and I think CUAs so it should be perfect for us. I'll test it and link on main page if it's good.

Didn't do anything the past couple days until just now I made a really good blend mode that can leave a very slight translucent colored background or use straight black passthrough without much issue with the shadows. Only catch is it was supposed to be the color mask mode but I somehow made it into additive blend ?

will post more builds later. I basically reformatted my code in a way that we avoid having to ask if/else questions and instead do some arithmetic. Takes me a bit longer to think about how to adjust things a certain way but it seems a little better with the artifacting. I also tried lowering my stream and game res to 0.75-0.80 and actually boosting my bitrate and that's been giving me better stream quality. I think native quest 2 res is just too high to efficiently compress even if the actual total amount of data averages out around the same

 

[bd-energy]

Tired to read/respond much but last night I discovered something in the client that was having an effect on the visual quality, don't yet know how to fix it fully without a new issue popping up but i hope I can figure it out soon. In other news, passthrough modes in latest confirmed working on Lynx R-1 (everything works just one big issue that's making it unusable I need to figure out without the device...)

 

[animetiddyenthusiast]

Tired to read/respond much but last night I discovered something in the client that was having an effect on the visual quality, don't yet know how to fix it fully without a new issue popping up but i hope I can figure it out soon. In other news, passthrough modes in latest confirmed working on Lynx R-1 (everything works just one big issue that's making it unusable I need to figure out without the device...)

Oh I just finished updating my fork to those commits. Well at least the quest version still works. What's the visual issue? I've been a couple commits behind but didn't see one in testing just now.

In other news my shaders have gotten much faster and more accurate. Still using delta E but I've optimized the conversion and have the key color start in LAB. I researched conditionals and branching pretty thoroughly so I honestly thought previous versions were fine on my end and it was just normal for my quest 2 to be over 90% GPU usage. Turns out I was super wrong and after optimizing I'm now hitting 50-70% gpu usage at a quick glance and everything feels much smootber.
To accomplish this I refactored all my ifs and conditionals into pure arithmetic evaluations using these helper functions.


vec4 when_gt(vec4 x, vec4 y) { return max(sign(x - y), 0.0); } vec4 when_lt(vec4 x, vec4 y) { return max(sign(y - x), 0.0); }

eg instead of


If (deltaE < 12) color.a = 0
Or
color.a = deltaE < 12 ? 0 : 1

I'd now write

color.a -= when_lt(deltaE, 12)

or
color.a *= when_gt(deltaE, 12)

First one evaluates to 1 - 1 if deltaE is in threshold and 1 - 0 otherwise without branching. Second one we just flip the signs so we can multiply by 0 to get the same results. We can up the complexity for little to no performance cost by evaluating multiple conditions, multiplying them by a factor, and adding them together like so

(x*when_lt(dE, threshold)) + (y*when_gt(dE, threshold2))

This example is kind of nonsense but allows for a quick evaluation for something that may require a branch otherwise

Currently I have a mask mode that offers perfect color removal with no noticeable outline when set up properly with a background color changer plugin. Once you've set the general position on the 2d color map you can actually just scroll across the entire range of hues and a matching key, though the hues farthest from green may have a slightly translucent background. It's not exactly full choice of key color but I think it's a good compromise while still utilizing colors that dotn otherwise pose a problem in the scene. My blend mode is actually using the same techniques as my mask mode but with more focus on luma and it still uses additive blend. I think it's a small improvement to the previous blend mode and a fine compromise if you can't set a background color

Updated builds coming soon and I'll probably publish on GitHub as well. Also anyone using this in vam needs this plugin NOW https://hub.virtamate.com/resources/scene-atomic-batch-operation-tool-lgatomtool.36603/

 

[animetiddyenthusiast]

animetiddyenthusiast updated ALXR Quest/Pico client v0.12.1 for server v18.5.0 with modified passthrough mask with a new update entry:

New plugin available on the hub that automates AR scene edits

Get this plugin right now: https://hub.virtamate.com/resources/scene-atomic-batch-operation-tool-lgatomtool.36603/

Seriously it's so good! tysm for making it if you see this! No other tool for the job is this feature complete so I was gonna make a shit tier version of this myself but saves me the work

Read the rest of this update entry...

 

[geevee]

I'm trying not to be mean but my dude read the names of the files

I read the names of the files. Let me try again:

Does alxr-client.apk support all android devices including Quest 2? If this doesn't support Quest or Pico then what DOES it support?
Does alxr-client-quest.apk include support for Quest 2 (since there are also Quest 1, Quest Pro, and Quest 3)?
If both apk's support Quest 2 which is better?

Thanks for any info.

 

[animetiddyenthusiast]

I read the names of the files. Let me try again:

Does alxr-client.apk support all android devices including Quest 2? If this doesn't support Quest or Pico then what DOES it support?
Does alxr-client-quest.apk include support for Quest 2 (since there are also Quest 1, Quest Pro, and Quest 3)?
If both apk's support Quest 2 which is better?

Thanks for any info.

Quest version will work on any version of the quest. Tbh it should boot on every android device on os v12 or newer, the main reason for different platform versions is to handle separate controls and methods of involving passthrough. I did include a separate quest pro build in one upload to hopefully better cater to their color passthrough but even then it worked on any quest.


I do apologize for the confusing download system. This site is very limiting since I can only post one archive at a time


Sorry I know the download system is confusing

 

[Seraphim]

Doggy had to bob-and-weave them heels

 

[animetiddyenthusiast]

animetiddyenthusiast updated ALXR Quest/Pico client v0.12.1 for server v18.5.0 with modified passthrough mask with a new update entry:

Last update for a while, quest only

Version parity with my client since I'll be unable to work on this for a week or two. should be slight improvement for quest users but I'm not sure what the last upload was. not different enough to remake for pico sorry babes

Read the rest of this update entry...

 

[animetiddyenthusiast]

animetiddyenthusiast updated ALXR Quest/Pico client v0.12.1 for server v18.5.0 with modified passthrough mask with a new update entry:

The End?

Virtual Desktop posted a new beta update with chroma key passthrough. They use nearly identical formulas to my version and achieve similar results while having far better performance and quality than alxr. I'm not done developing in this space but I may be done with this project. There are some updates coming soon to alxr that would be very exciting if not for this situation, but may still be enough to keep me involved.

Haven't decided yet, but tbh I have to just recommend you go use VD for...

Read the rest of this update entry...

 

[atani]

Good luck on your other projects and thank you for making these ?‍

 

[giallone]

I really understand your point but, don't forget that VD is paid (i have it) and alxr is not so, is a point at your favor in that regard.
I still have to try the AR in VD, i only used AR with your alxr version and the result is mind blowing for me...
thank you really a lot for doing it and the best of luck with your other projects

 

[animetiddyenthusiast]

I really understand your point but, don't forget that VD is paid (i have it) and alxr is not so, is a point at your favor in that regard.
I still have to try the AR in VD, i only used AR with your alxr version and the result is mind blowing for me...
thank you really a lot for doing it and the best of luck with your other projects

yeah I did consider that but tbf everyone here is an adult and I figure if you can afford a vr headset you can afford a $15 app. Still its a good point and after some consideration I will continue providing some updates