How to Customize ALVR-Client for Quest 2 Passthrough on Windows with Any Mask Color

krch

New member
Messages
8
Reactions
9
Points
3
The intent of this walkthrough is to give you the opportunity to create your own Client for your Quest2. Binaries are not part of this, but you'll find members of the community that have been creating theirs below, there are some other threads for the binaries. Consider security risks associated to download and install executables to your VR Headset.

If you own an Oculus Quest 2/Pro/Pico4 and want to customize the mask color, follow these steps to compile your own ALVR-Client on Windows. If you don't have any context take a look at this other threads:
  1. Passthrough Trick discussion;
  2. @thinoreos tutorial
  3. Background-color camera session plugin by @Sally Whitemane
Please note that this guide is updated as of 24th SEP 2023. This is just a clarification for: https://github.com/korejan/ALVR/wiki/ALXR-Client

Important prerequisites for this guide:
  • Familiarity with passthrough threads (see above)
  • Knowledge of the background color plugin
  • Ability to activate passthrough in Quest controllers
  • Basic programming knowledge,
  • Knowledge in how to use SideQuest
Special thanks to the openXR and ALVR teams, especially korejan, as well as members of the vamhub community, particularly @Tankshell, who provided guidance in the right direction on the forum, and @Pbpjan15, who demonstrated first the feasibility of this customization.

Happy customizing!

-------------------------------------------------------------------------------

PART 1: PREPARE THE ENVIRONMENT AND DEPENDENCIES:

1.1 Ensure you have the following software installed, obtained from trusted sources:
Android Studio Giraffe | 2022.3.1 /also tested with Flamingo
Build #AI-223.8836.35.2231.10406996, built on June 29, 2023
Runtime version: 17.0.6+0-b2043.56-10027231 amd64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o., Windows 11 10.0
rustc 1.70.0 (90c541806 2023-05-31)
Python 3.8.10
choco 2.1.0
LLVM 16.0.6
Vulkan SDK 1.3.250.1

1.2 Create the necessary environment variables using the examples below. Adjust them according to your environment, replacing "<your_user>" with your actual user, and modifying paths to match your installations:
ANDROID_NDK_ROOT = C:\Users\<your_user>\AppData\Local\Android\Sdk\ndk\26.0.10792818
ANDROID_SDK_ROOT = C:\Users\<your_user>\AppData\Local\Android\Sdk
JAVA_HOME =C:\Program Files\Android\AndroidStudio\jbr
LIBCLANG_PATH = C:\Program Files\LLVM\bin
also added to PATH environment variable:
C:\Program Files\Android\AndroidStudio\jbr\bin
C:\Users\<your_user>\.cargo\bin
C:\Users\<your_user>\AppData\Local\Programs\Python\Python38\

1.3 Open AndroidStudio,

1.3.1.- Create a new project by clicking on "Get from VCS" and clone the Git repository from korejan. (https://github.com/korejan/ALVR.git)

(Tip: Consider creating a new folder at the root of your hard drive, e.g., C:\P, to avoid potential path length issues during compilation.)

1690529103188.png


1.3.2.- Once the code is downloaded, open the SDK Manager and navigate to System Settings > Android SDK. Then, follow these steps:

A) Under SDK Platform, install "Android 12L (sv2) API level 32."
1690529119975.png

B) In SDK Tools, install the following:
b1)Android SDK-Build-Tools version 32.1.0-rc1,
1690529130279.png

b2)NDK 26.0.10792818 ,
1695582923134.png

b3)Android SDK Command-line Tools version 9. and
1690529143691.png

b4)CMAKE 3.22.10
1690529150104.png

(TIP: The Android Studio might add "Android emulator" and "sdkplatform-tools" for version 34.0.4, which may not be necessary, but they won't interfere. You can leave them there. )
1690529157351.png


------------------------------------------------------------------------------
PART 2: CHANGE SOME CODE:

1.- Change the passthrough mask:


Locate the file alvr/openxr-client/alxr-engine-sys/cpp/ALVR-OpenXR-Engine/src/alxr_engine/vulkan_shaders/passthroughMask_frag.glsl. Save a backup of the file before making changes.

Here are 3 versions , the third being the most advanced and optimal. (24 SEP 2023)

v1.
Instead of using the constant IDs, directly modify the main function and values like this:

Code:
//version 1  JUL 28th 23
void main()

{

vec4 color = SampleVideoTexture();

color.a = all(greaterThan(color.rgb,vec3(0.28,0.30,0.74))) ? AlphaValue : 1.0f;

FragColor = color;

}

///v 2 UPDATE 8-10-23 (Realized RGB can represented as a 3D cube, hence lets define a sphere zone around a color and not a threshold or zone as in the first version approach, you can find the complete code below, the error is this is not RGB)

Code:
#version 460
#ifdef ENABLE_ARB_INCLUDE_EXT
    #extension GL_ARB_shading_language_include : require
#else
    // required by glslangValidator
#extension GL_GOOGLE_include_directive : require
#endif
#pragma fragment

#include "common/baseVideoFrag.glsl"

layout(location = 0) out vec4 FragColor;

void main()
{
    vec4 color = SampleVideoTexture();
    vec3 sphereCenter = vec3(0.72, 0.71, 0.89);  // Define the center of the sphere Light Blue, almost white
    float sphereRadius = 0.15;                   // Define the radius of the sphere

    float distanceToSphereCenter = distance(color.rgb, sphereCenter); // Calculate the distance from the current pixel's position to the sphere's center

    bool isInsideSphere = distanceToSphereCenter <= sphereRadius;  // Check if the pixel is inside the sphere
    // Set the alpha value of the color based on whether the pixel is inside the sphere
    color.a = isInsideSphere ? 0.0f : 1.0f;  // Pixel inside sphere: fully transparent, Thanks to @uld, outside: fully opaque
    FragColor = color;
}


Thanks to @bd-energy ,@animetiddyenthusiast , @uld and @fiestamart , I understood more about the colorspaces and the comparison of colors.

I used the delta2000 equation found by @fiestamart in Rachmanin0xF's color the functions: and placed in common/color-functions.glsl then added to the project in the common directory and referenced it in the code:

https://github.com/Rachmanin0xFF/GLSL-Color-Functions

1695584624019.png


//v3 LAB color optimizations process described in:
alxr-quest-pico-client-v0-12-1-for-server-v18-5-0-with-modified-passthrough-mask
by @animetiddyenthusiast


Code:
#version 460
#ifdef ENABLE_ARB_INCLUDE_EXT
    #extension GL_ARB_shading_language_include : require
#else
    // required by glslangValidator
    #extension GL_GOOGLE_include_directive : require
#endif
#pragma fragment

#include "common/baseVideoFrag.glsl"
#include "common/color-functions.glsl" //added from Rachmanin0xF github

vec3 RGB_TO_LAB(vec3 rgb) {
    return XYZ_TO_LAB(RGB_TO_XYZ(rgb));
}

float when_lt(float x, float y) {
    return max(sign(y - x), 0.0);
}

layout(location = 0) out vec4 FragColor;
const vec3 KeyColor_LAB = vec3(21.0486,-5.2067,21.8668); //Modify as you wish in LAB format this code is between brown/green/yellow

void main()
{
  
    vec4 sampleRGB = SampleVideoTexture();
    vec3 sampleLAB = RGB_TO_LAB(sampleRGB.rgb);
    float deltaE = LAB_DELTA_E_CIE2000(sampleLAB, KeyColor_LAB);
    sampleRGB.a -= when_lt(deltaE, 10.0);   //tolerance, try 10 for instance
    FragColor = sampleRGB;
}

----------------------------------------------------------------------------------------------
PART 3 COMPILE:

According to https://github.com/korejan/ALVR/wiki/ALXR-Client

1.-Open your terminal in Android Studio and run:

cargo xtask build-alxr-quest --release

1690529943694.png


If you encounter any issues, refer to the 2 workaround mentioned in the "Dev Builds for Android (Any Flavor)" section in the same link.

Here is the first: kill the openjdk in Android Studio when processing reaches 0,
The second one is explained below by bd-energy and it's also in the aforementioned page.


1690531256606.png


1690529877766.png

The APK client should be around 3MB and can be found in the ..target\quest\release\apk folder in your project.


1690530130846.png


------------------------------------------------------------------------------------------------
PART 4 INSTALL


Transfer the compiled APK to your headset using SideQuest or similar tools. Delete the previous client and then transfer the new one.

1690530186695.png


Once you installed the Client, start your ALVRServer, Connect your Quest and start the Client; start VAM; and activate the passthrough; find the transparency in the ColorBackground Plugin, make sure you clean your scene according to your preferences.
 
Last edited:
Good instructions, glad to see this actually became viable to do. Thanks for your post!
 
Thank you for creating the guide and distributing the compiled data.
I am using QuestPro, and the compiled data is transparent, but it looks like a white veil.
The green version of Pbpjan15 and the black version of korejan are completely transparent.

I am currently trying to compile it myself, but the process stops at the following point.
I have also tried workaround, but it seems to be failing at an earlier stage.

I may not have a lot of information, but can you give me some idea of the cause?

can't proceed from this process
-----
warning: `alvr_sockets` (lib) generated 1 warning
Building [=======================> ] 255/259: alxr-engine-sys(build)
------
 
Thank you for creating the guide and distributing the compiled data.
I am using QuestPro, and the compiled data is transparent, but it looks like a white veil.
The green version of Pbpjan15 and the black version of korejan are completely transparent.

I am currently trying to compile it myself, but the process stops at the following point.
I have also tried workaround, but it seems to be failing at an earlier stage.

I may not have a lot of information, but can you give me some idea of the cause?

can't proceed from this process
-----
warning: `alvr_sockets` (lib) generated 1 warning
Building [=======================> ] 255/259: alxr-engine-sys(build)
------

This is practically the last step....your`re almost there!...At that point is where you open the task manager, and wait for the JDK process under the Android Studio to go to 0% CPU, then you kill it

There are some warnings , but it should compile...Let me know what errors do you see after you kill the process.

The most common cause is missing dependencies or wrong versions, but its never evident. You have to verify the errors and back-trace in the terminal log; its easier if you copy them to a notepad to analyze them, looking for "errors" and "warnings".




You are right about the transparency and the veil.. I'll take a look after, I guess is the alphavalue and the "f" datatype.. let me know if you find the solution.
 
Last edited:
This is practically the last step....your`re almost there!...At that point is where you open the task manager, and wait for the JDK process under the Android Studio to go to 0% CPU, then you kill it

There are some warnings , but it should compile...Let me know what errors do you see after you kill the process.

The most common cause is missing dependencies or wrong versions, but its never evident. You have to verify the errors and back-trace in the terminal log; its easier if you copy them to a notepad to analyze them, looking for "errors" and "warnings".

View attachment 276812
View attachment 276813

You are right about the transparency and the veil.. I'll take a look after, I guess is the alphavalue and the "f" datatype.. let me know if you find the solution.
Thank you for your response.
It seemed that the reason was that I had ignored the tip you had written.
I followed this and was able to compile successfully. (taskkill is a must)

I did not follow this.
(Tip: Consider creating a new folder at the root of your hard drive, e.g., C:\P, to avoid potential path length issues during compilation.)

I have not been able to play with the parameters yet and will try again when I have more time.
I will certainly report back if it works!
 
I am not sure why, but I have succeeded in removing the veil.

Here's the part.
color.a = all(greaterThan(color.rgb,vec3(0.28,0.30,0.74))) ? AlphaValue : 1.0f;

I changed it like this and it is now completely transparent.
color.a = all(greaterThan(color.rgb,vec3(0.28,0.30,0.74))) ? 0.0f : 1.0f;

If we do not change the original description, it will still be completely transparent, so if we write this way
I wonder if AlphaValue has become a non-transparent value.
*I have no programming knowledge and do not understand

Also, I started to try range-zone, but I can't compile it properly due to lack of study.

The [all(greaterThan(color.rgb,vec3(0.28,0.30,0.74)))] part,
[if (all(greaterThan(color.rgb,vec3(0.03,0.06,0.90))) && (all(lessThan(color.rgb,vec3(0.20,0.20) ,0.99)))))]
If I rewrite the code to "0.99", will it compile if it is normal?
 
I am not sure why, but I have succeeded in removing the veil.

Here's the part.
color.a = all(greaterThan(color.rgb,vec3(0.28,0.30,0.74))) ? AlphaValue : 1.0f;

I changed it like this and it is now completely transparent.
color.a = all(greaterThan(color.rgb,vec3(0.28,0.30,0.74))) ? 0.0f : 1.0f;

If we do not change the original description, it will still be completely transparent, so if we write this way
I wonder if AlphaValue has become a non-transparent value.
*I have no programming knowledge and do not understand

Also, I started to try range-zone, but I can't compile it properly due to lack of study.

The [all(greaterThan(color.rgb,vec3(0.28,0.30,0.74)))] part,
[if (all(greaterThan(color.rgb,vec3(0.03,0.06,0.90))) && (all(lessThan(color.rgb,vec3(0.20,0.20) ,0.99)))))]
If I rewrite the code to "0.99", will it compile if it is normal?

Hi,

Could you perhaps upload your file? If you find a solution :)

Many thanks.
 
Sorry for repeating myself, I added the else and it compiled successfully.

Compiled in the range (0.00, 0.00, 0.196)-(0.286, 0.31, 0.765),

I set the VaM background to 113,113,130, and the border is much less noticeable!
However, the silver hair and whitish areas show through, so it is difficult to find the right balance.
I will try different things.
Hi,

Could you perhaps upload your file? If you find a solution :)

Many thanks.

I am concerned because I am not familiar with a programing.
I will have krch look at it and if it is OK, I will upload it here.
 
Sorry for repeating myself, I added the else and it compiled successfully.

Compiled in the range (0.00, 0.00, 0.196)-(0.286, 0.31, 0.765),

I set the VaM background to 113,113,130, and the border is much less noticeable!
However, the silver hair and whitish areas show through, so it is difficult to find the right balance.
I will try different things.


I am concerned because I am not familiar with a programing.
I will have krch look at it and if it is OK, I will upload it here.

Hi, no problemo. OP updated their imbedded file (V2) and is a much better version. Many thanks.
 
If you own an Oculus Quest 2 and want to customize the mask color beyond the default black or green, follow these steps to compile your own ALVR-Client on Windows. Before you begin, make sure you understand the process by reading the relevant documentation and threads:
  1. Start by going through this thread to familiarize yourself with the passthrough trick: https://hub.virtamate.com/threads/d...till-work.25728/page-4?view=votes#post-109604
  2. Full support of Oculus Quest mixed reality with passthrough, check out this thread: https://hub.virtamate.com/threads/p...-with-passthrough-into-vam.10951/#post-106098
  3. Here is Sally Whitemane the creator who made a background-color camera plugins;

Here is a low-res sample of the result, thanks to one of the most talented creators around: C&G Studio , this is from "A Dangerous Candy" , o_O They make some "performance" versions, so no props, backgrounds , its easy to change the WindowCamera backcolor with any of 2 Sally Plugin and you're done.. You can see my curtains in my room even through her hair in motion! (borders are almost invisible)

View attachment 275891


Please note that this guide is updated as of 10th August 2023. Take your time to read carefully and be patient throughout the process. It may seem overwhelming with many documents, but I aim to provide you with the essential steps only. Its just a clarification for: https://github.com/korejan/ALVR/wiki/ALXR-Client

Important prerequisites for this guide:
  • Familiarity with passthrough threads for Quest 2 (see above)
  • Knowledge of the background color plugin
  • Ability to activate passthrough in Quest controllers
  • Basic programming knowledge,
  • knowledge in how to use SideQuest
Special thanks to the openXR and ALVR teams, especially korejan, as well as members of the vamhub community, particularly @Tankshell, who provided guidance in the right direction on the forum, and @Pbpjan15, who demonstrated first the feasibility of this customization.

Happy customizing!


-------------------------------------------------------------------------------

PART 1: PREPARE THE ENVIRONMENT AND DEPENDENCIES:

1.1 Ensure you have the following software installed, obtained from trusted sources:
Android Studio Flamingo | (2022.2.1 Patch 2) //UPDATE 2-8-23 also works with Android Studio Giraffe | 2022.3.1
Build #AI-223.8836.35.2231.10406996, built on June 29, 2023
Runtime version: 17.0.6+0-b2043.56-10027231 amd64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o., Windows 11 10.0
rustc 1.70.0 (90c541806 2023-05-31)
Python 3.8.10
choco 2.1.0
LLVM 16.0.6
Vulkan SDK 1.3.250.1

1.2 Create the necessary environment variables using the examples below. Adjust them according to your environment, replacing "your_user" with your actual user, and modifying paths to match your installations:
ANDROID_NDK_ROOT = C:\Users\your_user\AppData\Local\Android\Sdk\ndk\25.2.9519653
ANDROID_SDK_ROOT = C:\Users\your_user\AppData\Local\Android\Sdk
JAVA_HOME =C:\Program Files\Android\AndroidStudio\jbr
LIBCLANG_PATH = C:\Program Files\LLVM\bin
added to PATH:
C:\Program Files\Android\AndroidStudio\jbr\bin
C:\Users\your_user\.cargo\bin
C:\Users\your_user\AppData\Local\Programs\Python\Python38\

1.3 Open AndroidStudio,

1.3.1.- Create a new project by clicking on "Get from VCS" and clone the Git repository from korejan.

(Tip: Consider creating a new folder at the root of your hard drive, e.g., C:\P, to avoid potential path length issues during compilation.)

View attachment 273861

1.3.2.- Once the code is downloaded, open the SDK Manager and navigate to System Settings > Android SDK. Then, follow these steps:

A) Under SDK Platform, install "Android 12L (sv2) API level 32."
View attachment 273862
B) In SDK Tools, install the following:
b1)Android SDK-Build-Tools version 32.1.0-rc1,
View attachment 273863
b2)NDK 25.2.9519653 ,
View attachment 273864
b3)Android SDK Command-line Tools version 9. and
View attachment 273865
b4)CMAKE 3.22.10
View attachment 273866
(TIP: The Android Studio might add "Android emulator" and "sdkplatform-tools" for version 34.0.4, which may not be necessary, but they won't interfere. You can leave them there. )
View attachment 273867

------------------------------------------------------------------------------
PART 2: CHANGE SOME CODE:
I made three modifications to the code:

1.-(OPTIONAL) In the file alvr/xtask/src/dependencies.rs, go to line 158 and update the ID for the latest Oculus OpenXR Mobile SDK. Click on the download link and copy the correct ID from the URL. For example, if the version is 55.0, the ID in the download link may be 6559707940734404. Modify the link as follows:

Original: "https://securecdn.oculus.com/binaries/download/?id=4421717764533443"
Modified: "https://securecdn.oculus.com/binaries/download/?id=6559707940734404"



2.- In the file alvr/xtask/src/commands.rs, go to line 129 and add "--ssl-no-revoke". Change the following line:
&["-L", "-o", &destination.to_string_lossy(), "--url", url],
for:
&["-L", "-o", &destination.to_string_lossy(),"--ssl-no-revoke","--url", url],

Note: Disabling SSL certificate revocation validation with "--ssl-no-revoke" is potentially risky, as the Oculus OpenXR cert might be invalid. Consider finding a safer workaround.

3.- Change the passthrough mask:

Locate the file alvr/openxr-client/alxr-engine-sys/cpp/ALVR-OpenXR-Engine/src/alxr_engine/vulkan_shaders/passthroughMask_frag.glsl. Thanks to @Tankshell for the discovery. Note: Save a backup of the file before making changes.

v1.
Instead of using the constant IDs, directly modify the main function and values like this:

Code:
//version 1
void main()

{

vec4 color = SampleVideoTexture();

color.a = all(greaterThan(color.rgb,vec3(0.28,0.30,0.74))) ? AlphaValue : 1.0f;

FragColor = color;

}

///UPDATE 8-10-23 (Realized RGB can represented as a 3D cube, hence lets define a sphere zone around a color and not a threshold or zone as in the first version approach.)
Code:
V2. 

void main()

{

vec4 color = SampleVideoTexture();

vec3 sphereCenter = vec3(0.72, 0.71, 0.89); // Define the center of the sphere Light Blue

float sphereRadius = 0.15; // Define the radius of the theorical sphere



float distanceToSphereCenter = distance(color.rgb, sphereCenter); // Calculate the distance from the current pixel's position to the sphere's center



bool isInsideSphere = distanceToSphereCenter <= sphereRadius; // Check if the pixel is inside the sphere

// Set the alpha value of the color based on whether the pixel is inside the sphere

color.a = isInsideSphere ? 0.0f : 1.0f; // Thanks to [USER=54310]@uld[/USER], Pixel inside sphere: fully transparent, outside: fully opaque

FragColor = color;

}

The modified code results in a white mask, and thanks to @uld , the client has no longer a white veil. This is good for Quest 2 black and white passthrough, not sure in full color with Pro and Quest3. I guess it depends on the scene light and the room you are into. Feel free to experiment and create your own. that is the point of this thread. For the client code convert each RGB value to a range between 1.0 and 0.0, (check the OpenXR API for details).
Then use the Camera Background Color or Sally Session Plugin , from Sally; for this particular tutorial and alvr-client try settings: 220,220,240mm.

View attachment 273891


NOTE: There is lots of theory and mathematics here. On movies the technique is called chroma keying, the choice of green or blue screen depends on the subject being filmed and the final environment you want to replace the background with. Green and blue are the most commonly used colors because they are not typically found in human skin tones or most clothing, making it easier to separate the subject from the background.
In RGB (Red, Green, Blue) color space, the most common chroma key color is "Chroma Green," which is a specific shade of green with RGB values approximately R:0, G:177, B:64. This particular green color is often used in studios and is widely supported by most video editing software.
However, in some cases, "Chroma Blue" is used instead, especially when the subject being filmed may have green elements (such as green clothing or props). Chroma Blue is a specific shade of blue with RGB values approximately R:0, G:71, B:171.

Search for colour picker in google, or use the 3d color picker to visualize RGB in 3d
remember instead of RGB 255-0 you'll use 1.0 to 0.0, convert the values

OpenGL_R = R / 255 OpenGL_G = G / 255 OpenGL_B = B / 255
Let's apply this to the given RGB color (73, 77, 191):
OpenGL_R = 73 / 255 ≈ 0.28627 OpenGL_G = 77 / 255 ≈ 0.30196 OpenGL_B = 191 / 255 ≈ 0.74902


----------------------------------------------------------------------------------------------
PART 3 COMPILE:

According to https://github.com/korejan/ALVR/wiki/ALXR-Client

1.-Open your terminal in Android Studio and run:

cargo xtask build-alxr-quest --release

View attachment 273880

If you encounter any issues, refer to the workaround mentioned in the "Dev Builds for Android (Any Flavor)" section in the same link. For example, you can kill the openjdk in Android Studio when processing reaches 0.


View attachment 273892

View attachment 273879
The APK client should be around 3MB and can be found in the ..target\quest\release\apk folder in your project.


View attachment 273882

------------------------------------------------------------------------------------------------
PART 4 INSTALL


Transfer the compiled APK to your headset using SideQuest or similar tools. Delete the previous client and then transfer the new one.

View attachment 273883

This method may be useful for Quest Pro as well, and with the right dependencies, code modifications can be applicable to Pico4. If someone succeeds in adapting it for other platforms, please share your experiences.


//////////////////////////////////////////////////////////////////////////

UPDATE 2-8-23 in response to @worm380

Attached the V1 file resulting from this tutorial. No longer available

//////////////////////////////////////////////////////////////////////////

UPDATE 10-8-23

Remove the .blue2.var extension from the filename, then sideload to your quest and start the client. It works with ALVR 10.4.0.

Hey is this a typo? I couldn’t get it to work with ALVR. I couldn’t even find an ALVR version 10.4.0. Version 10 is from 2020. I went back to my previous install which is version 18.2.3 which works with the green screen. It would be great to give this a try with your compiled apk file. Can you clarify? Niceguy.
 
Hey is this a typo? I couldn’t get it to work with ALVR. I couldn’t even find an ALVR version 10.4.0. Version 10 is from 2020. I went back to my previous install which is version 18.2.3 which works with the green screen. It would be great to give this a try with your compiled apk file. Can you clarify? Niceguy.
Yes , it was a typo I meant ALVR 18.4.0; tnks! ALXR .10 works with any ALVR 18.X version; It seems support for 19 and 20 may come in the future; you can find 18.4.0 ALVR server in Korejan's nightly builds.
 
Last edited:
Tried to do this a while back but had a bunch of issues compiling even after going through his suggested fixes on GitHub. Appreciate the precompiled version. I'll probably try to do it myself once again.


Yes , it was a typo I meant ALVR 18.4.0; tnks! ALXR .10 works with any ALVR 18.X version; It seems support for 19 and 20 may come in the future; you can find 18.4.0 ALVR server in Korejan's nightly builds.

Did you by any chance try building the v20 branch? I tried and couldn't get it to work but I couldn't even get the master branch to work so I'm not sure how far along those are. V20 alvr is much better for performance and visuals so it will be a nice upgrade
 
Managed to get my build working and uploaded an updated apk for the latest nightly build that uses a new server version (v18.4 to v18.5).

I used the same runtimes as listed here except I used the openjdk v11 build included with visual studio. I also followed the advice from the github to select a build variant and build in android studio.

At this point it still didn't work. Turns out the major problem was my environment variables not being read properly even after changing them in windows. While keeping android studio open I opened the outermost ALVR folder in visual studio and launched powershell. From there I ran the following commands to set my environment variables

$Env:ANDROID_SDK_ROOT = "C:\Users\your name\AppData\Local\Android\Sdk"
$Env:ANDROID_NDK_ROOT = "C:\Users\your name\AppData\Local\Android\Sdk\ndk\25.2.9519653"

then I was able to build with no issues

EDIT: I have updated my resource to include both quest and pico v4 builds. If anyone needs a different build or wants a specific color let me know. I'm going to investigate a way to alter the shader at runtime or if that proves too hard I might just rename the app so I can have multiple versions with their own color
 
Last edited:
I have followed all the instruction but I got stuck during the compile as below


warning: `alvr_sockets` (lib) generated 1 warning
Building [=======================> ] 252/256: alxr-engine-sys(build) <- stuck here!!


I also made sure to clone the app under C:\P\ as well.
I have installed several different JDK to try but the result were all the same :(

I have waited for like 10 mins and killed 'OpenJDK Platform Binary' then I get error like below

******************************
error: failed to run custom build command for `alxr-engine-sys v0.12.1 (F:\P\alvr\openxr-client\alxr-engine-sys)`
note: To improve backtraces for build dependencies, set the CARGO_PROFILE_RELEASE_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation.

Caused by:
process didn't exit successfully: `F:\P\target\debug\build\alvr_filesystem-4c263e640ebaa645\out\../../../..\quest\release\build\alxr-engine-sys-e13fc9059ff528c6\build-script-build` (exit code: 101)
...
...
error: process didn't exit successfully: `target\debug\alvr_xtask.exe build-alxr-quest --release` (exit code: 101)
PS F:\P>

**********************************

Any suggestion would be greatly appreciated
Thank you
 
Long guide incoming to help @Emosion but for anyone else I have attached an apk of the newest quest build on this post. the master repo has updated for oculus v56 and I have altered the colors slightly on the mask and blend. I have not tested this build yet so use at your own risk, I'll probably update my resource post once I've tested it and I do plan to continue working on it.

I've been reading up on vulkan and openxr and hope to soon put out a version that allows for configuration of the color so we don't have to keep building it. I could just add more shaders to swap through or maybe just have the shader read from a file that we can access in sidequest. the dream would be an in app color picker but I don't know how to do that yet lol. I may also try to improve the blending and scene color detection to prioritize background color better. Tbh I kinda regret doing all this on this alias on a porn site bc if I continue with this I'll probably do a pull request on github to merge it into the main branch which would be a nice thing to share publicly but I guess I can just scrub my posts about it from here once I'm ready for release and just direct people to the github without claiming ownership.


Now for the guide:

follow the steps on github to build for your desired platform in android studio by opening alvr\openxr-client\alxr-engine-sys\android and choosing select build variant. make sure to go into gradle settings and project structure, set the target sdk to 33 using build tools 32 for the variants you want and set java to the visual studio version which is runtime 11. It can be a different version but that one works well and I recommend doing the actual build in vs so it makes sense to use its native java version. actually while we're on that subject I would go in and make sure you have every optional component for vs 22 that doesnt say deprecated.
1692936816558.png
1692936770105.png
1692936930407.png

Remove all environment variables related to java or android in both windows and studio then specify path to the vs jdk although android studio seems to be able to find it as long as you specify the right version.

Now make sure the repo is properly cloned, you can always redo if you think you broke something. Open the outer folder in VS so you have this structure and open developer powershell. It should actually sync up between vs and as.
1692934154207.png


Now make run the following in powershell to set the android paths doesnt really matter where you run it as long as its in the project

Code:
$Env:ANDROID_SDK_ROOT = "C:\Users\your name\AppData\Local\Android\Sdk"
$Env:ANDROID_NDK_ROOT = "C:\Users\your name\AppData\Local\Android\Sdk\ndk\25.2.9519653"

From the outermost folder run "cargo xtask bump-alxr-versions --nightly" then "cargo update" and "git submodule update --init --recursive" then cd alvr and build dependencies with cargo xtask build-android-deps. once that finishes switch over to android studio (LEAVE BOTH OPEN), confirm your build variant and jdk version again and make any code changes at this point. I personally did the actual code stuff in android studio but it doesnt matter too much which ide you use. Once that's ready, run cargo xtask clean in visual studio, switch back to android studio and build the project. it should finish without any errors. if it gives you an error here then you have a missing dependency and you need to just trace through the log and your steps to fix it. Make sure no java process is running in task manager details tab, then run "cargo xtask build-alxr-quest --release" in visual studio from alvr. it should build at this point if you did it right. it will advance past 252/259 and stall at 255 for a while. once it hits 256 you close java and it should advance and finish. you can find the apk in your build folder in the outermost folder.


With all that said you can also just ask me to build it if you have a request for some specific change as I already have my environment set up to easily push changes
 

Attachments

  • 1692936721664.png
    1692936721664.png
    34.8 KB · Views: 0
  • alxr_client_android.zip.var
    6.5 MB · Views: 0
Last edited:
Long guide incoming to help @Emosion but for anyone else I have attached an apk of the newest quest build on this post. the master repo has updated for oculus v56 and I have altered the colors slightly on the mask and blend. I have not tested this build yet so use at your own risk, I'll probably update my resource post once I've tested it and I do plan to continue working on it.

I've been reading up on vulkan and openxr and hope to soon put out a version that allows for configuration of the color so we don't have to keep building it. I could just add more shaders to swap through or maybe just have the shader read from a file that we can access in sidequest. the dream would be an in app color picker but I don't know how to do that yet lol. I may also try to improve the blending and scene color detection to prioritize background color better. Tbh I kinda regret doing all this on this alias on a porn site bc if I continue with this I'll probably do a pull request on github to merge it into the main branch which would be a nice thing to share publicly but I guess I can just scrub my posts about it from here once I'm ready for release and just direct people to the github without claiming ownership.


Now for the guide:

follow the steps on github to build for your desired platform in android studio by opening alvr\openxr-client\alxr-engine-sys\android and choosing select build variant. make sure to go into gradle settings and project structure, set the target sdk to 33 using build tools 32 for the variants you want and set java to the visual studio version which is runtime 11. It can be a different version but that one works well and I recommend doing the actual build in vs so it makes sense to use its native java version. actually while we're on that subject I would go in and make sure you have every optional component for vs 22 that doesnt say deprecated.
View attachment 281674View attachment 281673View attachment 281675
Remove all environment variables related to java or android in both windows and studio then specify path to the vs jdk although android studio seems to be able to find it as long as you specify the right version.

Now make sure the repo is properly cloned, you can always redo if you think you broke something. Open the outer folder in VS so you have this structure and open developer powershell. It should actually sync up between vs and as.
View attachment 281663

Now make run the following in powershell to set the android paths doesnt really matter where you run it as long as its in the project

Code:
$Env:ANDROID_SDK_ROOT = "C:\Users\your name\AppData\Local\Android\Sdk"
$Env:ANDROID_NDK_ROOT = "C:\Users\your name\AppData\Local\Android\Sdk\ndk\25.2.9519653"

From the outermost folder run "cargo xtask bump-alxr-versions --nightly" then "cargo update" and "git submodule update --init --recursive" then cd alvr and build dependencies with cargo xtask build-android-deps. once that finishes switch over to android studio (LEAVE BOTH OPEN), confirm your build variant and jdk version again and make any code changes at this point. I personally did the actual code stuff in android studio but it doesnt matter too much which ide you use. Once that's ready, run cargo xtask clean in visual studio, switch back to android studio and build the project. it should finish without any errors. if it gives you an error here then you have a missing dependency and you need to just trace through the log and your steps to fix it. Make sure no java process is running in task manager details tab, then run "cargo xtask build-alxr-quest --release" in visual studio from alvr. it should build at this point if you did it right. it will advance past 252/259 and stall at 255 for a while. once it hits 256 you close java and it should advance and finish. you can find the apk in your build folder in the outermost folder.


With all that said you can also just ask me to build it if you have a request for some specific change as I already have my environment set up to easily push changes

Thank you so much for your detail guide. I will follow the steps when I get home!!


The reason for compiling the code is to test out what @uld mentioned about removing the outline by setting
the AlphaValue set to 0.0f instead of deafult set (AlphaValue = 0.3f)

color.a = all(greaterThan(color.rgb,vec3(0.28,0.30,0.74))) ? AlphaValue : 1.0f;

I have Quest Pro and the passthrough model shows the significant white outlining due to color passthrough with your complied version. So hoping setting AlphaValue = 0.0f eliminates the white outlines

Another test is changing original code from

layout(constant_id = 9) const float AlphaValue = 0.3f;
layout(constant_id = 10) const float KeyColorR = 0.01f;
layout(constant_id = 11) const float KeyColorG = 0.01f;
layout(constant_id = 12) const float KeyColorB = 0.01f;
..
void main()
{
vec4 color = SampleVideoTexture();
color.a = all(lessThan(color.rgb, key_color)) ? AlphaValue : 1.0f;
FragColor = color;
}

to like below

layout(constant_id = 10) const float KeyColorR = 0.003f;
layout(constant_id = 11) const float KeyColorG = 0.003f;
layout(constant_id = 12) const float KeyColorB = 0.003f; //

(1 / 256 = 0.0039 so masking less than 0.003f means to mask color= 0)

I think this would only mask the color that is 100% back RGB(0,0,0) where RGB(255,255,255) is 100% white
then set my background color to RGB(0,0,0) then it would only mask my background to minimize the masking of the blackish area by doing so. That is what I am thinking and wanted to try to see if works good for Quest Pro.
 
Last edited:
Hey, dev of alxr here. I didn't know this was going on and I feel kind of bad that people are resorting to this. Also apologies for the convoluted structure of the project atm, this partly historical and partly priority / little free time for the reasons why it's like this. I will be doing a major clean-up/simplification soon.

@krch In your guide please delete part 2-1 & 2-2, this is not relevant and not needed for building alxr, the submodule already contains the sdk and I keep it up to date.

For openjdk hanging in windows, you guys probably missed the comment in the wiki, if you open `alvr\openxr-client\alxr-engine-sys\android` in Android-studio build once and leave it open then you can build multiple times without having to manually kill the process. Alternatively build it in linux (you can use wsl) where it's not an issue.

@animetiddyenthusiast (and eveyone else invovled).

First I would like to explain why the passthrough mask mode is (still) like this. I've always been intending on doing a complete re-write of the masking shader to do proper chroma/luma keying with user-defined keys & thresholds nothing like it currently is. What has been preventing me from having done this yet is because of a few reasons, nowadays i have limited free-time and I've been prioritizing and focusing on other areas of alxr (or around it).

The other reason is that in order for this to be useful and not frustating is a need for UI with real-time feedback, i personally believe the UI should be done in the client, there are other client-side settings that are just not visible/easy to set. Doing this will take time to do, I'm thinking of adding ImGUI support to the client but will need to make framework to make this too work as 3D UI elements with XR controls

The other alternative is to put the UI in the server dashboard and make some new network packet types to send to the client. It seems like that would be quickier but adding new UI to the server dashboard is headache inducing that I don't want to deal with that (you'll need to deal with hand-writing html/css/js/jquery & rust interop). It will also need to be completely redone once if/when I make the clients compatible with v20 upwards as v20 uses a completely different UI framework now.

As a really short-term solution I can add android system properties that you can set but you'll need to use adb to set them and they will need to be set before the client is started unless I add a thread for polling system property changes.

As for how the masking shader should work there are two ways I would go about doing this (and what you guys should consider doing I think):

A. Convert to LAB colour space and use a version of the delta-e equations to do key comparison.

And/Or

B. Use the YCbCr colour space for comparision, the video texture is typically in a YCbCr format but currently I'm always using a Vulkan YCbCr conversion sampler so either I add a new code path to disable using it or you convert the pixel back to YCbCr colorspace (which seems a bit silly).

In either case you should add some threshold parameters. I recommend looking up using Vulkan specialization constants but I can sort those things out for anyeone.

If you would like to discuss anything about customizing the shader/Vulkan you contact me on discord, you'll find me in the vam server.
 
Something I forgot to mention, I would recommend testing shader changes on the ShaderToy website, you can find a few chroma-keying shaders that you can quickly modify.
 
Hey, dev of alxr here. I didn't know this was going on and I feel kind of bad that people are resorting to this. Also apologies for the convoluted structure of the project atm, this partly historical and partly priority / little free time for the reasons why it's like this. I will be doing a major clean-up/simplification soon.

@krch In your guide please delete part 2-1 & 2-2, this is not relevant and not needed for building alxr, the submodule already contains the sdk and I keep it up to date.

For openjdk hanging in windows, you guys probably missed the comment in the wiki, if you open `alvr\openxr-client\alxr-engine-sys\android` in Android-studio build once and leave it open then you can build multiple times without having to manually kill the process. Alternatively build it in linux (you can use wsl) where it's not an issue.

@animetiddyenthusiast (and eveyone else invovled).

First I would like to explain why the passthrough mask mode is (still) like this. I've always been intending on doing a complete re-write of the masking shader to do proper chroma/luma keying with user-defined keys & thresholds nothing like it currently is. What has been preventing me from having done this yet is because of a few reasons, nowadays i have limited free-time and I've been prioritizing and focusing on other areas of alxr (or around it).

The other reason is that in order for this to be useful and not frustating is a need for UI with real-time feedback, i personally believe the UI should be done in the client, there are other client-side settings that are just not visible/easy to set. Doing this will take time to do, I'm thinking of adding ImGUI support to the client but will need to make framework to make this too work as 3D UI elements with XR controls

The other alternative is to put the UI in the server dashboard and make some new network packet types to send to the client. It seems like that would be quickier but adding new UI to the server dashboard is headache inducing that I don't want to deal with that (you'll need to deal with hand-writing html/css/js/jquery & rust interop). It will also need to be completely redone once if/when I make the clients compatible with v20 upwards as v20 uses a completely different UI framework now.

As a really short-term solution I can add android system properties that you can set but you'll need to use adb to set them and they will need to be set before the client is started unless I add a thread for polling system property changes.

As for how the masking shader should work there are two ways I would go about doing this (and what you guys should consider doing I think):

A. Convert to LAB colour space and use a version of the delta-e equations to do key comparison.

And/Or

B. Use the YCbCr colour space for comparision, the video texture is typically in a YCbCr format but currently I'm always using a Vulkan YCbCr conversion sampler so either I add a new code path to disable using it or you convert the pixel back to YCbCr colorspace (which seems a bit silly).

In either case you should add some threshold parameters. I recommend looking up using Vulkan specialization constants but I can sort those things out for anyeone.

If you would like to discuss anything about customizing the shader/Vulkan you contact me on discord, you'll find me in the vam server.
Wow, it feels as if we're wandering through a cave and suddenly awakening one of the dragons. That's amazing! Welcome, @bd_energy, to our humble endeavor of deciphering and mastering this. You'll encounter a couple of motivated and curious individuals who are eager to learn the right way. See you all on Discord!
hobbit-smaug.gif
 
Long guide incoming to help @Emosion but for anyone else I have attached an apk of the newest quest build on this post. the master repo has updated for oculus v56 and I have altered the colors slightly on the mask and blend. I have not tested this build yet so use at your own risk, I'll probably update my resource post once I've tested it and I do plan to continue working on it.

I've been reading up on vulkan and openxr and hope to soon put out a version that allows for configuration of the color so we don't have to keep building it. I could just add more shaders to swap through or maybe just have the shader read from a file that we can access in sidequest. the dream would be an in app color picker but I don't know how to do that yet lol. I may also try to improve the blending and scene color detection to prioritize background color better. Tbh I kinda regret doing all this on this alias on a porn site bc if I continue with this I'll probably do a pull request on github to merge it into the main branch which would be a nice thing to share publicly but I guess I can just scrub my posts about it from here once I'm ready for release and just direct people to the github without claiming ownership.


Now for the guide:

follow the steps on github to build for your desired platform in android studio by opening alvr\openxr-client\alxr-engine-sys\android and choosing select build variant. make sure to go into gradle settings and project structure, set the target sdk to 33 using build tools 32 for the variants you want and set java to the visual studio version which is runtime 11. It can be a different version but that one works well and I recommend doing the actual build in vs so it makes sense to use its native java version. actually while we're on that subject I would go in and make sure you have every optional component for vs 22 that doesnt say deprecated.
View attachment 281674View attachment 281673View attachment 281675
Remove all environment variables related to java or android in both windows and studio then specify path to the vs jdk although android studio seems to be able to find it as long as you specify the right version.

Now make sure the repo is properly cloned, you can always redo if you think you broke something. Open the outer folder in VS so you have this structure and open developer powershell. It should actually sync up between vs and as.
View attachment 281663

Now make run the following in powershell to set the android paths doesnt really matter where you run it as long as its in the project

Code:
$Env:ANDROID_SDK_ROOT = "C:\Users\your name\AppData\Local\Android\Sdk"
$Env:ANDROID_NDK_ROOT = "C:\Users\your name\AppData\Local\Android\Sdk\ndk\25.2.9519653"

From the outermost folder run "cargo xtask bump-alxr-versions --nightly" then "cargo update" and "git submodule update --init --recursive" then cd alvr and build dependencies with cargo xtask build-android-deps. once that finishes switch over to android studio (LEAVE BOTH OPEN), confirm your build variant and jdk version again and make any code changes at this point. I personally did the actual code stuff in android studio but it doesnt matter too much which ide you use. Once that's ready, run cargo xtask clean in visual studio, switch back to android studio and build the project. it should finish without any errors. if it gives you an error here then you have a missing dependency and you need to just trace through the log and your steps to fix it. Make sure no java process is running in task manager details tab, then run "cargo xtask build-alxr-quest --release" in visual studio from alvr. it should build at this point if you did it right. it will advance past 252/259 and stall at 255 for a while. once it hits 256 you close java and it should advance and finish. you can find the apk in your build folder in the outermost folder.


With all that said you can also just ask me to build it if you have a request for some specific change as I already have my environment set up to easily push changes
Great Work @animetiddyenthusiast , I was traveling and unable to respond to this thread, and this saved much time that I would use to answer @Emosion. Just like you and other fellow members. It seems I'm developing a kind of second identity where I can enjoy this hobby when I'm not working to survive. I also have the desire to contribute with actual code, but there's a lot I still don't understand. However, given the recent turn of events, I don't think there could be a better scenario or ally than @bd-energy making contact and giving us insight of what's going on. I will review the walkthrough and remove my apk version. I'm leaving you with the APK for publishing, in your thread or this, It would be great to get to a point where we can no longer compile for each change in masks or shaders. I have just one request though: consider using MD5 or a similar method to help users protect themselves.
 
Last edited:
posted an updated build, too tired to say much rn. mdqh with wireless adb makes deployment much less painful since I can stop and start without needing my headset on
 
Convert to LAB colour space and use a version of the delta-e equations to do key comparison.

This method worked amazingly, with a high enough delta-e the outline is mostly gone. My changed code to passthroughMask_frag.glsl is below:

Code:
#include "common/baseVideoFrag.glsl"
#include "common/colorFunctions.glsl"

layout(location = 0) out vec4 FragColor;

vec3 RGB_TO_LAB(vec3 rgb) {
return XYZ_TO_LAB(RGB_TO_XYZ(rgb));
}

void main()
{
vec4 sampleRGB = SampleVideoTexture();
vec3 sampleLAB = RGB_TO_LAB(sampleRGB.rgb);

vec3 darkGreen = vec3(0.01568f, 0.15294f, 0.06274f); // DarkGreen

vec3 maskLAB = RGB_TO_LAB(darkGreen);

float deltaE = LAB_DELTA_E_CIE2000(sampleLAB,maskLAB);

sampleRGB.a = deltaE < 30 ? 0.0f : sampleRGB.a;
FragColor = sampleRGB;
}

common/colorFunctions.glsl is a repo I found that already had a delta2000 equation available github.com/Rachmanin0xFF/GLSL-Color-Functions

ranging from deltaE < 11 to deltaE < 45 worked well enough for me depending on the scene.
 
really good find. I was getting annoyed by the loss in precision and trying to rewrite with a totally different approach but this is much better. that repo also has some other useful stuff I'll dig into later but for now here's updated build with your mask shader. blend mode sucks on this build though, sorry

Appreciate all of the work on this. Looks great now, almost can't see the outline anymore. Is this just for 18.5 or does it work with newer versions?
 
Back
Top Bottom