This method worked amazingly, with a high enough delta-e the outline is mostly gone. My changed code to passthroughMask_frag.glsl is below:
#include "common/baseVideoFrag.glsl"
#include "common/colorFunctions.glsl"
layout(location = 0) out vec4 FragColor;
vec3 RGB_TO_LAB(vec3 rgb) {
return...