I also have this problem and I know the solution: When a camera node is attached to the model while making the CUA in Unity, this Camera will be chosen for the Camera rig in VR mode.
As you can see here my camera is locked to this position but the UI/Hands can freely move around. You probably...