VaM 1.x [POC] Latent Motion Synthesis: Real-time AI Skeleton Driving (VAE + Voxta)

[grimes_AIVRLab]

Hello Creators,
I’m excited to share a project I’ve been working on: a real-time skeleton-driving framework for VaM using a Variational Autoencoder (VAE).
Unlike traditional timeline animations or recorded loops, this system generates per-bone pose data in real-time based on the emotional context of the conversation.

How it works:
Emotion Inference: Using Voxta as the command center to infer emotional states (Excited, Boring, etc.) from the conversation.
Generative Motion: A custom Python hook server receives these states and generates motion data via a VAE model trained on animation data.
Hybrid Control: While the body and limbs are AI-driven, eye gaze, blinking, and basic head motions remain rule-based for stability.

Showcase Video:

Why this approach?
By moving away from pre-recorded timelines, characters can have a "living" presence with natural swaying and emotional transitions that never repeat exactly the same way.
I’d love to hear your thoughts or feedback on this generative approach!

 

[grimes_AIVRLab]

Next Showcase

 

[SubKingMaso]

Wow how extensive does this get? Is she capable of performing femdom actions? Like sounding, fingering etc? and I wonder if it's possible to integrate a multiplayer AI character ran through a server connected to a mocap BonePose where the game itself has connected virtual motion trackers of the figure it's basically like Motion capture except instead of capturing a real person it captures a virtual one running on a separated instance that is processed by the motion capture focusing on specific colored joints, the voice commands triggers them allowing for a more natural and lifelike motion that isn't even animation but rather real time.

Sounds extremely demanding tho but if they could basically create a Virtual Player 2 possessed by AI's motion and voice this would be a seamless and smooth experience I'd tell you.

It would become the offline brand of VR chat

 

[BillTheButcher]

Wow how extensive does this get? Is she capable of performing femdom actions? Like sounding, fingering etc? and I wonder if it's possible to integrate a multiplayer AI character ran through a server connected to a mocap BonePose where the game itself has connected virtual motion trackers of the figure it's basically like Motion capture except instead of capturing a real person it captures a virtual one running on a separated instance that is processed by the motion capture focusing on specific colored joints, the voice commands triggers them allowing for a more natural and lifelike motion that isn't even animation but rather real time.

Sounds extremely demanding tho but if they could basically create a Virtual Player 2 possessed by AI's motion and voice this would be a seamless and smooth experience I'd tell you.

It would become the offline brand of VR chat

It is extremely demanding with one LLM already and voxta, I can't imagine two LLM's running. Maybe one runs local and the other don't. Especially in VR. This guy must have a nasa super computer.