Plugins + Scripts VaM_Physical_AI_from_Mimickit

VaM_Physical_AI_from_Mimickit​

Disclaimer: This is an unofficial community project. It is not affiliated with, associated with, or endorsed by Mimickit or MeshedVR.

Demo Video:

NSFW



[Looks :Xspada Kira]
--------------------------------------------------

Experience physical AI trained with Mimickit directly inside Virt-A-Mate VR mode! You can interact with the AI agent in real-time by pushing and grabbing it using your VR controllers.

This package includes a pre-trained physical AI model, allowing you to try it out immediately without doing any training yourself.

GitHub Repository & Source Code:
[https://github.com/yuyaslabo/VaM_Physical_AI_from_Mimickit]

--------------------------------------------------

Key Features
• VR Hand Integration: Streams your VaM VR hand tracking data into the Newton simulator.
• Physical AI Interaction: Enables natural physical behaviors (Pushing & Grabbing).
• Gravity Pull Mechanic: Grab empty space to attract the AI agent's hand towards your controller.

--------------------------------------------------

Hardware & Requirements
This project runs the AI inference via Newton in the background while running Virt-A-Mate.

• Desktop Mode: Works on a single PC. However, physical interactions inside Virt-A-Mate are not available in this mode.
• VR Mode: Running in VR mode requires heavy computing power. It will likely require a separate dedicated PC with an NVIDIA GPU just to handle the AI inference alongside the VaM PC. (Note: A single PC setup with an RTX 4070 Ti was insufficient for VR mode during testing).
• Hardware Compatibility: Developed and tested on Meta Quest 3 OpenXR. Controller tracking on other Head-Mounted Displays (HMDs) has not been verified.

--------------------------------------------------

Setup Instructions

1. Install Mimickit: [https://github.com/xbpeng/MimicKit] (Please make sure to select and configure Newton as the simulator backend during the installation process).
2. Follow the official Mimickit instructions to ensure it runs correctly on your system.
3. Download the latest .zip release from the Releases section of the GitHub repository and extract it. [https://github.com/yuyaslabo/VaM_Physical_AI_from_Mimickit]
4. Copy the contents of the extracted MimicKit folder into your main Mimickit installation directory.
Important: This will overwrite files inside MimicKit/mimickit/envs. Make sure to back up the original files before overwriting.
5. (The pre-trained model checkpoint/model.pt is already included in the zip file, so you do not need to train it yourself).
6. Run the following command based on Start_command.txt:
7. In Virt-A-Mate, load MimickitReceiver.cs as a Person plugin and VRhand_Mimickit2.cs as a Scene plugin.
8. Done! The AI will start moving, and you can now interact with it using your hands.

--------------------------------------------------

Network & Configuration
The data flows in a loop via UDP:
VaM VRhand ➔ 127.0.0.1 UDP 9999 ➔ Newton ➔ 127.0.0.1 UDP 9998 ➔ VaM Bone

• IP addresses and ports are fully configurable.
• If you are using a dual-PC setup, please modify the IP addresses to match your local network environment.
• Virt-A-Mate Side: You can change the VR hand outbound IP/port and the bone inbound port directly from the plugin UI.
• Newton/Mimickit Side: You can change the configurations in data/envs/amp_g1_vam_env.yaml under vam_udp_ip, vam_udp_port, and vrhand_udp_port.

--------------------------------------------------

Call for Testing (Feedback Wanted!)
Since this is an early community release, I would really appreciate your feedback to help with future development!

Please leave a comment below to let me know if it worked or didn't work for you. Please include:
• Your HMD (e.g., Quest 3, Index, etc.)
• Your PC Setup / GPU
• Whether the physics interaction worked smoothly

Thank you for your support!