So, I got a Vive Facial Tracker and decided to make it work in VAM.
Here's the result!
It's not perfect but can provide extra emotion for characters in lewd or non-lewd scenes.
There is more work to do mapping the tracker face weightings to VAM morphs. Right now the effect still looks kinda robotic and glitchy but I think this can be improved. The plugin also needs a UI to allow users to tweak settings. This is still at an alpha release stage.
One challenge is that the Facial Tracker just tracks mouth, lip and tongue movement. Eyes and brows remain static, which makes characters look like creepy dolls and limits how expressive you can be. HTC sells an eye tracking module which might help solve this, but it only works on Vive Pro and I don't have one.
A solution is to use AcidBubbles' Glance plugin to animate eye morphs and make your character more alive.
We can't yet record realtime morph changes in VAM, so for now your face movements won't be saved in the scene file.
Instructions
1. Download SRanipal from HTC's website and install it (this is the facial tracker driver)
https://dl.vive.com/SRSDK/runtime/VIVE_SRanipalInstaller.msi
2. Make sure you have all required VAM dependencies:
- Jackroo Mocap Based Expressions
- WeebU morphs (used for cheek puff/suck)
If you are missing required morphs, you will get an error message but the plugin should still work. You'll just have less facial movement.
RIght now, male characters won't have cheek puff/suck or pout as I didn't find any male morphs and the female ones couldn't be found by the plugin when attached to a male Person atom (maybe I'm doing something wrong).
3. Download both files from this page:
- FacialTrackerVamPlugin.var - this is the VAM plugin, attach its FacialTrackerVamPlugin.cslist to a Person atom
- FacialTrackerVamLink.zip - extract these files anywhere and run FacialTrackerVamLink.exe to connect your tracker to VAM
Why do we need to run a standalone .exe file? HTC requires us to use their own DLLs (code libraries) to get tracker data. However VAM forbids use of unmanaged DLLs in plugins. FacialTrackerVamLink.exe sends tracker data as messages thru a UDP socket and these are read by the VAM plugin.
Source code is provided in the .zip so you can review and compile it yourself if wanted.
Suggestions, forks and bug reports welcome!
Troubleshooting tips
- Your firewall should allow VAM.exe and FacialTrackerVamLink.exe, at least for localhost.
- Make sure the SRanipal driver is installed and working. HTC's Vive Developer site has a sample Unity app in their SDK download that you can use to test your tracker and make sure it's working.
- Try unplugging your tracker and plugging it back in again
- Enable debug output in FacialTrackerVamLink.exe by setting SHOW_OUTPUT=true in config.ini. This will show the raw weightings from the tracker. If they are all 0.0 and don't change, you may need to adjust the tracker position... or unplug/replug it..
Here's the result!
It's not perfect but can provide extra emotion for characters in lewd or non-lewd scenes.
There is more work to do mapping the tracker face weightings to VAM morphs. Right now the effect still looks kinda robotic and glitchy but I think this can be improved. The plugin also needs a UI to allow users to tweak settings. This is still at an alpha release stage.
One challenge is that the Facial Tracker just tracks mouth, lip and tongue movement. Eyes and brows remain static, which makes characters look like creepy dolls and limits how expressive you can be. HTC sells an eye tracking module which might help solve this, but it only works on Vive Pro and I don't have one.
A solution is to use AcidBubbles' Glance plugin to animate eye morphs and make your character more alive.
We can't yet record realtime morph changes in VAM, so for now your face movements won't be saved in the scene file.
Instructions
1. Download SRanipal from HTC's website and install it (this is the facial tracker driver)
https://dl.vive.com/SRSDK/runtime/VIVE_SRanipalInstaller.msi
2. Make sure you have all required VAM dependencies:
- Jackroo Mocap Based Expressions
- WeebU morphs (used for cheek puff/suck)
If you are missing required morphs, you will get an error message but the plugin should still work. You'll just have less facial movement.
RIght now, male characters won't have cheek puff/suck or pout as I didn't find any male morphs and the female ones couldn't be found by the plugin when attached to a male Person atom (maybe I'm doing something wrong).
3. Download both files from this page:
- FacialTrackerVamPlugin.var - this is the VAM plugin, attach its FacialTrackerVamPlugin.cslist to a Person atom
- FacialTrackerVamLink.zip - extract these files anywhere and run FacialTrackerVamLink.exe to connect your tracker to VAM
Why do we need to run a standalone .exe file? HTC requires us to use their own DLLs (code libraries) to get tracker data. However VAM forbids use of unmanaged DLLs in plugins. FacialTrackerVamLink.exe sends tracker data as messages thru a UDP socket and these are read by the VAM plugin.
Source code is provided in the .zip so you can review and compile it yourself if wanted.
Suggestions, forks and bug reports welcome!
Troubleshooting tips
- Your firewall should allow VAM.exe and FacialTrackerVamLink.exe, at least for localhost.
- Make sure the SRanipal driver is installed and working. HTC's Vive Developer site has a sample Unity app in their SDK download that you can use to test your tracker and make sure it's working.
- Try unplugging your tracker and plugging it back in again
- Enable debug output in FacialTrackerVamLink.exe by setting SHOW_OUTPUT=true in config.ini. This will show the raw weightings from the tracker. If they are all 0.0 and don't change, you may need to adjust the tracker position... or unplug/replug it..
