Plugins Vive Facial Tracker Plugin

[mektar]

mektar submitted a new resource:

Vive Facial Tracker Plugin - Use the Vive Facial Tracker in VAM

So, I got a Vive Facial Tracker and decided to make it work in VAM.

Here's the result!

It's not perfect but can provide extra emotion for characters in lewd or non-lewd scenes.

There is more work to do mapping the tracker face weightings to VAM morphs. Right now the effect still looks kinda robotic and glitchy but I think this can be improved. The plugin also needs a UI to allow users to tweak settings. This is still at an alpha release stage...

Read more about this resource...

 

[VirtAmateur]

This looks very promising. LFE found a way to integrate his facial motion capture with Timeline for recording, you could try asking how he did it or collaborating https://hub.virtamate.com/resources/facial-motion-capture.328/

I'd wish the guy reporting a virus gave more details, but a quick virustotal scan found nothing on the FacialTrackerVamLink.zip.

 

[mektar]

Oh cool. Looks like LFE's plugin saves face data when new frames are received from the companion iPhone app, then exports it to a json file readable by Timeline.

https://github.com/lfe999/FacialMotionCapture/blob/master/src/Controllers/RecordingController.cs

I may try to borrow this idea . It should be simpler to implement in my plugin as I'm only recording morph weights, not bone movement.

The .exe shouldn't trigger any virus warnings as it just sends messages on localhost. Running random exes from the internet is usually a Bad Idea so I get the concern.

If Meshed could include the facial tracker dlls in a future release then that might solve the problem, but at that point he would probably build his own integration (which would be great!)

 

[Acid Bubbles]

This indeed looks very cool! (I don't have a Vive so I can't try it). The next version of Timeline will allow live recording of multiple morphs or float params (see

for an example with fingers) so that'll complement this very well! Then there would just be some eye morphs to animate manually

 

[mektar]

That's awesome. Problem solved!

Although the tracker is branded "Vive", it works with any VR headset (or even on desktop). You have to 3D print your own mount though, there are some on Thingiverse for Index and probably other headsets. I need to figure out a good way to mount the tracker and a Leap Motion at the same time.

https://www.vive.com/us/accessory/facial-tracker/

 

[SupaRioAmateur]

very interesting!

This looks very promising. LFE found a way to integrate his facial motion capture with Timeline for recording, you could try asking how he did it or collaborating https://hub.virtamate.com/resources/facial-motion-capture.328/

I'd wish the guy reporting a virus gave more details, but a quick virustotal scan found nothing on the FacialTrackerVamLink.zip.

the ios facial tracking has one major drawback which only detects tongue_out but not the movement.
this looks very cool with the tongue movements. great work @mektar

 

[Animalone]

Hey, i tried the Facial Tracker Plugin but the Face of the Person just explodes XD
what can i do ?

 

[Nawen]

mektar submitted a new resource:

Vive Facial Tracker Plugin - Use the Vive Facial Tracker in VAM



Read more about this resource...

Nooooooo , the Vive Pro Eye tracking, so close..... just one last thing and VAM Adult Vtuber'ing will be possible xD, please someone make this last thing happen.. I know it all miracles and you guys are amazing as is. But just ooone last miracle xD

 

[cyber_daddy]

i put in multipliers for the weights to fix the explosion that happened for me as well. i also added support for lowering your jaw with a closed mouth(quite subtle) and rolling in your lips.

 

[lizsfx2016]

Hi, don't know if there's anyone that can help out with this but I'll try here anyways. Just installed my new Vive Facial Tracker onto my Vive Pro 2. I have this bizarre problem where the "data/video feed" coming from the facial tracker is extremely slow/laggy to the point where it appears to be dropping frames which in turn is causing facial tracking to look awful and unsmooth.

Trying it with FaceGym, the program that comes with the Vive Tracking SDK showcases the issue which is happening with VAM. The fact that it's happening with other programs tells me it may not be a VAM/Plugin issue. Any ideas on what could be causing the problem? I've tried everything including different USB3 ports on my PC but it's still being problematic. I can showcase this with a video if it's any help. I can't find anything online regarding this.

 

[cyber_daddy]

i would still guess a usb issue. maybe its the cable but more likely a compatibility issue. try a different pc or laptop. if it works there then try a bios upgrade. if that doesnt solve it might be your os. try linux or windows on an usb drive to see if it changes something

 

[Nawen]

its ONLY the facial tracker? everything else runs smooth but the tracker is being slow? . if so it could be the USB or some kind of Security Software maybe?

 

[lizsfx2016]

its ONLY the facial tracker? everything else runs smooth but the tracker is being slow? . if so it could be the USB or some kind of Security Software maybe?

Literally everything else is butter smooth. (The headset, Valve Index Controllers and the 3 Vive Trackers). On the facial tracker, it basically drops frames and looks like it's running at 3-5 FPS. I do know underlying base of my machine (Mobo/CPU Core i7 980x) is somewhat long in the tooth at 10-11 years old and is in dire need of replacement.

 

[H3]

I'm waiting for you to resume developing this plugin

 

[Chrisbo8000]

Has anyone got the facial tracker to record in Timeline?

 

[ngreenex]

You can record timeline even with android cam face recognition - https://hub.virtamate.com/resources/andriodfacemocap（安卓手机面部捕捉）.27531/
So now we have a proof that recording with timeline is possible. Somebody, please make an update for this vive tracker plugin to make timeline recording possible. I am ready to pay for this.

 

[Infinite_limit]

I'm having issues right now with it. It stops working after a few minutes. Haven't figured out the cause yet.