• Hello Guest!

    We have recently updated our Site Policies regarding the use of Non Commercial content within Paid Content posts. Please read the new policy here.

    An offical announcement about this new policy can be read on our Discord.

    ~The VaMHub Moderation Team
  • Hello Guest!

    We posted an announcment regarding upcoming changes to Paid Content submissions.

    Please see this thread for more information.

Resource icon

Plugins HeadTracker v0.4

This is the Head Tracking plugin. It moves camera according to your head moves.

This plugin only manages camera: you turn your head to the left, VaM's camera turns to the left and image on your display runs to the right (so you see what was behind the left side of your monitor). Same with other directions. Eyes are not tracked, just head.
To have a quick impression, take a look on how similar headtracking is used in flight simulators: the first minute of this video
gives a good idea how does such tracking works. Notice how game image moves to the sides when the guy turns his head. Note the guy uses different software and hardware. (that is a random video from youtube I found searching for "flight simulator head tracking". I am not affiliated with that person or channel)

I have to warn you it is shaky [october 2020]. I have some ideas how to fix this, but those are not implemented yet.

First, you set up a webcam either on top or under your monitor, then you run HeadTracker6DoF.exe , which is a head pose sensor (sources are available on my github). Next, you add this plugin.

downloads page at github: https://github.com/Eugene-E0a80fd8080ff8e/HeadTracker6DoF/releases
project sources and docs: https://github.com/Eugene-E0a80fd8080ff8e/HeadTracker6DoF

You may use F2 button to change middle point. Move camera with your head to something what should become a new middle point, hold F2 while returnng your head to a normal position. Camera stays still while you holding F2.

You should add HeadTracker.cs as a session plugin.

To run HeadTracker6DoF.exe , you need to download either from VaM-hub or from my github. The file would be HeadTracker6DoF.zip . Unzip it and run the exe file.

Note it works nicely with Passenger plugin from acidbubbles.

HeadTracker6DoF is based on dlib 68-point head landmarks detector. It also uses OpenCV.

version v0.3 adds support of opentrack -- the software which enables similar functionality in flight simulators.
Author
Eugene-E0a80fd8080ff8e
Downloads
5,571
Views
8,103
First release
Last update
Rating
4.50 star(s) 4 ratings

Latest updates

  1. v0.4 -- new version is compatible with VaM 1.20.77.6

    - new version is compatible with VaM 1.20.77.6 - new UI to change camera resolution and other...
  2. v0.3 — major bugfixes and opentrack support

    Major bug has been fixed. This should ease shaking. If you have adjusted deadband, dampings and...
  3. v0.2.1

    You can now select resolution for a camera. Higher resolution should help with face detection...

Latest reviews

This is definitely a step in the right direction. Using SmoothTrack on my iPhone and OpenTrack on the PC it works a charm. Would be nice to zoom in and out. Maybe it can just I haven't sorted it yet. Nice job.
Upvote 0
Thank you for another nice plugin for people who use VaM in desktop mode. VR is more immersive but I would not want to use it for too long.

My vision of a perfect use in desktop mode is something like:

I'm sitting in front of a monitor using my head to track an atom in 6DoF. My hands are also tracked using Leap Motion. All of this is happening in perfect sync with no lag. It would be like sitting in front of a real mini 3d world and not a flat monitor, and my hands are moving inside it...

Well, I can dream, can't I? Haha.

Btw, how do I move the Z-axis with this plugin? (as in, moving my head closer and further from the camera should make the screen zoom in and out)
Eugene-E0a80fd8080ff8e
Eugene-E0a80fd8080ff8e
Do you mean something like this : https://www.youtube.com/watch?v=91tYEgpmN4M ?
I have tried to do similar in VaM and it does not look any immersive without that mesh forming a well. You need a specially designed scene to have that effect :(

As of now, Z-axis detection is very imprecise. Relying on it causes very unpleasant in-and-out shaking. In my experiments I had to override with z=0.8 meters.
The HeadTracking6DoF.exe sensor actually detects and transmits both head rotation and position relative to camera.
Upvote 0
great!
Upvote 0
Don't get me wrong - I really dig your work, and I'm beyond happy someone in the VaM community is paving the way in terms of DIY-(body)tracking, but ... I really liked your PhoneController app, and I was hoping you'd further explore the possible improvements you mentioned when you published it? (IIRC, you pondered using some custom libraries (SENSORS_6DOF?) to help with drift correction).

For background: I'd wager many people in the community can manage to save up the dough for a Quest or similar at some point, but there's only very, very few peeps that can afford dedicated commercial fullbody-tracking solutions. Afaik, it's rare even amongst the semi-professional scene creators.

Granted, there's Driver4VR, which offers various DIY-solutions for lowerbody-tracking - but my impression is that they're more focussed on precise foot-tracking for action games & VRchat? (I haven't tried it yet, but I would wager that, say, a kinnect would have trouble correctly tracking at least one coordinate of the hip, since the sensors all face in one direction)

TL;DR - I think there'd be a lot of demand in the VaM community for a good, reasonably accurate DIY-hiptracking solution. Full lowerbody-tracking would be more awesome yet, but I think the hiptracking is going to be most critical, since so much of animation & posture is influenced by that atom/controller.
Eugene-E0a80fd8080ff8e
Eugene-E0a80fd8080ff8e
Although this might seems like a step towards body tracking, it is not (yet).
This plugin only manages camera: you turn your head to the left, VaM's camera turns to the left and image on your display runs to the right (so you see what was behind the left side of your monitor). You have to keep your eyes on the monitor. Eyes are not tracked, just head.

This is similar to what MS Flight Simulator's people use. Search youtube for "FaceTrackNOIR" -- there a lot of videos for this application of headtrackng.

For hiptracking :) , this seems to be easy task -- just slap an ArUco marker on your butt, and you good to go. Probably copying an example snippet from OpenCV docs would be enough for PoC.
Upvote 0
Back
Top Bottom