Plugins Kinect2VAM - Full body tracking on the cheap

[MetroSlider]

This is a fantastic plugin, I literally got a Kinect 2.0 to try this out and its really well done!

I'm a bit confused about the speed of the motion capture. I've got it working nicely but no matter how much I try adjusting the Max velocity, Dampening and Spring in the 'Control & Physics 2' section, I can't get it to stop being sloppy movement. What am I missing? What specific options do I need to change, and where are they located? I'm sure this is probably a simple thing to remedy, but I've only been using VaM for a month so I'm still figuring some stuff out.

 

[VeeRifter]

This is a fantastic plugin, I literally got a Kinect 2.0 to try this out and its really well done!

I'm a bit confused about the speed of the motion capture. I've got it working nicely but no matter how much I try adjusting the Max velocity, Dampening and Spring in the 'Control & Physics 2' section, I can't get it to stop being sloppy movement. What am I missing? What specific options do I need to change, and where are they located? I'm sure this is probably a simple thing to remedy, but I've only been using VaM for a month so I'm still figuring some stuff out.

There is some automatic adjustment of joint hold position spring and damper settings in the plugin which is derived from the live tracking state data being sent from the Kinect. This is to prevent jitter if a joint is not correctly tracked at that particular instant in time.
Live joint tracking data states being sent from the Kinect are: State 2 = Tracked, State 1 = Inferred, State 0 = NotTracked
If the joint state is 'Tracked' the spring value is set to its default of 2000 and damper to its default of 35.
If the joint state is 'Inferred' the spring value is lowered to 1000 and damper increased to 70.
If the joint state is 'NotTracked' the spring value is further lowered to 500 and damper increased to 100.
So as tracking accuracy deteriorates the spring and the damper values are modified to mitigate any erratic movement.
These adjustment sliders (and several others) can be found under the 'Physics' tab when an individual joint, rather than the 'Person' atom, is selected.

 

[MetroSlider]

There is some automatic adjustment of joint hold position spring and damper settings in the plugin which is derived from the live tracking state data being sent from the Kinect. This is to prevent jitter if a joint is not correctly tracked at that particular instant in time.
Live joint tracking data states being sent from the Kinect are: State 2 = Tracked, State 1 = Inferred, State 0 = NotTracked
If the joint state is 'Tracked' the spring value is set to its default of 2000 and damper to its default of 35.
If the joint state is 'Inferred' the spring value is lowered to 1000 and damper increased to 70.
If the joint state is 'NotTracked' the spring value is further lowered to 500 and damper increased to 100.
So as tracking accuracy deteriorates the spring and the damper values are modified to mitigate any erratic movement.
These adjustment sliders (and several others) can be found under the 'Physics' tab when an individual joint, rather than the 'Person' atom, is selected.

So you're saying that the plugin makes the adjustments so I shouldn't have to change them manually? Because the motion tracking on the skeleton is perfect, but in vam everything is more sluggish movement that isn't nearly as quick and responsive as the example/test by Jiraiya that I found on the VaM Hub. I'm just trying to figure out what I'm doing wrong because I've tried modifying the individual settings in the Physics tabs of the joints and nothing changes. Even after editing the physics of the joints after doing the calibration on the plugin they still aren't snappy.

 

[VeeRifter]

So you're saying that the plugin makes the adjustments so I shouldn't have to change them manually? Because the motion tracking on the skeleton is perfect, but in vam everything is more sluggish movement that isn't nearly as quick and responsive as the example/test by Jiraiya that I found on the VaM Hub. I'm just trying to figure out what I'm doing wrong because I've tried modifying the individual settings in the Physics tabs of the joints and nothing changes. Even after editing the physics of the joints after doing the calibration on the plugin they still aren't snappy.

Ah, I understand where you are coming from now. What you are talking about here is latency.
In an ideal situation the movement of the skeleton as observed at the Kinect server would be immediately reflected by movement of the Person in the VAM client.
The reality is that it takes a finite amount of time for the server to collect all this joint data, send it via UDP to VAM, which then has to read it back and convert it to movement in a format compatible with Unity joint positions. This is the latency time, and it very much depends on the spec. of your rig as to how long this lag is. The better the spec. the lower the latency.

 

[MetroSlider]

Sorry for the confusion but, its definitely not latency as far as I can tell. It moves when I move. I have an 8700k, 32GB of RAM and a 1080Ti so I don't think my rig is the problem. Let me show you an example of what I mean.

https://imgur.com/a/QJCsTxj

(2 slides on this, one of Jiraiya's scene and one of mine with the skeleton view on the right)

When you look at the skeleton, you can see its quick and snappy. But in VaM its like she's just flopping her hands around. That's why I thought it was a problem with the Spring or Dampening settings. Is this how it's supposed to look? If you look at the example/test by Jiraiya that I found on the VaM Hub the movement is much snappier, rather than flopping around as if the Dampening is too high.

In the description of Jiraiya's Scene they've said "I made this using a Quest2 headset which tracks hand and head position, however almost the entire body can be tracked with the Kinect." So I assume that will help with the hand movement, but the rest of the motion seems to look snappier than in my example, again, leading me to think its the spring or dampening settings or something. I figured that although the Quest2 setup helps with the orientation of the hands, even the kinect skeleton shows quick and snappy movement for the wrists and the hand direction so it shouldn't be this floppy?

But if you tell me that my example is actually how its supposed to look, then that's fine. The last thing I want to do is waste your time on this. Its still a fantastic plugin.

 

[VeeRifter]

Sorry for the confusion but, its definitely not latency as far as I can tell. It moves when I move. I have an 8700k, 32GB of RAM and a 1080Ti so I don't think my rig is the problem. Let me show you an example of what I mean.

https://imgur.com/a/QJCsTxj

(2 slides on this, one of Jiraiya's scene and one of mine with the skeleton view on the right)

When you look at the skeleton, you can see its quick and snappy. But in VaM its like she's just flopping her hands around. That's why I thought it was a problem with the Spring or Dampening settings. Is this how it's supposed to look? If you look at the example/test by Jiraiya that I found on the VaM Hub the movement is much snappier, rather than flopping around as if the Dampening is too high.

In the description of Jiraiya's Scene they've said "I made this using a Quest2 headset which tracks hand and head position, however almost the entire body can be tracked with the Kinect." So I assume that will help with the hand movement, but the rest of the motion seems to look snappier than in my example, again, leading me to think its the spring or dampening settings or something. I figured that although the Quest2 setup helps with the orientation of the hands, even the kinect skeleton shows quick and snappy movement for the wrists and the hand direction so it shouldn't be this floppy?

But if you tell me that my example is actually how its supposed to look, then that's fine. The last thing I want to do is waste your time on this. Its still a fantastic plugin.

I believe that Jiraiya used possession mode (either the built-in one or one of acidbubbles plugins) along with Kinect2VAM to make that capture.
That's why the hand movements are snappier in that demo (Kinect only provides tracking for wrists. Hands are leaf nodes not joints). To try it yourself just possess the person before hitting the calibrate button.

 

[MetroSlider]

I believe that Jiraiya used possession mode (either the built-in one or one of acidbubbles plugins) along with Kinect2VAM to make that capture.
That's why the hand movements are snappier in that demo (Kinect only provides tracking for wrists. Hands are leaf nodes not joints). To try it yourself just possess the person before hitting the calibrate button.

Unfortunately I tried the embody plugin with no improvements. I tried various settings, but still had the floppy hands. Thanks for your quick responses though, I appreciate it.

 

[VeeRifter]

VeeRifter updated Kinect2VAM - Full body tracking on the cheap with a new update entry:

Kinect2VAM v1.3

Fixes a bug that was disabling any active Possession mode when the 'Select None' button was clicked.

Read the rest of this update entry...

 

[VeeRifter]

Unfortunately I tried the embody plugin with no improvements. I tried various settings, but still had the floppy hands. Thanks for your quick responses though, I appreciate it.

I've just posted an update that fixes a possession bug. That may have been what was causing your floppy hands issue if you had used the 'Select None' button whilst possession mode was active.

 

[Trabant Dave]

Hi VeeRifter, I've downloaded the plug-in and associated files and installed as per, but the server does not connect; I use a third-party firewall Avast, and am struggling to set up the firewall rule - the Kinect scans when using KScan software so I can only assume it's my inability to set up the firewall; can you advise?

 

[VeeRifter]

Hi VeeRifter, I've downloaded the plug-in and associated files and installed as per, but the server does not connect; I use a third-party firewall Avast, and am struggling to set up the firewall rule - the Kinect scans when using KScan software so I can only assume it's my inability to set up the firewall; can you advise?

The Kinect2VAM.exe server app should still display the active skeleton even when the VAM client is not connected.
First check that the Kinect device lights up when you start the Kinect2VAM.exe server. Then click on the "Show Skeleton" button and stand in front of the sensor and move around. You should see your movement reflected in the skeletons movement and status bar should say "Body tracking active. VAM not connected".
If you get this far then there is nothing wrong with the server. So, if you cannot get the VAM client to connect then you can be confident that it's your firewall that's blocking it. To prove it, try temporarily disabling the Avast firewall and see if it connects after that. (You can physically disconnect your PC's network adapter from the Internet whilst you are doing this if you are concerned about malware access).
If it does then connect you will must set up a rule in Avast to let the server/client traffic through.
Communication protocol is UDP and the local network port it uses is 6969.
I don't use Avast so I have no idea how to go about doing this. You will need to research for yourself on how to do this.

 

[Sinful_Rose]

Hi, can I use the server component on a pc and the client on a different pc with VAM installed? Can they talk each other on network? I use the Quest 2 remotely from my rig pc which is in a different room from my play space. So I could use a laptop connected with the kinect to run the server component and the rig to run VAM and the client. Will it work?
Thanks

 

[VeeRifter]

Hi, can I use the server component on a pc and the client on a different pc with VAM installed? Can they talk each other on network? I use the Quest 2 remotely from my rig pc which is in a different room from my play space. So I could use a laptop connected with the kinect to run the server component and the rig to run VAM and the client. Will it work?
Thanks

No

 

[Slam Thunderhide]

Hello,
That link you posted for the adaptor is for the 360 version and is unavailable. Can you give me the model number or an alternative to getting this connected to my computer?

 

[VeeRifter]

The link I provided in the overview was for the correct adapter for the Kinect V2. There is also a link on that same Amazon page (in the top left) for a 360 adapter, so I think that's what you were reading. However, you are correct in saying that it is now unavailable on Amazon.
Here's one for sale on EBay: ORIGINAL MICROSOFT XBOX KINECT FOR WINDOWS HUB POWER SUPPLY CABLE MODEL NO. 1637
I believe there's also another Chinese knockoff manufactured under model No.1806.

 

[Magiwarriorx]

I'm having significant IR interference from the Kinect. Index/Knuckles + 1.0 base stations. I've tried the optical sync cable and moving the stations out of view of the Kinect; still get substantial desync and white outs. Any advice?

 

[VeeRifter]

I'm having significant IR interference from the Kinect. Index/Knuckles + 1.0 base stations. I've tried the optical sync cable and moving the stations out of view of the Kinect; still get substantial desync and white outs. Any advice?

Sorry, I can't help you on this one. I have an Oculus Rift S which has inside-out tracking not utilising IR.

 

[jillian]

Apologies in advance for total newbie problem/question.
I have the v.2 sensor from eBay UK and bought the US version of the adaptor connector (I think this is the correct adaptor Xbox Kinect Adapter for Xbox One S/Xbox One X Windows 8/8.1/10 Power AC Adapter PC Development Kit ). I tried downloading the Microsoft app (
Kinect for Windows Runtime 2.2.1905) with the drivers but I get "USB device not recognized".
Do I have the wrong adaptor connector? Or am I not installing the drivers correctly? Or do I have a bad sensor?
Any help will be much appreciated.

 

[jillian]

Also confusing is how to load the drivers. If I follow your link to the Microsoft download instructions for download are to
"Right click kinectsensor.inf and click Install. Do not run KinectRuntime-x64.msi directly."
I don't see the Kinect sensor.inf - so I ran the x64 directly. Your instruction was to run the app. Maybe I have not installed the drivers correctly?

 

[jillian]

I think the problem is the adaptor I bought on Amazon. I think it may be for v.1 - I can't find an equivalent to the UK powered one you show. Can you help with what and where to get the correct one please?

 

[VeeRifter]

I think the problem is the adaptor I bought on Amazon. I think it may be for v.1 - I can't find an equivalent to the UK powered one you show. Can you help with what and where to get the correct one please?

The adapter you have appears to be the correct one if you are in the US. However the driver needs to be installed exactly as per Microsoft instructions.
Are you sure you extracted it, rather than just opening the zip in explorer ?
Extract the contents of "KinectRuntime-v2.2_1905.zip" to somewhere on your PC i.e. Desktop. You should then have a folder named "drivers" which contains a sub folder named "x64". The contents of this folder are shown in the attached screenshot. "kinectsensor.inf" is the fourth file from the top.
(You may have to change Windows Explorer View settings to show filename extensions if you have the "Hide extensions for known file types" option checked.)
Right click on "kinectsensor.inf" and select "Install" from the drop down menu.

 

[jillian]

Thank you for your kind prompt reply. I was incorrectly trying to install from the zip and having extracted found the kinectsensor.inf which I ran from install.
I still get USB device not recognized.
I also ran the v2 SDK 2.0 and will uninstall that.
Can you think of anything else that could be preventing recognition?
Many thanks again.

 

[jillian]

I have followed the instructions and still get USB device not recognised - could it be because I'm connecting to a USB 3 hub and not the sole USB hub on my desktop? Is the kinectsensor.inf the only part of the v2.2_1905 that I should use or should I also rum the Kinect runtime installer?

 

[jillian]

The adaptor I have now makes me suspicious that it's for the version one as I notice in the one page manual it calls for "ver 2.0 1409 setup.exe"

I have another one arriving from eBay today that claims to be v2. I'll try that and see if it gets recognised. Thanks for your help and suggestions.

 

[jillian]

I have the new adaptor and followed instructions again but still USB device not recognised. Here is the 2.0 browser view attached

. It says Kinect running but USB defintion a problem.

Any suggestion as to what I may have done wrong?