Plugins Kinect2VAM - Full body tracking on the cheap

[VeeRifter]

VeeRifter submitted a new resource:

Kinect2VAM - Full body tracking on the cheap - Uses a Microsoft Kinect V2.0 3D scanner to provide full bodytracking data to VAM

View attachment 67916

DESCRIPTION: This is a UDP server/client combination consisting of of a PC based server app which collects and streams real time body joint tracking data from a Microsoft Kinect V2.0 3D sensor, and a VAM client plugin which receives this data and applies it to the Person atom it is attached to.
Pre-requisites are a Kinect V2.0 device fitted with with a PC adapter and its supporting runtime drivers installed on your Windows 10 PC.
PC ADAPTER
[URL...

Read more about this resource...

 

[btacoses]

Here's what I have installed and running: K2EX, SteamVR, VaM current version, scene with atom and plugin on atom. I have gone into the Windows firewall settings and allowed KinectToVAM to have access. I have confirmed in KinectToVR 0.8.0 EX that Kinect Status, Driver Status, and SteamVR Status all show "Success" and I can see the tracking skeleton accurately in KinectToVR. I can't figure out why on the plugin it will not allow for me to connect. It just persistantly shows as Disconnected.

 

[VeeRifter]

Here's what I have installed and running: K2EX, SteamVR, VaM current version, scene with atom and plugin on atom. I have gone into the Windows firewall settings and allowed KinectToVAM to have access. I have confirmed in KinectToVR 0.8.0 EX that Kinect Status, Driver Status, and SteamVR Status all show "Success" and I can see the tracking skeleton accurately in KinectToVR. I can't figure out why on the plugin it will not allow for me to connect. It just persistantly shows as Disconnected.

I assume that you did click on the "Connect" button in the plugins UI ?

 

[btacoses]

I assume that you did click on the "Connect" button in the plugins UI ?

Yes, several times.

 

[VeeRifter]

Yes, several times.

Did you allow Windows firewall to create a rule enabling local traffic ? Try temporarily disabling Windows firewall to see if that fixes the problem.

 

[danou]

Hey! nice plugin, looks very promising

Is it normal that as soon as i activate the plugin on a character the character suddenly respawn on a different place ?

 

[VeeRifter]

Hey! nice plugin, looks very promising

Is it normal that as soon as i activate the plugin on a character the character suddenly respawn on a different place ?

Yes, the Person atom is placed at Unity world position 0,0,0 for calibration purposes. Once calibrated you are free to move the Person atom wherever you like, and the tracking will remain in sync. If you then later hit the reset button, the Person atom will be returned to that same position ready for a new calibration.

 

[Case]

May I pose some noobie questions?

1. I'm a little confused as to which version of the Kinect is needed for this?
   (There's eg "Microsoft Xbox One Kinect V2 Sensor" "Microsoft Xbox One Kinect Sensor", "Xbox One X S 360 Kinect V2", "Xbox one Kinect", Xbox 360 Kinect" ...)
2. Searching on Amazon (GER), the one entry that has a thumbnail that matches the pic in the post for this plugin sez "Out of Stock"? Is this thingy out of production?
3. Listings on EBay are around 120-140€ (140-160$): Is this a fair price? Is this gadget worth it (given that it seems out of production & plenty more modern alternatives available)
4. Any chance for a plugin for different DIY Mocap solutions?

 

[VeeRifter]

May I pose some noobie questions?

1. I'm a little confused as to which version of the Kinect is needed for this?
   (There's eg "Microsoft Xbox One Kinect V2 Sensor" "Microsoft Xbox One Kinect Sensor", "Xbox One X S 360 Kinect V2", "Xbox one Kinect", Xbox 360 Kinect" ...)
2. Searching on Amazon (GER), the one entry that has a thumbnail that matches the pic in the post for this plugin sez "Out of Stock"? Is this thingy out of production?
3. Listings on EBay are around 120-140€ (140-160$): Is this a fair price? Is this gadget worth it (given that it seems out of production & plenty more modern alternatives available)
4. Any chance for a plugin for different DIY Mocap solutions?

1. It's a Kinect V2.0 There's no mistaking thhe difference. There's a picture of it in the overview. It's definitely not the Xbox 360 which was the first one produced.
2. Probably, since the Kinect is no longer made either. I bought the power adapter recently on Amazon.uk for around £25, but your luck may vary.
3. No, I bought my pre-owned Kinect on eBay for £25, but they are no longer in production so I guess I was lucky.
This is more for those who already have one gathering dust somewhere.
4. None, I made this plugin to help someone out and I have no plans whatsoever to do any more mocap stuff (except maybe for the Xbox 360 Kinect)
But even then, that's just a remote chance, as I need a break from all this to get on with some real life stuff.

 

[living1069]

Great plugin, thank you!
I have issues about calibration：
1. What is the indication of a successful calibration?
2. Where is the "snapshot" stored to adjust my alignment? I do not know how to align further.
3. After successful calibration, how can I possess the atom? Any further steps needed?

 

[gt123]

could you update it for kinect V1(X360)?

 

[VeeRifter]

could you update it for kinect V1(X360)?

It's not as simple as an update. These are totally different devices, both hardware and software wise. It would need a complete rewrite of both server and client code. It's unlikely I will be making a 360 version at present. Maybe sometime in future.

 

[VeeRifter]

Great plugin, thank you!
I have issues about calibration：
1. What is the indication of a successful calibration?
2. Where is the "snapshot" stored to adjust my alignment? I do not know how to align further.
3. After successful calibration, how can I possess the atom? Any further steps needed?

1. Your VAM Person precisely tracks your body movements.
2. The 'snapshot' is stored dynamically as an array of offsets. You cannot access this directly.
if you want to fine tune by recalibrating a single or multiple joints, select those only before hitting calibrate button again (do not use the reset button as this will clear all joint offset data)
3. You can possess the Person in the normal manner either before or after calibration.

 

[real virtuality]

is it possible to record tracked movements?
thanks a lot

 

[Stephan]

ist es möglich, getrackte bewegungen aufzuzeichnen?
Danke vielmals
[/ZITIEREN]
Ja, sicher

 

[haloworld]

could you update it for kinect V1(X360)?

It's not as simple as an update. These are totally different devices, both hardware and software wise. It would need a complete rewrite of both server and client code. It's unlikely I will be making a 360 version at present. Maybe sometime in future.

Hi @VeeRifter, I tried to replace the Kinect SDK v2.0 in your UDP server project with v1.8 so that it can basically work for Kinect V1, except for some known problems may need you to resolve
- I don't know how to convert the BoneOrientation object in SDK v1.8 into JointOrientation in v2.0, so I simply applied a method which is obviously wrong
- the client side may also need to be tunned for Kinect V1, as the number of joints tracked by Kinect V1 and V2 is different
- the movement in the VaM is much slower than the skeleton window, not sure if there is anything can be optimized

Attached the modified project, hopefully it can help.

Besides, I noticed that the plugin doesn't work well with Embody plugin, I wonder if you can let Embody plugin control the hands and heads when Kinect2Vam handles the rest of the joints?

 

[Frett]

Thank you for working on this plugin.

I have bought a Kinect V2 to test out the plugin. I have got it working but with some flaws.

I have managed to record an animation with VAM animation but I have to arm every single atom for record each time. Is there no way to arm them all ?

When I tried to record mocap animation with timeline V242 (latest is V244) then I select all needed joints under targets and start record animation.
But this is not working. As soon as record starts the body will nearly freeze or flip around until record is stopped. No movements are recorded.

Seems like timeline is canceling the kinect input when record starts. Can you confirm this ?

What is the best way to record your body movements then please ?

I have tried all this in VAM Desktop mode but would like to use it in VR later.

 

[VeeRifter]

I have managed to record an animation with VAM animation but I have to arm every single atom for record each time. Is there no way to arm them all ?

No, that's the way VAM's built-in animation is set up and you can't change it.

When I tried to record mocap animation with timeline V242 (latest is V244) then I select all needed joints under targets and start record animation.
But this is not working. As soon as record starts the body will nearly freeze or flip around until record is stopped. No movements are recorded.
Seems like timeline is canceling the kinect input when record starts. Can you confirm this ?

Sorry, I cannot confirm your findings. Latest Timeline records mocap OK for me. You have to add all joints as targets and ensure each target is selected (white highlighted) before hitting the Record button. Press Stop to finish, then immediately go to Kinect2VAM UI and uncheck 'Kinect Tracking On' before attempting playback otherwise Timeline will be fighting Kinect for control of joints.

What is the best way to record your body movements then please ?

Personal choice. Whichever method you find easiest.

 

[VeeRifter]

VeeRifter updated Kinect2VAM - Full body tracking on the cheap with a new update entry:

Kinect2VAM v1.1

In order to preserve the integrity of any scene, the Person atom will no longer be teleported to World position 0,0,0 when the plugin is first loaded or calibration is reset. Instead the Person atom will just remain in the same pose and position, wherever they are located in the scene. It is now up to the user to determine how they then pose the Person atom prior to calibration. Obviously, The requirement to physically match that pose prior to calibration still remains.

Read the rest of this update entry...

 

[VeeRifter]

Hi @VeeRifter, I tried to replace the Kinect SDK v2.0 in your UDP server project with v1.8 so that it can basically work for Kinect V1, except for some known problems may need you to resolve
- I don't know how to convert the BoneOrientation object in SDK v1.8 into JointOrientation in v2.0, so I simply applied a method which is obviously wrong
- the client side may also need to be tunned for Kinect V1, as the number of joints tracked by Kinect V1 and V2 is different
- the movement in the VaM is much slower than the skeleton window, not sure if there is anything can be optimized

Attached the modified project, hopefully it can help.

Besides, I noticed that the plugin doesn't work well with Embody plugin, I wonder if you can let Embody plugin control the hands and heads when Kinect2Vam handles the rest of the joints?

Sorry it's taken so long to respond to your post. Many thanks for your input on this. Please don't worry about Bone Orientations, the plugin only uses Position and tracking state data for VAM.
I've adjusted the size of server and client side arrays accordingly to the number of tracked joints, and also the position of each joint in that array.
I've just released a new plugin for the XBox 360 Kinect which performs quite well for me, with mimimal lag, although tracking is not the best. Others may not share the same experience. Depends a lot on the system spec. I suspect.
As regards possession, the plugin (both versions) already checks whether a joint is possesssed, and disables Kinect tracking for that joint if it is.
Mabe you could try using the built-in possession mode, or use AcidBubbles 'Improved POV' plugin instead of 'Embody'.

 

[Jackaroo]

Would someone be willing to post a video of the motion capture ability of this? I'd love to try this to make some short animations, but dont know if i should go ahead and buy a kinect

 

[VeeRifter]

Would someone be willing to post a video of the motion capture ability of this? I'd love to try this to make some short animations, but dont know if i should go ahead and buy a kinect

There's already a demo of the kinect v2.0 motion capture posted on the hub here made by user @Jiraiya.

 

[VeeRifter]

VeeRifter updated Kinect2VAM - Full body tracking on the cheap with a new update entry:

Kinect2VAM v1.2

Registered the "Kinect Tracking On" toggle to make it available to triggers.

Read the rest of this update entry...

 

[eros75008]

Hi @VeeRifter, thanks a lot for your great work, I bought a K2 just for your plugin and its unbelievable. You're talented !!!
I tried a mocap and it feels very good. I tried to, to possess a second Atom to play with the first Atom and it worked pretty well.
I wonder if we can use first view creator with hand collision on VR without addin it to an atom ? just wondering...

 

[VeeRifter]

Hi @VeeRifter, thanks a lot for your great work, I bought a K2 just for your plugin and its unbelievable. You're talented !!!
I tried a mocap and it feels very good. I tried to, to possess a second Atom to play with the first Atom and it worked pretty well.
I wonder if we can use first view creator with hand collision on VR without addin it to an atom ? just wondering...

Thanks very much for your positive comments, much appreciated !
Not sure exactly what you mean in your question though ?