Question Using Kinect V2 for full body tracking.
- Thread starter minilady
- Start date
Jiraiya
Well-known member
- Messages
- 480
- Reactions
- Points
- 93
KinectToVR is free, I was told to get a KinectV1 (xbox 360) rather than the v2, there were lots of reasons but in the end I just got it because it was stupidly cheap. Calibrating it with KinectToVR was fairly easy and it worked well in VaM, sadly you can't turn around and need to remain facing the Kinect for it to work well.
Try out Embody plugin to go with it.
Ok, so I just did this again myself. Embody works REALLY WELL to connect the vive trackers to hips and feet!!!
The kinect setup is really easy (if you follow the calibration guide on the stand in 3 points thing).
Once that was done, I loaded up embody demo and put myself inside the body. Danced. badly. Lots.
then put another person in the scene and walked around them. Put Balance on them and pushed and pulled them around.
Honestly, full body tracking and Embody is AMAZING.
Ok, so I just did this again myself. Embody works REALLY WELL to connect the vive trackers to hips and feet!!!
The kinect setup is really easy (if you follow the calibration guide on the stand in 3 points thing).
Once that was done, I loaded up embody demo and put myself inside the body. Danced. badly. Lots.
then put another person in the scene and walked around them. Put Balance on them and pushed and pulled them around.
Honestly, full body tracking and Embody is AMAZING.
Last edited:
Upvote
0
I think this is the plugin you are looking for. It's VaMotionBridge by Almadiel.
I believe it only works with the Kinect V2 since the v2 API must provide a server app to send skeleton data. Please let us know if it works for you.
I have just been given a 360 Kinect v1 but the v1 API doesn't have a server app, so it looks like I will have to write my own to use it with VAM.
I've tried KinectToVR and while it works, it depends on SteamVR
which I don't want to use, as I can quite happily use VAM with my Oculus Rift-S without adding another unnecessary layer of complexity.
Last edited:
Upvote
0
Jiraiya
Well-known member
- Messages
- 480
- Reactions
- Points
- 93
OOoh, so I wasn't aware VaMotionBridge bypassed the need for SteamVR !
That sounds much better.
Sadly, I too have a Kinect V1. If the V2 works better I might get one as they are fairly cheap.
Have you got links to any documentation that explain how the V2 works differently with VaM? This suggests you don't need KinectToVR ?
That sounds much better.
Sadly, I too have a Kinect V1. If the V2 works better I might get one as they are fairly cheap.
Have you got links to any documentation that explain how the V2 works differently with VaM? This suggests you don't need KinectToVR ?
Upvote
0
That's what I'm hoping, but I can't test VaMotionBridge 'cause I don't have a v2 Kinect.
Upvote
0
Don't forget the PC adapter/power supply then. They are different to the v1.0.
Upvote
0
Jiraiya
Well-known member
- Messages
- 480
- Reactions
- Points
- 93
I am well aware.
If you pull the original cable out (its plugged into a normal USB3-B socket) you can use a normal USB3 cable. Open the KinectV2 up and +12v goes to pin 10 (labelled) and Ground goes to one of the 4 larger surrounding metal posts.
I am just going to solder a wire into it for 12v. I might even make a little adapter to get 12v from the spare PC molex plug on the PSU so I don't need a power brick.
If I could find the socket for the odd connector I could make a little internal PCB for the PC that had a USB3 chip and the power connector and plugged into pcie. That is probably going a bit far though
Upvote
0
Hmm , I just downloaded the Kinect v2 API. Installed it and had a look at what is included. Can't see any sign of a UDP server that could be used to send skeleton data to VAM.
I'm beginning to think that the VaMotionBridge plugin is just one half of what was originally posted by Almadiel. I believe he must have also provided a custom UDP server to run under Windows to send the necessary skeleton data. Not having access to the original download files he posted on Reddit I can still only guess.
I'm beginning to think that the VaMotionBridge plugin is just one half of what was originally posted by Almadiel. I believe he must have also provided a custom UDP server to run under Windows to send the necessary skeleton data. Not having access to the original download files he posted on Reddit I can still only guess.
Upvote
0
Jiraiya
Well-known member
- Messages
- 480
- Reactions
- Points
- 93
That sucks. Well, I still have the V2 on the way whatever.
I can do some comparisons with how well the tracking works between the two at least.
Would be good if you could use 2 at once to get better tracking for limbs obscured from one of them. Sadly, I am pretty sure that doesn't work due to them interfering with each other.
I guess we just appeal to people to see if they have the files and upload them.
I can do some comparisons with how well the tracking works between the two at least.
Would be good if you could use 2 at once to get better tracking for limbs obscured from one of them. Sadly, I am pretty sure that doesn't work due to them interfering with each other.
I guess we just appeal to people to see if they have the files and upload them.
Upvote
0
I haven't given up yet. I think I can borrow a v2 Kinect and if push comes to shove I'll have a go at writing my own UPD server.
Doable, but It could take a long while though, as I will have to research both Kinect API and how to use UPD sockets.
Doable, but It could take a long while though, as I will have to research both Kinect API and how to use UPD sockets.
Upvote
0
Jiraiya
Well-known member
- Messages
- 480
- Reactions
- Points
- 93
Take a look at the Vam Multiplayer plugin That basically does the same thing!
The server side Python script receives and sends the position data for the models. You could probably modify that if you can work out how to grab the kinect data. The real advantage here would be you can FULLY pose the model. Not just hand/feet positions.
You could in theory take the full skeleton from the Kinect and send it ALL to VaM, so elbows and knees etc would go where they really are not just where it's convenient for VaM to get the hand or foot right. Forget a few extra trackers for feet.
You could have almost full body tracking.
That basically does the same thing!The server side Python script receives and sends the position data for the models. You could probably modify that if you can work out how to grab the kinect data. The real advantage here would be you can FULLY pose the model. Not just hand/feet positions.
You could in theory take the full skeleton from the Kinect and send it ALL to VaM, so elbows and knees etc would go where they really are not just where it's convenient for VaM to get the hand or foot right. Forget a few extra trackers for feet.
You could have almost full body tracking.
Upvote
0
Jiraiya
Well-known member
- Messages
- 480
- Reactions
- Points
- 93
The more I think of this, the more just forgetting SteamVR and sending the full skeleton directly into vam sounds amazing.
SteamVR is obviously limited and can't do a lot of things. If you send data direct to VaM all those limits are gone.
You could send full finger position data. So if somebody can find how you extract the hand tracking from the Quest(2) you could directly send that to VaM for full hand/finger tracking.
SteamVR is obviously limited and can't do a lot of things. If you send data direct to VaM all those limits are gone.
You could send full finger position data. So if somebody can find how you extract the hand tracking from the Quest(2) you could directly send that to VaM for full hand/finger tracking.
Upvote
0
Jiraiya
Well-known member
- Messages
- 480
- Reactions
- Points
- 93
Got my Kinect V2. Had to do a little dismantling and mod it to use a normal USB3 cable and 12v input, but it seems to work.
The field of view seems better than the Kinect V1 for a start. The tracking seems better but there is some twitching on the foot (that the kinect To VR people warned about with this version) rotation.
I couldn't see any form of UDP server either, but the SDK Browser has a lot of stuff there.
I am pretty sure if you hook into the Kinect API in the right way it shouldn't matter which sensor you have just to read the skeleton position.
After a very frustrating session of having my knees locked backwards and hip in the wrong place I can't wait to see real full body tracking from this instead of the stupid 3 vive tracker garbage.
The field of view seems better than the Kinect V1 for a start. The tracking seems better but there is some twitching on the foot (that the kinect To VR people warned about with this version) rotation.
I couldn't see any form of UDP server either, but the SDK Browser has a lot of stuff there.
I am pretty sure if you hook into the Kinect API in the right way it shouldn't matter which sensor you have just to read the skeleton position.
After a very frustrating session of having my knees locked backwards and hip in the wrong place I can't wait to see real full body tracking from this instead of the stupid 3 vive tracker garbage.
Upvote
0
Jiraiya
Well-known member
- Messages
- 480
- Reactions
- Points
- 93
So looking around more into tracking, and I found this,
github.com
Opentrack specifically mentions UDP server for position data/tracking.
I wonder if this could help on the PC side? You said specifically that there was a PC UDP server that was missing and I am wondering if it's related.
GitHub - opentrack/opentrack at 13bcebde13ac2d5030a10d213830957275ba81e2
Head tracking software for MS Windows, Linux, and Apple OSX - GitHub - opentrack/opentrack at 13bcebde13ac2d5030a10d213830957275ba81e2
github.com
I wonder if this could help on the PC side? You said specifically that there was a PC UDP server that was missing and I am wondering if it's related.
Upvote
0
Quick update: I have use of a Kinect v2.0 and have already created a Windows app (Kinect2VAM) which has an option to display the active skeleton, and more importantly, it can send the active skeleton data to VAM over UDP. To avoid confusing VAM, I have set it up to only track the first body it sees, and ignore any others that appear later in its field of view. Had to modify the original VAM plugin UDP code, plus various other changes in order to get this to even work at all. However, I have eventually succeeded in getting rudimentary tracking of a Person atom inside VAM. Looking at Almadiels plugin source, I now see it very much as a work in progress, rather than a finished product, and it's going to take a lot more work to get something that's useful. Things like Person scaling, offsets from VAM actual joint positions to those from the Kinect, individual joint selection for tracking, and that's just for starters. I don't intend to publish any more progress reports, but in the meantime I'll DM you with something you can play with to give you an idea of the challenges involved.So looking around more into tracking, and I found this,
Opentrack specifically mentions UDP server for position data/tracking.GitHub - opentrack/opentrack at 13bcebde13ac2d5030a10d213830957275ba81e2
Head tracking software for MS Windows, Linux, and Apple OSX - GitHub - opentrack/opentrack at 13bcebde13ac2d5030a10d213830957275ba81e2
I wonder if this could help on the PC side? You said specifically that there was a PC UDP server that was missing and I am wondering if it's related.
Upvote
0
Jiraiya
Well-known member
- Messages
- 480
- Reactions
- Points
- 93
Yes, there are adapters for both V1 and V2 to use with a PC with no mods. For my liking they cost too much however.
In the end I chose to cut the plug off the V2 cable (it's custom and only any use for the Xbox One OG or the PC kit). I cut a normal USB3 plug off a cheap cable from ebay and soldered the wires together (colour coded and they mostly match). There were 3 wires left over, an extra ground, 12v and 'unused'. I chose to put a bit of extra wire and a PC molex power connector on the end, and plugged it into my PCs internal PSU for 12v. No extra plug/brick/12vpsu etc. It goes on and off with the PC too.
So the kinect V2 itself didn't need opening at all. It just took some time modding the end of the wire.
Of course you can just buy the adapter kit if you are skill/time limited instead of money limited.
First hit on ebay https://www.ebay.com/itm/164400556269
I prefer my solution as there are less wires and plugs but ymmv.
In the end I chose to cut the plug off the V2 cable (it's custom and only any use for the Xbox One OG or the PC kit). I cut a normal USB3 plug off a cheap cable from ebay and soldered the wires together (colour coded and they mostly match). There were 3 wires left over, an extra ground, 12v and 'unused'. I chose to put a bit of extra wire and a PC molex power connector on the end, and plugged it into my PCs internal PSU for 12v. No extra plug/brick/12vpsu etc. It goes on and off with the PC too.
So the kinect V2 itself didn't need opening at all. It just took some time modding the end of the wire.
Of course you can just buy the adapter kit if you are skill/time limited instead of money limited.
First hit on ebay https://www.ebay.com/itm/164400556269
I prefer my solution as there are less wires and plugs but ymmv.
Upvote
0
God
Active member
- Messages
- 57
- Reactions
- Points
- 33
I love you Jiraiya, in a very creepy ackward way, i'll order in a couple weeks and join the mocap party. I've spent 5 months making a huge library if 8K textures with special normals with effects normals normally isnt capable of, resulting in very realistic organic skins i'm releasing this week, and made a morph engine cutting down the time needed to make complex looks to minutes instead of hours\days, all i'm missing is a good way to mocap it all.
Upvote
0
Jiraiya
Well-known member
- Messages
- 480
- Reactions
- Points
- 93
Wait till you hear this.
Another user here, who I was talking to compiled the PC side of the KinectV2 tracking app and got it working with the existing script.
This means, no SteamVR "vive" trackers, a direct to VaM full skeleton tracking.
I was body popping (badly) side kicking, kneeling, sitting, doing lunges. Dancing, the floss, walking around. All points on my body tracked, not just three fake Vive trackers on feet and hips.
It works so well I just spent an hour shittily dancing in front of a virtual mirror. As it wasn't me that compiled it, I can't release it. I am asking him to allow it to be published. It would be nice to have it free on the hub. But basically it turns the Kinect V2 into full body tracking (FULL not 3point, or 7 point, FULL).
The main benefit of this is the full tracking not 3 vive points. However because it doesn't go through SteamVR that means it will work for desktop use too as well as those that use it via other VR means (the Oculus app for example).
Another user here, who I was talking to compiled the PC side of the KinectV2 tracking app and got it working with the existing script.
This means, no SteamVR "vive" trackers, a direct to VaM full skeleton tracking.
I was body popping (badly) side kicking, kneeling, sitting, doing lunges. Dancing, the floss, walking around. All points on my body tracked, not just three fake Vive trackers on feet and hips.
It works so well I just spent an hour shittily dancing in front of a virtual mirror. As it wasn't me that compiled it, I can't release it. I am asking him to allow it to be published. It would be nice to have it free on the hub. But basically it turns the Kinect V2 into full body tracking (FULL not 3point, or 7 point, FULL).
The main benefit of this is the full tracking not 3 vive points. However because it doesn't go through SteamVR that means it will work for desktop use too as well as those that use it via other VR means (the Oculus app for example).
Upvote
0
Jiraiya
Well-known member
- Messages
- 480
- Reactions
- Points
- 93
Oh god. You still use SteamVR?!
(Unless you are using the AMD Fidelity plugin, STOP using SteamVR for your own sanity).
There is a thread about launching VaM directly through Virtual Desktop which I do and does NOT invoke the SteamVR Frame stealing demon that spits in your eyes.
Honestly. Try it. It's just a batch file that passes the vam.exe to the Virtual Desktop client. The game launches quicker, loads quicker, gets better framerates and doesn't get the horrible stutter and jitter that made me want to vomit.
Oh, yes, this will work with SteamVR or any other way you run VaM because it's totally independent of anything other than VaM. The Kinect client on the PC is a standalone .exe and the data goes into VaM via a script directly to the character skeleton. You CAN'T do this via SteamVR. That only supports Vive trackers are are horrible to setup and use in VaM. So this bypasses all of it
Yes, You guessed it was VeeRifter. Well, if he is happy to distribute one of us will upload. I can't/won't do anything without his permission though as he compiled it.
(Unless you are using the AMD Fidelity plugin, STOP using SteamVR for your own sanity).
There is a thread about launching VaM directly through Virtual Desktop which I do and does NOT invoke the SteamVR Frame stealing demon that spits in your eyes.
Honestly. Try it. It's just a batch file that passes the vam.exe to the Virtual Desktop client. The game launches quicker, loads quicker, gets better framerates and doesn't get the horrible stutter and jitter that made me want to vomit.
Oh, yes, this will work with SteamVR or any other way you run VaM because it's totally independent of anything other than VaM. The Kinect client on the PC is a standalone .exe and the data goes into VaM via a script directly to the character skeleton. You CAN'T do this via SteamVR. That only supports Vive trackers are are horrible to setup and use in VaM. So this bypasses all of it
Yes, You guessed it was VeeRifter. Well, if he is happy to distribute one of us will upload. I can't/won't do anything without his permission though as he compiled it.
Upvote
0
Similar threads
- Question
- Question
- Question