Thursday, 24 February 2011

Creating SL Animations using the Kinect

I have been experimenting of late with the Xbox Kinect as a cheap Mocap source to generate BioVision Hierarchy (BVH) files for upload into Second Life, and the results are pretty encouraging.

What you Need

Hardware
A standalone Kinect, with power supply (this comes as standard when purchased as a standalone), if you buy it as part of an Xbox bundle you will need to buy an additional cable sold separately to connect it to the PC. I bought my Kinect from a local gaming store for Euro139.

Kinect, showing its USB connector and power adaptor
Software
Brekel Kinect 3D Scanner
OpenNI
SensorKinect Drivers
NITE User Tracking Module
Bvhacker

Procedure
For my procedure I used a laptop running 32 bit Windows XP , although Brekel Kinect is confirmed to work also with win7 x86 & x64 as well as XP x64 & x86, but no Mac/Linux version of Brekel Kinect planned.

1. Once you have unpacked your Kinect you need to place it on a table or a tripod (which is what I did), at around 0.6m - 1.8m from the ground. You can plug it into a power socket at this stage, but do not connect the USB connector to your PC/laptop just yet.

Kinect mounted on an inexpensive camera tripod
2. Download and install the following software in this order:

Brekel Kinect 3D Scanner v0.36

OpenNI Alpha Build for Windows v1.0.0.23

PrimeSensor v5.0.0 (Modules for OpenNI)
Click the Downloads button and choose the zip file. After downloading extract the avin2-SensorKinect folder, and within this go to the bin subfolder, and run SensorKinect-Win32-5.0.0.exe

PrimeSensor v5.0.0 with correct options selected
PrimeSense NITE Beta Build for Windows v1.3.0.17
Use this key during installation: 0KOIk2JeIBYClPWVnMoRKn5cdY4=
(This is NOT a pirate key)

NITE Installation Wizard
Copy and paste the License Key from above

 Now download and install Bvhacker 


3. Once the software is installed you can  connect the Kinect USB lead. The Found New Hardware dialog should appear.
No need to connect to Windows Update
When it asks if it can connect to Windows Update, choose No, not his time. On the next dialog screen choose to Install the software automatically.

The hardware that will be detected are the Kinect Motor, Xbox NUI Audio and Kinect Camera. The drivers for the Xbox NUI Audio will not be found and will fail, but ignore this, they are not needed.

Device Manager showing the new device, PrimeSensor
When the drivers have been installed you can check that the hardware and its drivers are ready to be used by Windows by going to Device Manager (Start, Settings, Control Panel, System, Hardware, Device Manager) and you should now see a new entry, PrimeSensor, and under that will be the Kinect Camera and Motor.

4.You can now launch Brekel Kinect.

Brekel Kinect GUI
Brekel may launch with a Dark skin. You can change this by unchecking it under the Window menu item. I found I had to do this as some of the check boxes did not show too well with this almost black skin.

The only setting you now have to change is to enable NITE User Tracking at the centre bottom of the window.

The only setting to change, make sure NITE User Tracking is enabled
Once NITE User Tracking is enabled you should get a message in red on the central image that says: User1 [Looking for Psi pose]. Ensure that your full body is visible in the video windows, then adopt the Psi pose (like you are a trident, or surrendering). Once the pose is recognised it will take just a few seconds for tracking to commence (be patient, and KEEP STILL!), and the message should now go green.
Psi Pose
 As soon as you are being tracked (indicated by white dots covering your body) you can start recording animations. Start with simple ones like the ubiquitous salute, or hand clap. To start recording to BVH click the Start Capture BVH button (near the top right). If you are on your own you may find that tracking is lost if you go out of the field of view of the Kinect, in this case either try to have the PC/laptop within reach, or have a friend start the capture. There is a four second delay (useful if you are on your own) before recording starts. When done, click the Stop Capture to BVH button.

For full details on the capture procedure, and what can go wrong, go to the main menu, Help, How To - Capture motion to a BVH file. Additional resources may also be found on the main Brekel website.

Brekel can save BVH in two main formats, HumanIK (used by MotionBuilder and others) and Biped (used by 3DSMax). Neither of these can be imported directly into Second Life, as the naming convention for Root, joints and bones are slightly different, this is where Bvhacker comes in.

To prepare the BVH file for Second Life, start Bvhacker then File, Open, and load the BVH file. Once loaded you can play it to see what it looks like using the Start button at the bottom of the window. If all is well go back to the menu, Hack, and choose Attempt SL joint naming. This should go without a hitch, and you can now save it out ready for importing into Second life.

Have fun,

Rock

32 comments:

Stroker69 said...

Great tutorial Rock. Thanks for the effort! I'll set it up this weekend.

Anonymous said...

The Brekel 3D scanner is not accessible. secured login only?

Rock said...

Sorry, link now fixed

Connie in a Sec said...

Hey Rock. This is great. Thanks

Anonymous said...

ten out of ten for "great tutorial" :)

Anonymous said...

Thanks for the guide, the results are not so good tho? its not your fault! but wow its no where near smooth, maybe something i am doing wrong?

Gaga Gracious said...

Hi Rock

Good article and you have come up with great tutorial for Kinect!

I wrote some stuff about it on my blog recently and found some interesting videos.

It's nice to see you back blogging and it too. It would be nice to meet up in some place again. And what you doing with Anubia?

All the best

Rock said...

You can play with the smoothness settings in Brekel Kinect, and smooth some more if needed in bvhacker.

It is early days yet, but I am sure we will get a pretty good Mocap system out of the Kinect, with a quality suitable for most SL animations.

Rock said...

Hi Gaga,

Thanks.

I will check out your blog entries. I would love to meet up again, and maybe do some guest articles on each other's blog. I was away from blogging for a few months while I got a book finished. I should be back regularly now.

Rock

Gaga Gracious said...

Thanks for you comments on my blog 2011 predictions for the metaverse, Rock.

I would love to work with you again on some articles. In fact I have a lot of stuff on Avination to write about which could probably stretch to two articles so a good review with pics might go well for C&M. I have other stuff about commercial Opensim grids that will be a bit more controversial so I will save that for my own blog, lol

Hmm, let me see. I am also following the Aurora sim development which has a great list of improvements they will make to Opensim. I want to write about that too and share.

Thinking back to that little tour you gave me of your SL sky build I remember it well. I love Middle Eastern architecture so I hope you will re-build Anubia and not leave that grid sitting all empty and unloved.

Opensim grids are growing in numbers and traffic now with a growth of 100,00 registered users since last summer. People are dumping SL homes and living on the free metaverse at a fraction of the cost while still selling content back in SL. Anubia could be a thriving little community again if it were hypergrip connected.

Anyway, Lets meet up soon *smiles*

Gaga

Sarah Kline said...

I discovered assuming a T pose before recording starts helped the shoulder rotation problems I had when bringing in to bvhacker.

Rock said...

Thanks for that Sarah. I will try it myself.

There also appears to be a couple of joints misnamed, but I need to confirm that, and the fix.

Rock

Willow Plays said...

I'm getting ridiculous shoulder arm issues. Even going T-Pose doesn't help. Seems like the axis the arms are on is WAY off. Any fix?

Rock said...

I have not had time to look at the arms issue yet, been working on the two Kinect process. If you look at this thread, Post #18, it may solve your problem, but do let us know if it works for you or not.

http://www.sluniverse.com/php/vb/general-sl-discussion/55794-using-kinect-generate-sl-animation.html

Rock

Sarah Kline said...

Well I renamed the shoulder joints to lCollar & rCollar before 'attempt renaming', then did my retargeting. I cropped how I wanted then pressed !Zero and Control M to complete the processing. And the arms are fine. What's needed now is an easy way to clean up jitter and foot drift.^^

Rock said...

Excellent.

You may be able to cure jitter by playing with the Smoothness setting in Brekel. Try 0, 20, 40, 60, 80, 100 and see what effects that has. I will contact Dave over at bvhacker to see if any cures for jitter lurk there.

Rock

Rock said...

Just to clarify, are you seeing the jitter when you open the BVH in bvhacker and press the play button, or are you seeing it only when you apply the anim after uploading into SL?

Sarah Kline said...

Hi I've been using 80.. any more and movement slows to halt at 100. The slight jitter is in bvhacker. Even pro mocap needs cleaning up so results are pretty good considering!

Sarah Kline said...

What I am noticing though is there there seems no data coming in to bvhacker for the foot. So the character tends to float around on its shins. Anyone else have this?

Deepgreenseas said...

Hi, just recently bought a Kinect and tried following the directions above, but have run into some trouble. I'm using Brekel Kinect v0.38 to create a bvh file using the Second Life bvh capture mode (new to this version of Brekel Kinect?). I then brought the bvh into bvhacker 1.8.0.3. I renamed the joints under the Hack menu, and that checked out. The animation played correctly, but when I imported it into SL, it turned into scrambled eggs. I went back to bvhacker and tried a number of things: I noticed that the abdomen joint was missing, so added that successfully, and ran the retarget to SL under Hack. I also added a T pose frame at the start. Ran again in bvhacker, and all looked just fine. But again, when I brought into SL, my avatar kind of floated oddly in the air, with arms straight out to the sides doing weird leg and abdomen contortions having nothing to do with what I saw in bvhacker. So, I'm at a total loss what I'm doing wrong. Any ideas?

Rock said...

Hi DeepGreenSeas,

It certainly looks like a bvhacker setting isn't quite right, especially as it plays ok inside bvhacker. I would post your query on his forum as he is pretty good at solving problems like this. It is at GoogleGroups forum, here: https://groups.google.com/group/bvhacker?hl=en

Rock

mrbones said...

Howdy,

I read on the Bvhacker Goggle Group that you have uncheck the translation box or something.

Good luck and Cheers

Drea said...

I was wondering if someone could help :) I installed everything how the post said, my kinect is plugged in, and detected, drivers are fine. But when i load up Brekel Kinect, the screens stay black. When i ran the OpenNI sample, it saw me fine. Has anyone had this issue, or know whats going on? Thank you :)

Andrea

Rock said...

Hi Andrea,

I would post your query in the Brekel forum. Jasper is pretty good at sorting out problems like this.

http://www.brekel.com/?page_id=368

Rock

Samm said...

So is the kinect solid in making animations or not so much yet? I'd love to be able to make anims with the useless toy. Lol.

Rock said...

Hi Samm,

The current progress of myself and others is mostly discussed in SLuniverse, link to the forum thread here:

http://www.sluniverse.com/php/vb/general-sl-discussion/55794-using-kinect-generate-sl-animation.html

Short answer is that we are making good progress, but I cannot assess yet how good we can get the anims after getting the best out of the hardware and software.

Once we get the best out of what is currently available we will then assess if that suits our quality needs, and if not we will start looking at how that can be achieved, for example by synching two Kinects together, better software drivers, etc.

Rock

Samm said...

Alright well after a while I decided to play around with the Kinect anyway so I followed the instructions. One thing however is that when I plug the Kinect USB in and I try to install the drivers, neither the motor or camera drivers install.

All I have is PrimeSensor > Xbox NUI Motor

Samm said...

And if it makes any difference I have windows XP x86.

Samm said...

And here is a picture of the error.

http://s752.photobucket.com/albums/xx163/chineseengineer/?action=view&current=n.png

Rock said...

Hi Samm,

I am no expert on the OpenNI drivers, they just workd fine on kmy machine.

I would recommend asking the OpenNI support forum to help here. Here is the link:

https://groups.google.com/group/openni-dev?hl=en

They should be able to sort it out for you.

Rock

Anonymous said...

Thanx for the tutorial.
I asked on the xbox forum about the cable adaptor for my kinect (I bought it with xbox, so don't have usb conector) and told me to be carefull about loosing original drives and breaking kinect for xbox. No risk on that, right? ^^'
thanx

Cat

Anonymous said...

OpenNI Alpha Build for Windows v1.0.0.23 link is not active ^^'

Cat