Tuesday 5 April 2011

Streaming Video into Second Life/Opensim - The Easy Way


If you are considering streaming video into Second Life/Opensim (SLOS), either from a dynamic video source such as a webcam, or from pre-recorded video files (and for the purpose of this article I will concentrate on pre-recorded video files), then you have several choices. First the Prerequisites:

Prerequisites
  • You must either be the owner or estate manager, or a member of a group that owns the property you plan to stream into (unless you are just streaming to others who are)
  • Your video source must be in the QuickTime format, or in a format that QuickTime supports
Note: The two most common Quicktime formats are *.mov and *.mp4, but there are other supported formats and these are listed on the QuickTime wiki page. If the video you are attempting to stream does not play in QuickTime Player then it will not play in Opensim, so it is a good idea to download and install the QuickTime Player, and check that your video source plays OK. Details on the specific format used for SLOS are available here.

Choices
The  choices available to would-be streamers are as follows:

  • Set up a video streaming server on a dedicated server or Virtual Private Server (VPS)
  • Lease a video streaming service
  • Do it the easy and zero-cost way
Setting up a video streaming server is not for the faint-hearted, and the streaming software can be quite expensive. There are open-source alternatives available, but can prove very daunting for the non-tech savvy, as they often need compiling from source and usually are command-line driven.

Leasing a video streaming service, such as Streamhoster or StreamUK, can likewise prove costly, depending on the number of users who simultaneously connect to the stream. There are some low-cost or free video streaming sites, such as Ustream, but where you have much less control, and compatibility with the QuickTime format is not always assured.

The whole point of streaming is that you do not need to wait for a video file to download completely before you can start viewing that video, you can start watching almost instantly. True streaming is one-to-many. If you are watching a stream and another user joins you, they will start watching the stream at the same point as yourself.

Youtube is the best known example of video streaming, but they use the Flash flv format which cannot be used in SLOS. It is possible to access an mp4 version of Youtube videos using any of a number of add-ons for browsers, particularly Firefox, that create a 'Download mp4' option on the Youtube page, and then rather than downloading the mp4 you could right-click and use the Copy Link Location option to get the URL link which you can paste directly into the About Land, Media tab, Media URL in SLOS. However, this often breaks as Google keeps changing the code for the mp4 streams on Youtube, and the URL links are now extremely long.

Do it the Easy Way
Until recently, I used the easiest method of all, and simply uploaded my mp4 files to a file hosting site, and linked directly to them. This works, but it is not streaming. I have to wait a minute or two for the video to download before it starts playing, and others who join me must do the same, so we are all watching the same video but at different points. If you leave the parcel while the video is playing, then return, the video will start again from the beginning, no matter how briefly you left the parcel.

The good news is that there is a middle way, and is easy, and has zero cost.

You can employ something called pseudo http streaming, but rather than one-to-many it is many one-to-ones. What this means is that I can watch a stream, but when another avatar joins me, the video will stream from the beginning for him, rather than at the same point as my video stream, but we both enjoy the advantages of streaming and do not need to wait for any files to completely download before watching the feed.

How do you set up pseudo http streaming?
Both mp4 and flv files are capable of being streamed using pseudo http streaming, and of course it is the  mp4 format that we are interested in. Servers do not stream mp4 files by default, and require a server side module to enable it. If you have your own server and wish to do this yourself, look here.

Fortunately, several free file hosting sites appear to be streaming module enabled. So I will explain the workflow for employing one of these sites.

Prepare your Files
You will need to first get your video files into the mp4 format. A good, free, and malware-free converter is Super (c) available from here (download link near bottom of page) and another is H.264, available here.

Super (c) Video Converter
As you can see, there are a number of parameters that can be set within the mp4 format. Some must be kept as standard, such as the Frame/sec of 29.97, and a sampling frequency of 44100 for the audio. The others can be experimented with. As I do not need excellent quality video within my Opensim I have gone for low settings: 320:240 video scale size; and video and audio bitrates of 160kbps and 32kbps respectively. This translates to a typical 3 minute music video file size of just under 20MB. This keeps well within the limits imposed by free file hosting sites, and ensures a fast start when played. Even better would be to select the typical 3G video scale size of 176:144, which gets the file size down to 10MB, and the difference in quality is almost imperceptible in SLOS.

Now, there is a trick to all this: many video studio software suites, and video encoders / converters, produce mp4 files with their metadata located at the end of each file, and most do not inform you about this, or give you any option to change it. This means that the entire file must be downloaded first before playback can begin. However, by moving the metadata to the front of the file the video can be streamed and watched almost instantly. A handy, free converter is available to do just this, simply called MetaData Mover. This great utility works on a folder of videos, rather than just on a single file. So create a new empty folder on your desktop and copy the files you want processing into it. Point MetaData Mover to the folder and it will process every file in the folder, moving the metadata from the back to the front of the file. One file typically takes 10 seconds to process.

You are now ready to upload your files.

File Hosting
The first file hosting site I tried did the job perfectly, so I am sure there are many more that are streaming module enabled. I used Fileden, which is a file hosting site, just requiring the registering of a free account. Once you have uploaded a file you are given the direct URL to the file. Not all file hosting sites provide this facility, so do shop around for one that suits you.

Once you have the URL to your mp4 file, just paste it into the media box of About land and the video should start playing almost instantly. If you are using some kind of video jukebox then just load the jukebox with the URLs of your uploaded video files. Needless to say, ensure that you avoid copyright infringement.

Playing your Video Streams
To set up a land parcel for playing video in SLOS, as mentioned previously,  requires that you either be a land owner, estate manager, or member of a group that has the privileges to set the Media for streaming video. This option can be accessed by either clicking the address in the title bar of the viewer, or by right clicking the ground and selecting the About Land option, then go to the Media Tab.

There are two main settings here for video. The first is the texture that will be replaced by the video when it plays, and the second is the URL for the video file. The texture can be any picture texture you like, but the same texture must be on the video screen that you intend to use. So don't use a common texture, like brick, or every house in your sim will turn into a video screen! You will also need to set up your Audio and Video settings in Preferences to enable streaming media. See Torley's video here on how to do all of this.


Have fun,

Rock
Sunday 3 April 2011

Kinect and SL/Opensim Animations: An Update


Following the previous article outlining my experiments with using the Kinect for creating animation files for Second Life and Opensim, I thought an update might be in order to summarise some of the lessons learnt, both by myself and other experimenters.

Since the last article on the 24th February, two of the key software applications used in the workflow, Brekel Kinect 3D Scanner, and Bvhacker have had further development. So what's new in the latest releases?

Brekel Kinect 3D Scanner
Two further releases have been issued since the last article:

v. 0.39
The most important changes in this release for SL and Opensim users is that now BVH positions are now OFF by default. Some experimenters were creating bvh files with Brekel, importing them in bvhacker, and in there they played OK, but when optimised by bvhacker, and then uploaded into Second Life, all kinds of problems were seen. As Deepgreenseas reported:
When I brought into SL, my avatar kind of floated oddly in the air, with arms straight out to the sides doing weird leg and abdomen contortions having nothing to do with what I saw in bvhacker. So, I'm at a total loss what I'm doing wrong. Any ideas?
 The problem was that in the BVH Capture section of the Brekel window, there is a check box for "Write
Positions." This check box was set ON by default, and so the resulting bvh file had both translation and rotation data recorded, while bvhacker and SL/Opensim require only rotation data. In v.039 this setting is now OFF by default, so only rotation data is present in the bvh files now.

Other improvements included easier installation, a more compact GUI,  a warning when running below 30fps (and thus dropping frames), more tooltips, and other improvements and the squashing of some bugs.

v.0.40
New features/improvements include:

  • knees/feet should be better now when turning around
  • Second Life format for bvh export option added *
  • hand rotations are now supported using some custom calculations 
  • added motor/LED support
    • you can tilt the Kinect up and down (see slider in the CPU/FPS dock)
    • the LED color will be set according to the skeleton calibration phase
    • please see the How To under the Help menu on how to update the driver 
This last improvement, the ability to control the Kinect movement, is a big benefit, and will save me from all those fiddly adjustments to my tripod settings.

* I have not yet tried the Second Life option when saving bvh files. I would be interested if it now solves Twisted Pharoah's problem below.

Bvhacker
The latest release is now v.1.8, and the new features and changes are as follows:

  • Added three new video tutorials to the help section
  • Now possible to merge joints with their parent - useful for converting skeletons with roll bones
  • Now possible to delete descendants - useful for removing multiple finger or toe joints
  • Now possible to zero out descendants - see new video tutorial for use
  • Added preferences dialog (Ctrl P)
  • Now possible to set the default file view for open and save dialogs
  • Now possbile to show or hide the head mesh display
  • Added Zoom In and Zoom Out buttons to the view shortcuts
  • Removed 'Show first frame' view option

There is also a note on the file properties pane saying that the joint count includes End Sites. This had confused me at first as earlier research had shown that SL/Opensim typically uses 24 joints, yet when I counted them in bvhacker I could only see 19. This is because the 5 End Sites (one for the head and two each for the hands and feet) should also be added. End Sites can also be manipulated, to make the feet point backwards, for example, but I will leave it to the more imaginative experimenters how they use these End Sites.

I will now provide some further info about joints/bones and some of the problems faced by differing naming conventions used in various applications.

Joints/Bones
Some sources refer to the constituent parts of a bvh file as being composed of bones. This also had me confused at first, but Dave (the author of bvhacker) explained it well:
Although the terms 'bone' and 'joint' can be used interchangeably, I
prefer 'joint', as the position, rotation and translation apply to the
point in space where the joint actually is. Mathematically, the 'bone'
is actually just the space in-between two joints, so I personally find
it easier to think in terms of joints instead of bones.
Unfortunately, SL/Opensim mainly uses the bones naming convention, so rForearm is the bone between the wrist and elbow in the right arrm.

The Nite tracking software for the Kinect and Brekel use joint names, so they do use RightWrist and RightElbow. The list of joints they use are:

  • Hips
  • Chest
  • Neck
  • Head
  • End Site

  • LeftCollar
  • LeftShoulder
  • LeftElbow
  • LeftWrist
  • End Site

  • RightCollar
  • RightShoulder
  • RightElbow
  • RightWrist
  • End Site

  • LeftHip
  • LeftKnee
  • LeftAnkle
  • End Site

  • RightHIp
  • RightKnee
  • RightAnkle
  • EndSite

The list of bones/joints used in SL/Opensim are:
  • hip
  • abdomen
  • chest
  • neck
  • head
  • End Site

  • lCollar
  • lShldr
  • lForeArm
  • lHand
  • End Site

  • rCollar
  • rShldr
  • rForeArm
  • rHand
  • End Site

  • lThigh
  • lShin
  • lFoot
  • End Site

  • rThigh
  • rShin
  • rFoot
  • EndSite
As you can see, the NITE tracking software for the Kinect does not use both joints in the torso (chest and abdomen), just the chest, and explains the difference in the number of joints/bones, i.e. 23 and 24.

Bvhacker is designed to do the conversions between standard bvh files in joint-naming format and  the SL format using bone-naming format. However, it has been noted that the bone names in NITE/Brekel was causing some problems in Bvhacker, which renamed two joints to the same bone name. This was found by Twisted Pharaoh in the SLuniverse forum:

BVHacker renames 2 joints lShldr and 2 others rShldr.

What you should do is rename LeftShoulder into lCollar manually, RightShoulder in rCollar then do the Attempt SL Joint naming.
To manually rename joints in Bvhacker just load the bvh file, then in the joints pane on the left, select the joint and right click, and select Rename Joint.

Other Tips
Sarah Kline:
I discovered assuming a T pose before recording starts helped the shoulder rotation problems I had when bringing in to bvhacker.
 If you have any other Tips or suggestions for improving the current workflow, (if you find a cure for slight jitter, for example), then please add your comments to the article and I will provide a further update later.

Rock