How to record a holographic video message with Azure Kinect
πŸ“Ή

How to record a holographic video message with Azure Kinect

πŸ“Œ

Software used:

⬛
Depth Recorder

In this tutorial, I'll go over how to use Microsoft's Azure Kinect camera with Depth Recorder to easily capture your very own home holographic recording. Technological advances in camera and depth capture are moving us closer and closer to capturing and experiencing our world in its truest 3D.

Depth Recorder makes recording holographic video an intuitive process β€” you're able easily to frame your 3D capture with a live feed, add your choice of video effect and background options, and record quick videos that are perfect to send as hologram messages.

You know, like, "Help me, Obi-Wan Kenobi. You’re my only hope."

To follow this Tutorial .... you will need β†’

Hardware

  • Looking Glass Portrait
  • Microsoft Azure Kinect
  • your PC

icon

Depth Recorder is currently Windows only. Bootcamp will run Windows on Mac!

πŸ“Œ

Make sure you have Depth Recorder and HoloPlay Studio downloaded or installed before we get started on the computer you've connected to your Looking Glass display in Desktop Mode. Please also make sure you have already downloaded HoloPlay Service, which you will need to run any and all Looking Glass applications and plugins.

Connect your Azure Kinect camera and Looking Glass Portrait to your computer (in Desktop Mode) before running Depth Recorder. See the Azure Kinect hardware setup guide for more information on the device β€” note, however, that its power cable requires a USB 3.0 port to supply sufficient power!

Desktop Mode

To run HoloPlay Studio, your Looking Glass Portrait should be in Desktop Mode, meaning that your display is connected to your computer. Once it's connected, the LED indicator will pulse blue (meaning its ready to plug into the matrix!)

Switch the Portrait into Desktop Mode by pressing the ON/OFF button. Review the Looking Glass Portrait Getting Started Guide for a more detailed refresher.

image

Azure Kinect

(In case you don't already know,) the Azure Kinect is a depth capture device that uses two cameras: one for RGB capture and a depth sensor to capture 3D information.

It's certainly one of the best depth capture devices out there β€” it can capture up to 4K resolution with its RGB camera and the depth sensor technology and accuracy is certainly nothing to sneeze at.

image

Depth Sensor

πŸ“–

Learn Alert

Microsoft's Azure Kinect outputs three videos: RGB, IR (infrared) Image, and a color depth map. The depth sensor uses Time-of-Flight to create depth imaging, meaning it's sending out infrared light at a cadence to and measuring how long it takes to reflect back.

image
image

These photons emitted from the infrared light provide the information needed to construct the depth map. It's basically a photon rave and the end result is you, mapped out on the Z-axis.

Fortunately,

you don't have to worry about any of this to make your first holographic video message (though there might be a quiz at the end of this tutorial 😎).

All you really need to know right now is that Depth Recorder takes in this depth data from your Azure Kinect in real timeβ€” enabling you to view yourself live holographically, adjust effects and framing, and record up to 10 seconds of footage at a time.

Let's create your first holographic video message

image

Once you start the Depth Recorder app with the Looking Glass Portrait (in Desktop Mode), the live feed of the camera will appear in the Looking Glass display alongside the application window in your monitor:

image

Making Adjustments

Before we record our first video, we want to adjust our feed just right using the settings available to us in the monitor window.

First, Let's Adjust the Framing of the Video

image

Check out what you look like in the holographic display (super 3D, of course). You can use your mouse on the Looking Glass window to adjust the position of the camera with the following controls:

  • Left click + drag to rotate
  • Right click + drag to pan
  • Scroll wheel to zoom
  • Double click the video image to reset the position

Frame yourself in the display using the controls above to the position you like!

Apply Effects

Play around with the effects listed in the Effects window to give a different feeling to your video recording. If you keep the "Show Base Video Feed" checked, the effect you select will overlay on top of the video that you begin with when you open the application. Unchecking it while an effect is selected removes that base video.

Try out the effects with "Show Base Video Feed" checked and unchecked and see what you like best! I'm partial to Radar with the base video feed off, myself... it gives me the vibe that I'm calling in from Space Control with an urgent message.

image
image

Customize Background

When left blank, the background of the video feed is simply black. Play around with the different background presets and go with the one you like best for your first holographic video message.

The "Real World Background" option allows you to visualize what the Azure Kinect is seeing. If you want to see objects appear in your background, pull them up pretty close to you and the camera.

πŸ’‘

Quick tipβ€” To get a better a view of your Real World Background, try zooming out with scroll wheel in the display and rotating the angle of view!

image

Adjust Focus & Exposure

Adjusting Focus will pull or push the video feed forward and backward to help you center your subject in the focal (or zero parallax) plane of the display. Feel out where you would like the video to look the most focused by playing with these values.

Adjusting Exposure changes the exposure of the video feed to brighten or darken your scene. It does not adjust the background exposure, however.

image

Start your Recording!

After you've set up your feed and you're ready to record, hit the "Start Recording" button. Recording will capture up to 10 seconds for each clip. Once it's done processing (which might take a second), it automatically saves in C:/Documents/DepthRecorder as a quilt video with 48 views.

πŸ“–

Learn Alert

Hey, but what is a Quilt recording?

Quilts are an image and video standard that Looking Glass uses to produce 3D experiences.

Each tiled image or video represents a view of the 3D scene, with the bottom-left tile representing the leftmost view of the scene and the top-right tile representing the rightmost view of the scene.

So in this case, Depth Recorder is capturing 48 views of your scene to render to a quilt that can be interpreted in HoloPlay Studio as a recorded hologram!

Your quilt recording from Depth Recorder will looking something like this!
Your quilt recording from Depth Recorder will looking something like this!
πŸ“Œ

A Quick Note about Audio No, there is nothing wrong with your hearing - no audio is captured with your quilt recording in Depth Recorder. A work around for this is to simultaneously capture audio with a separate device (like your iPhone or Android phone), and then pull this audio in via video editing software like Premiere Pro. You can easily pull quilt videos into Premiere Pro to drag pre-recorded audio or music onto video recordings. Then, pull that edited quilt recording into HoloPlay Studio like normal!

Bring your Quilt Recording into HoloPlay Studio

Open up HoloPlay Studio β€” we're going to pull your new recording into our holographic haven.

image

With HoloPlay Studio, you can import your quilt recordings from Depth Recorder to view immediately and seamlessly add to your playlist to run in Standalone Mode.

Add Hologram

Select Add Hologram > Quilt Image/Video > Import File and navigate to your freshly recorded quilt video. It'll immediately appear in your Playlist and simultaneously (paused) in your Looking Glass Portrait. In order to view it in it's holographic glory, hit play in HoloPlay Studio's playback bar.

πŸ“Œ

If you didn't add a background to your capture in Depth Recorder, your video will appear black in the Looking Glass Portrait until you hit the play ▢️ button in the playbar.

Play your Hologram Video

Like any playlist, it plays. But this playlist is the only one in the world (right now) that plays holograms. You can play and pause your hologram, skip forward or backward in your playlist , and loop a single hologram on repeat.

image

Press Play ▢️ to watch the magic you just recorded with your Azure Kinect!

Ok this really looks like a 3D photon party.
Ok this really looks like a 3D photon party.
image

Hologram Properties

When a quilt image or recording is selected, the Properties panel will open up on the right with Columns and Rows.

Depth Recorder exports all your videos as a quilt recording with 8 columns and 6 rows. You won't need to worry about inputting those numbers yourself, as they will automatically show up in the Properties panel to the left of your selection.

Leave β˜‘οΈŽ Use device aspect ratio (under Advanced) checked β€” that just means that HoloPlay Studio is using your connected display's aspect ratio to play the quilt.

πŸ“Œ

If you rename your quilt video exported from Depth Recorder on your computer, the quilt Properties may not automatically populate correctly and you will have to input them manually.

Frame and crop your hologram with the following controls:

  • Mouse wheel / two finger scrollβ€” zoom in and out
  • Mouse click + dragβ€” pan around image

Once you're done making your properties and framing adjustments, those changes will stay until you decide to edit it again.

πŸ’‘

Quick tipβ€” Hit the the loop button in the Playback bar to keep your video playing on repeat while you edit the properties and framing.

image

Edit Playlist

Your playlist is your collection of holograms you've brought into HoloPlay Studio. Let's briefly go over the options in the Playlist panel.

Edit

image
image

Right click the photo you just imported into your Playlist, to

  • Duplicate,
  • Edit Length,
  • Delete, or
  • Rename your hologram.
  • You can also find these options by going to Edit in the HoloPlay Studio's window bar.

Most of these options are pretty self explanatory, but I want to take the time to go over Edit Length. Edit Length allows you to edit the time that a hologram plays in your playlist, both in Desktop Mode and Standalone Mode. If the hologram is a video (like this one is), you won't be able adjust its length of play.

πŸ’ͺ🏽

Hologram Exercise: Rename your holographic video recording so that, when you import more, you know which recording is which in your Playlist!

Reorder Playlist

image

If I have a specific order I want my holograms to play in, how do I do that? I'm glad you asked.

When you hover over your playlist selection, you'll see a small grid appear to the left of the hologram's title.

By dragging that, you can reorder your playlist any way you'd like.

Once you've populated your playlist with your new quilt recordings, it's time to Sync our Playlist.

Sync Playlist

Now that you've added your holographic recording to your Playlist, click on the Sync Playlist button. You're syncing to save your holograms to your device for Standalone Mode to play without having to be connected to your computer!

image

image

Standalone Mode

To view your holographic playlist you created in HoloPlay Studio, switch your Looking Glass Portrait to Standalone Mode. You can have the Looking Glass Portrait connected to either your wall adapter or to your PC/Mac.

Switch the Portrait into Standalone Mode by pressing the ON/OFF button. Review the Looking Glass Portrait Getting Started Guide for a more detailed refresher.

image

icon

For more detailed definitions on HoloPlay Studio's functionality, including more information on Quilt images and videos, please visit our docs site.

image

So, how was capturing and pulling your holographic video recording into your Looking Glass Portrait? I'm personally very excited about the rapidly evolving technology of depth sensors and capture devices and can't wait to continue writing tutorials about more ways to capture videos with depth maps in formats that work seamlessly with HoloPlay Studio.

As we continue to update Depth Recorder, we are looking forward to adding audio recording functionality, possible Intel RealSense support, and even possibly supporting iPhone capture. Of course, we would love your feedback on how your experience with the software is, so please don't hold back and shoot as an email with your thoughts πŸ™‚.

πŸ€— Please feel welcome to:

  • Share your videos and capture tips with us in our community's Discord.
  • Shoot us an email at support@lookingglassfactory.com with any questions or feedback
  • tweet at us (@lkgglass) or tag us on instagram (@lookingglassfactory) 😎

Why did the photons go to bed?

To catch some Zs!!! πŸ₯πŸ₯πŸ₯

🎁

Attached is a quilt recording I created with Depth Recorder of my cat Artemis, for anyone that desires to see what the best cat in the world looks like as a hologram.

Recording_2021_0304_225501_qs8x6a.75.webm34739.6KB