Building a Magical Augmented Reality Sandbox

CREDITS AND SOURCES

Before any info about the project or instructions I want to mention and credit several sources. These are the sites I used when building our AR Sandbox and I recommend visiting them for more info and to clarify any instructions:

Original project site by Oliver Kreylos – http://idav.ucdavis.edu/~okreylos/ResDev/SARndbox/

System76 Ubuntu install instructions – https://system76.com/weekend-project/arsandbox

Thom Wolf’s imgur post (this is different software though that is simple but lacks flowing water) – http://imgur.com/gallery/Q86wR 

Oliver Kreylos youtube channel – https://www.youtube.com/user/okreylos/videos

 

IMG_20170508_080255

Introducing our Augmented Reality Sandbox. A project created by Oliver Kreylos at UC Davis, the premise is quite simple:

  1. An xbox kinect above the sandbox measures the height of the sand
  2. This information is processed by the computer
  3. The projector then shows different heights of sand in different colours.

This means that students (or teachers) can dig out their own landscape that will be coloured according to the topology of the ground. Additionally, a fluid simulation allows you to hold out your hand and create rain that then flows over the contours and ditches of the landscape. It’s pretty amazing stuff and something that is not too difficult to make.

I’d been sitting on most of the parts for the last couple of months but just needed the time to get started. In the end I decided to make it a one day project, which turned out very achievable.

What can it do?

The AR Sandbox allows students to explore, dig and rearrange the terrain of the map, the software will then update the topography to show the height of the land in different colours.

 

IMG_20170508_104038

 

It also allows students to create flowing water from ‘rain’ they create by holding their hand over an area:

There are a number of other cool things that can be done with it and we will fit in it’s use with several Units of Inquiry in our school ranging from landforms to seasons to water. There are additional scripts as well to introduce things like lava and volcanoes into it.

 

HOW TO BUILD YOUR OWN AR SANDBOX

Here is a list of required ingredients:
  1. Xbox Kinect V1 (the one from the Xbox 360, not the Xbox One)
  2. A projector (any will do but shorter throw means bigger sandbox and less projector height)
  3. A computer to run the SandBox software on. This has a few requirements so a section below this list is just on the computer part.
  4. A monitor or display connected to the computer as well as the projector.
  5. USB thumb drive (at least 16GB for linux install)
  6. A 12V 1A or 2A power adapter or a Microsoft xbox kinect to pc adapter
  7. Wood (measure to the sizes you need, we’ll talk size later)
  8. Sand to about 15-20cm depth (play sand recommended but any will do)
  9. An old CD, or some card (to build a calibration disk)
  10. Some rolls of tape of various heights (to hold up the calibration disk)
  11. An old USB cable you don’t want, soldering equipment and electrical tape if you are not buying the microsoft kinect PC USB adapter.

We, fortunately, had everything on the list apart from the wood, so not a lot of expense.

 

Computer Requirements:

The computer to run the sandbox software will depend on what you want it to do. If you only want your sandbox to display the coloured contours of the land then most modern laptops of desktops will be fine. If you want to be able to do the rain and water flows then you’ll need something with some more powerful graphics. The specs list below is actually a build we did to run an HTC Vive VR set on, which is also now running our AR Sandbox. The project creators highly recommend against  running this on Mac OS X. I agree with this as it is really designed as a linux project and Apple don’t make a computer with graphics powerful enough to do the water flows, outside of the very top of the line Mac Pro. Fortunately, powerful enough hardware for a PC is pretty cheap. The instructions below are, therefore, only for setting up a Linux computer with an Nvidia graphics card.

CPU: i3, i5, i7 CPUs will all be fine. We are actually using a Pentium G4560, which is a super good budget CPU recently launched.

RAM: 8GB will be fine

Graphics Card: Nvidia GTX1060 is going to be the best bang for buck here and perfectly handle the water simulation stuff. The 3GB GTX 1060 would be absolutely fine for this, we are using the 6GB version.

Storage: 30GB or so (our computer runs windows on it’s main partition and linux in a 30GB partition). Do not try to run this in a VM, set up your computer as a dual boot or solely Linux.

OS: Linux Mint 64bit or Ubuntu 64bit (full screen button doesn’t work quite right in Ubuntu though)

Additional: Install the latest Nvidia driver for Linux

 

If you’ve never run Linux before then please don’t let this be a deterrent from having a go with this project. It’s pretty straight forward and instructions are complete for every single step.

Our computer already runs Ubuntu Linux as it occasionally finds itself used as a minecraft pocket edition server for our ipads. If you have the choice though setup Linux Mint instead.

 

MAKING YOUR KINECT WORK WITH A PC

The Kinect you need for this is the V1 from the Xbox 360. The new V2 for Xbox One does not currently work with the AR Sandbox. Although the connector looks like it is a USB it will not work if you connect it to a computer. This connector is actually carrying the normal USB wires plus also an extra 12V to power the Kinect. Microsoft do sell an adapter to convert this to a PC USB with a separate power supply but I couldn’t be bothered to try and find one of those and would ruin my 1 day build plans. Nice and easy is to find a 12V 1A or 2A power adapter. I actually found 3 of these in the draw and chose to use one from an old Roku. Good instructions for rewiring a Kinect for PC USB can be found here: http://www.instructables.com/id/Wiring-an-Xbox-Kinect-for-USB/

First step is to cut the Kinect cable near the top. Inside you’ll find 5 wires. 4 of these are standard USB wires and coloured Red, Black, Green and White. Cutting the connector from an old usb adapter and just splice these together with the ones in the USB cable. The final wire inside is brown and carries the 12V power. Cut the barrel connector from the 12V power supply and splice the brown wire to the positive. The negative needs to attach to the black usb wire (so three wires together on the black). To identify the positive either use a multimeter to be 100% or look for the wire with the dashed white line on it.

FOAQBHUHH2VL3XA.LARGE

Isolate each joined wire with electrical tape and then tidy it up with either heat shrink or more tape.

IMG_20170507_100835

It’s probably worth then connecting it to a windows computer with the Kinect SDK and Development tools to just test that it is working ok. Run any of the demos and see that the picture is working and the connection remains stable.

If all is good then move on to building the frame

 

BUILD INSTRUCTIONS

Starting with the physical structure first it really is up to you how you do this. I’ll share my plans below but there is plenty of scope to make this in lots of different sizes. The projector will be a deciding factor to some degree in this though. Also remember that the Kinect is a 4:3 aspect ratio, so to get the best resolution try to stick to a 4:3 ‘screen size’.

I made a decision that I would build the Sandbox on top of an IKEA LÄTT children’s table. It is super cheap and a good enough size and height for primary age children. It is not as big as some AR Sandboxes you will see, but fine for us and has the advantage of not needing a, more expensive, short throw projector. First job was to hold the projector, switched on, above the table to estimate the height it would need to be mounted. The projector I used was an Epson Powerlite 1715C. These were a mainstay of our school projectors a few years ago so we have a handful around. It has a pretty good ratio of height to image size, which worked out to be about 180cm to more than fill the table, Don’t just measure the distance to fill the table area though as the height will be higher with sand in it and you’ll end up not covering the whole sand surface (I made this mistake). Below is a dodgy sketch of the frame and sizes.

IMG_20170510_004100 IMG_20170506_233758

Hong Kong wood shop liked working in a mixture of inches and cm

IMG_20170506_224249

IMG_20170507_102321

IMG_20170507_101307

IMG_20170507_100111

IMG_20170506_224240

IMG_20170507_104750

Assembly was mostly done with self tapping screws, steel brackets and a few nuts and bolts. A good electric screwdriver makes this process much less stressful. I chose not to mount the kinect yet as I needed to see the camera output on the computer screen before deciding height. I made a frame for it though and 3D printed a mounting bracket I found on Thingiverse.com. This would be attached later.

IMG_20170507_100124

IMG_20170507_100129

Another piece to make is a lid for the sandbox. This is important for calibration later and should be completely flat and non-shiny. I cut mine with a jigsaw from a piece of acrylic, keeping the brown protective paper on it.

IMG_20170507_122846

IMG_20170507_122905

Once the box part was built I sealed it all in using normal silicone bathroom sealant to prevent sand leaking out. This will take a few hours to dry.

IMG_20170507_101302

Once everything was assembled I moved the sandbox from home to school. I figured once the sand was in I didn’t want to remove it again and it would be too heavy to transport. Once all in place the sand can be added. I didn’t need nearly as much as I thought I would, but good to have too much rather than too little.

IMG_20170507_104750

As mentioned before, I originally calculated the height for the projector to fill the top of the table. Big mistake as once the sand was in the projector now only filled a rectangle in the middle of the sand with a 2 inch border around it. Live and learn, so had to extend the height of the frame. Your projector can overlap the sides of the sandbox a little, this is ok but try to just fill the sand area.

IMG_20170507_110406

At this point the computer needs setting up (or you could start that while waiting for the silicone to dry).

 

COMPUTER SETUP

If your computer already runs Linux then skip ahead this section:

 

Setting up Linux on your computer.

To start off you will need to decide how you want the computer to boot. If you are planning to dual boot with Windows on an existing machine then follow these instructions first to free up some space on the windows disk.

Next to create a Linux install USB stick. This is super simple to create on either Windows or a Mac. If on Windows download RUFUS or if on Mac go with UNETBOOTIN. I preferred to download the Linux ISO file first, as it was faster, and then choose this image. Here are the needed options in RUFUS.

Image result for rufus linux mint

Once complete boot your thumb drive in the computer. Choose the option to install the OS. If you are dual booting with an existing Windows install then follow these instructions very carefully for partition setup:

If this computer is only running Linux then just install and choose options for network, region, etc.

Either way, you will eventually reach the Linux desktop once installed. First thing to do is to install the latest Nvidia driver for the graphics card. Do not rely on the included 3rd party driver, it did not work for me.

To install the Nvidia driver you may be prompted during the Linux install. If not then don’t panic, you can bring up control panel and then ‘driver manager’, once the list of drivers comes up choose the nVidia one…. or bring up a terminal and enter the following lines, pressing ‘enter’ after each:

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt-get update
sudo apt-get install nvidia-378 nvidia-settings

Restart your computer now! Once rebooted make sure you have your display and sandbox projector mirrored in the display settings in control panel.

 

SETUP SOFTWARE

The next parts are to install the AR Sandbox, Vrui and Kinect software. I have actually done this two ways now and both are pretty straight forward. The first was to follow a simplified set of instructions for Ubuntu linked here. This works fine, is straight forward, but will end up on older Sandbox software (V1.6), updating this setup is not so simple. I recommend following the steps below instead and you will be fully up to date on version 2.3. Your choice though.

First step is to set up Vrui. Bring up a terminal in Linux and enter the following, pressing enter after each line:

cd ~
wget http://idav.ucdavis.edu/~okreylos/ResDev/Vrui/Build-Ubuntu.sh
bash Build-Ubuntu.sh

You may be asked for your user password and to answer ‘Y’ to a question. This will take a couple of minutes to complete. At the end you’ll see a spinning globe in a window. Close this window and return to the terminal.

Next up is to install the Kinect package. In terminal enter the following, pressing enter after each line:

cd ~/src
wget http://idav.ucdavis.edu/~okreylos/ResDev/Kinect/Kinect-3.2.tar.gz
tar xfz Kinect-3.2.tar.gz
cd Kinect-3.2
make
sudo make install
sudo make installudevrules

Then it is the AR Sandbox software. In terminal enter the following, pressing enter after each

cd ~/src
wget http://idav.ucdavis.edu/~okreylos/ResDev/SARndbox/SARndbox-2.3.tar.gz
tar xfz SARndbox-2.3.tar.gz
cd SARndbox-2.3
make

This will now have installed the sandbox software and you can connect your Kinect if it is not already connected. Check for a green flashing light on the front of it.

We now need to get the Kinect calibration info from the firmware. In terminal enter the following, pressing enter after:

sudo /usr/local/bin/KinectUtil getCalib 0

We can now finally get some output from the Kinect. This next bit is the point where I finally attached the kinect mounting to the frame. In terminal enter the following, pressing enter after:

cd ~/src/SARndbox-2.3
RawKinectViewer -compress 0

You will see then two views from the Kinect camera. The left side is the depth information it sees and the right is the visible light view. Align your Kinect so it can see all of the box below. 3 – 4 ft is kinda good distance for our size box and it does not matter if you see the box frame and the floor around it.

 

CALIBRATION

You will now need to tell the software some things about your setup and do some calibration. These steps are super important to be done right, just go slow and no problem to repeat any of them if something goes screwy. I have linked a youtube video multiple times below from Oliver Kreylos that shows the calibrarion process. I’ve provided more info bu make sure to watch this video.

First is to calculate the base plane distance. Now actually we are not calculating the base but the top of the sandbox. So place the lid you made on top of the sandbox and follow the instructions in this video up to the 4:00 minute mark:

https://www.youtube.com/watch?v=EW2PtRsQQr0

The video reaches 4 mins when you have, in the terminal window, two lines appeared. Take the second line, highlight it and copy. Then if you are using Linux Mint enter the following into terminal:

cd ~/src/SARndbox-2.3
pluma etc/SARndbox-2.3/BoxLayout.txt 

Or if you are using Ubuntu enter the following into terminal:

cd ~/src/SARndbox-2.3
gedit etc/SARndbox-2.3/BoxLayout.txt 

Replace the top line in this txt file with the line you just copied from the terminal. Also then replace the equals sign with a comma. Your line will look something like this:

(-0.0076185, 0.0271708, 0.999602), -98.0000

Save and exit the text editor.

Next, take the cover off the sandbox. We will now measure the corners of where the sand reaches. The same video below from 4:10. If you watch to 10:10 it will go through copying the bits that I’ll mention below as well.

https://www.youtube.com/watch?v=EW2PtRsQQr0

So now in terminal (as shown in the video) you need to copy the 4 new lines that have appeared. These will replace the 4 lines in the boxlayout.txt file. So same as before in Linux Mint:

cd ~/src/SARndbox-2.3
pluma etc/SARndbox-2.3/BoxLayout.txt

Or in Ubuntu:

cd ~/src/SARndbox-2.3
gedit etc/SARndbox-2.3/BoxLayout.txt

And paste your 4 lines over the last 4 lines of the file, replacing them. There is no commas or anything to change this time. Save the file and exit.

Your projector is probably already on. If not switch it on. If you want to see a nice grid over your sand to check it still fits ok then enter into terminal (make sure it’s full screen when it appears):

XBackground

The final step is to calibrate the projector. This takes a little time to do and you will need a little circle with a cross drawn along it’s vertical and horizontal diameters. An old cd covered in paper works well for this. It should look a little like this:

Calibration Tools

image from system76.com

To start the calibration enter the following into terminal (replace the width and height with the resolution of your projector eg. ‘CalibrateProjector -s 1024 768’:

cd ~/src/SARndbox-2.3
./bin/CalibrateProjector -s <width> <height>

The instructions for what to do now are in the video we have been using but I found this free hand method did not work reliably for me. It’s still worth watching the video from 10:10 onwards. But I found it better to do 3 full passes of the points at low, middle and high points (36 points). For better info on this projector calibration check out: https://system76.com/weekend-project/arsandbox and scroll down to section 7. But here below is the video, as I say from 10:10

https://www.youtube.com/watch?v=EW2PtRsQQr0

After doing this you should be able to move the disk around and it will show the red crosshair closely tracking the cross on the disk. If it is then you are done calibrating.

You will need to then, through a little trial and improvement, work out the height for where the sandbox shows the water level when you dig. To do this when you run the sandbox software, as shown below, then see where the water level is (probably above the sand level at first) and then go back to the boxlayout.txt file we edited earlier and adjust the negative value in the top line up or down until it is where you want it. Ours is at -90 but yours will be different.

 

Launch Sandbox Software

To launch the Sandbox software enter into terminal:

cd ~/src/SARndbox-2.3
./bin/SARndbox -uhm -fpv

Then it is time to, hopefully, marvel at what you have created :-).

Other bits:

To make rain in your sandbox hold you hand flat about a foot higher than the sand with fingers spread out. If the evaporation rate is too slow then try using SARndbox –uhmfpvevr -0.2 to increase it a bit.

Any questions or problems let me know (quicker answers usually through twitter.