Georg Klein Home Page

April 2010: PTAM Blog

There's now a blog at wordpress for all things PTAM-related. Further updates will appear there rather than here.

Oct 2009: Microsoft

As of October 2009, I've moved to Seattle to work for Microsoft's MSN Advanced Engineering team. I will keep maintaining the PTAM sources, but updates to this page will slow and cease.

Oct 2009: Compositing for Small Cameras, revisited

We've finished the journal version of last ISMAR's paper on compositing. The updated version has a considerably more detailed camera calibration step, a better approximation of the camera's Bayer interpolation method, and it also models some on-camera effects which can be varied by the user (in-camera sharpening control, and the saturation control.) The TVCG preprint will be available soon, meanwhile some results images can be seen Click here.

June 2009: PTAM on iPhone

We've written a variant of PTAM which runs on an iPhone 3G. This still needs a lot of work - the camera's narrow field-of-view is problematic - but we can already make and track small maps. See a video on YouTube.

Update a pre-print of an ISMAR'09 paper describing this work is up. [ISMAR 2009 Paper]

Sept 2008: Improving the Agility of Keyframe-based SLAM

Our ECCV'08 paper tries to make our SLAM system more robust to rapid camera motions. We achieve this by adding a very fast frame-to-frame rotation estimator (to handle large accelerations) and adding edges to the SLAM map (so that we can still track when the images get motion-blurred). We also describe our trivial keyframe-based relocaliser: It's all about small, blurry images. We'll be demoing this system at the conference. The current results video is rather dry, I'll make a new one in Marseille and put it up in October.

[ECCV 2008 Paper]

Sept 2008: ISMAR'08

Our system won the ISMAR 2008 tracking competition. While I initially expected to have to make a big map of the entire room, I ended up using Bob's multiple-map approach, using the ECCV'08 relocaliser to automatically switch between sub-maps made for each picking station. Read Bob's ISWC paper here. Also our compositing paper (see below) got an honorable mention.

Sept 2008: PTAM Source Code Available

We've made the source code to an implementation of last year's ISMAR paper available here.

[Source code page] [ISMAR 2007 Paper]

June 2008: Compositing for Small Cameras

We've written a short paper for the year's ISMAR on compositing: how to blend virtual images onto the real video images for augmented reality. This was motivated by wanting to understand how to properly handle semitransparent objects in OpenGL (use pre-multiplied alpha, it turns out!) but we ended up trying to simulate the poor image quality you get from web-cams.

Click here to look a page with some blending comparisons.

[ISMAR 2008 Paper]

November 2007: ISMAR and source code

Our paper was well received at ISMAR, winning the best paper prize. I've made the slides and extra video material from the talk available (the slides don't contain many words, so may not be all that useful.) The video shows sequences recorded while in Japan - i.e., not in my office like all the results which went with the paper. Several of them show some of the system's failure modes.

A few people have asked if the source code for the tracking system is available. It isn't yet, but we will be making it available in the near future. It will take some time to get the code in a distributable state but I hope to release it before the end of the year. Update: We're discussing licensing possibilities with Isis Innovation (the university's technology transfer company) and deciding how best to proceed. Update Sep 2008: Code is now online, see above.

[ISMAR 2007 extra video results (54 meg xvid)]

August 2007: Parallel tracking and mapping (PTAM)

We've developed an alternative to sequential, frame-by-frame SLAM to tracking in unknown environments. Instead of rigidly linking tracking and mapping, we split these tasks into two separate threads which run asynchronously. Tracking can focus on robustness and real-time performance. Mapping is done using key-frames and standard SFM techniques (read: bundle adjustment) and so scales very differently to our previous EKF-SLAM approaches; linear complexity with map size means that we can use thousands of low-quality point features instead of a few dozen good ones. The trade-off is cubic complexity with number of key-frames, but this is acceptable for our target application (hand-held cameras in small AR workspaces.)

Results are very promising - this is easily the most accurate tracking system I've worked on, and it is way more robust than our previous SLAM system. It's not quite as robust to rapid motions as our previous edge-model/gyro systems but we're working on this (and the ability to recover from failure somewhat makes up for this.)

AR in unknown scenes is always going to be difficult without a remote expert to annotate the map. Here, we restrict ourselves to finding a dominant plane in the scene, and then running simple VR/AR games on this plane: essentially, you can have little AR critters running around on your tabletop. At present, no attempt is made to exploit the map to e.g. find occluding geometry; this is an area of future work.

The video below is the ISMAR 2007 submission material. It shows the operation of the tracker, followed by a few table-top AR examples: The first (with Cartman) merely shows off the system's ability to handle large scale changes, the second shows a lens simulation with which the user can burn the real world, and finally Darth Vader makes a return to AR in ``Ewok Rampage.''

Updated: paper now up

[ISMAR 2007 submission video (47 meg divx)]
[ISMAR 2007 Paper]

July 2007: SLAM relocalisation

Last autumn's joint work with Brian Williams and Dr. Ian Reid will be shown at ICCV 2007. Instead of trying to prevent a monocular SLAM system from ever losing track (which is rather futile) we attempt to quickly recover from failure by identifying old map features in the image. We use a modified version of EPFL's randomised tree classifier, trained on features in the map; when tracking is lost the classifier is used together with 3-point-pose to recover the system's position.

The result is a system which is far more useful for handheld SLAM than any we've had before: tracking failures (as long they are detected by the system) do not matter any more, and the system doesn't have to be treated like a raw egg. SLAM Project Page..

The video below shows the system operating in three scenarios: an art gallery and two tabletop settings. The first two show the system building a map and recovering from frequent tracking failure, the last video is a proof-of-concept AR application which shows an augmented character running around a desk.

[ICCV submission video (xvid)]
[2007 ICCV Paper]

April 2006: Edge tracking with a particle filter

Yet more edge Tracking - here we demonstrate 6-DOF edge tracking using a particle filter instead of the usual least-squares optimisations - essentially this is CONDENSATION with full 3D. This gives some robustness advantages but means more jitter. The tricky thing here was rendering and evaluating hundreds of 6DOF poses (complete with hidden-line removal) per frame; we achieve this using the graphics card, exploiting a GL occlusion extension to read likelihood scores back from the GPU.

The video below shows tracking results for four edge models: a box, a printer, the Drummond/Cipolla ship part, and a maze model used for AR. For some of these sequences the blur motion estimator described below is used. This allows vision-only tracking of the ship-part sequence, for which we needed inertial sensors back in 2002. Tracking fails for the maze model.

[28 meg divx5 avi]
[2006 BMVC Paper]

2005: A Single-Frame Visual Gyroscope

Instead of using rate gyroscopes to predict camera motion in a frame, we attempt to estimate this information from the video input by analysing the motion blur present in a single video frame. By looking at the parts of the image which remain un-blurred, blur direction and magnitude can be determined very quickly (just over 2 milliseconds per frame on a 2.4GHz machine).

This method only produces a 3-DOF answer and has a sign ambiguity; also the magnitude of blur is often inaccurate. However the system can usefully be combined with edge tracking.

[2005 BMVC Paper]

2004: Tablet-Based Augmented Reality and Occlusion Refinement

Tablet PCs offer an alternative to head-mounted displays for delivering augmented reality. We develop a tablet-based AR application which combines inside-out edge tracking and outside-in LED tracking for robust registration: the inside-out system provides a high level of registration accuracy while the outside-in system provides robustness and recovery from shake and camera occlusions.

To provide a high level of rendering quality, we look at the specific case of virtual graphics occluded by real objects for which we have a model. We show that instead of just clipping virtual graphics using the real geometry projected into the z-buffer, an individual treatment and refinement of occluding edges produces a far more convincing integration of real and virtual objects.

[15 meg divx5 avi]
[2004 ISMAR Paper]

2003: HMD-Based Augmented Reality

Edge-based visual tracking method can be applied to a head-mounted camera. A see-through head-mounted display and a novel calibration technique are used to produce prototype maintenance applications.

[2003 ISMAR Paper]

2002: Real-Time Visual Tracking

This work combines edge-based visual tracking with inertial sensors. By modifying the behaviour of edge detection according to information from rate gyroscopes, sequences with substantial amounts of motion blur can be tracked.

[2002 BMVC Paper] [2004 IVC Paper]