Visuo-Haptic Augmented Reality

Human perception is multi-modal: the senses of touch and vision do not operate in isolation, but rather closely coupled. This observation has inspired systems that allow users to see and touch virtual objects at the same location in space. The movie on the right shows one of our systems, which allows users to see and touch a virtual car. Using a head-worn display, the user can see the virtual car (the small inset on the bottom right of the movie shows the user's view). The user can also feel the car with his right hand through a haptic device called Phantom.

An easier way is to use passive haptics: overlaying graphics on physical objects. The movie on the right shows a user holding a milk carton (prepared with visual markers) in his hand. As long as the virtual objects coincide with real objects, the illusion works. We can also display slight variations in shape; however, major changes to the shape require a different approach.

Using our visuo-haptic platform, we have implemented a painting application. Users can paint with a virtual brush on a virtual, earthenware teacup (our teacup is a traditional Japanese teacup, called chawan). Our goal was to make this interaction as easy as possible. We decided to let the users control the virtual teacup with a graspable object. Additionally, we have invented a new interaction technique to make color selection from real objects very easy.

Most recently, we are refining this system in three ways: 1) we are using Canon's newest head-worn display (VH-2007) and their newest Software Platform (MRPlatform IV). To our knowledge, there are only two labs that have access to these. 2) We are collaborating with Tokyo Institute of Technology's Sato Lab. We will soon receive a brand-new prototype of their SPIDAR haptic device. 3) We are currently performing sub-millimeter calibration of the Phantom with a CNC Mill.

Canon VH-2007 head-worn display Canon's VH-2007 head-worn display. Calibration for Phantom and CNC Mill Calibration setup for Phantom and CNC Mill. Sato Lab's SPIDAR device. Sato Lab's SPIDAR device.

People

Publications

Copyright Disclaimer: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each authors copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

The colors next to the publications denote the lab in which they were created:
  Magic Vision Lab. University of South Australia. Adelaide, Australia.
  Wearable Computer Laboratory. University of South Australia. Adelaide, Australia.
  Canon's Leading-Edge Technology Research Headquarters. Human Machine Perception Laboratory. Tokyo, Japan.
  Fachgebiet Augmented Reality. Munich University of Technology. Munich, Germany.
  Fachgebiet Computer Aided Medical Procedures. Munich University of Technology. Munich, Germany.

Conferences

2013
Kinect for Interactive AR Anatomy Learning Meng, M., Fallavollita, P., Blum, T., Eck, U., Sandor, C., Weidert, S., Waschke, J., and Navab, N.
Kinect for Interactive AR Anatomy Learning
Poster in Proceedings of the IEEE International Symposium on Mixed and Augmented Reality, pages 277-278, Adelaide, SA, Australia, October, 2013. Best Poster Honourable Mention
poster | bib

HARP: A Framework for Visuo-Haptic Augmented Reality Eck, U., and Sandor, C.
HARP: A Framework for Visuo-Haptic Augmented Reality
Poster in Proceedings of the IEEE Virtual Reality Conference, pages 145--146, Orlando, FL, USA, March, 2013.
PDF | poster | bib

BurnAR: Involuntary Heat Sensations in Augmented Reality Weir, P., Sandor, C., Swoboda, M., Nguyen, T., Eck, U., Reitmayr, G., and Dey, A.
BurnAR: Involuntary Heat Sensations in Augmented Reality
In Proceedings of the IEEE Virtual Reality Conference, pages 43--46, Orlando, FL, USA, March, 2013.
PDF | video | bib

2012
ClonAR: Rapid Redesign of Real-World Objects Csongei, M., Hoang, L., Eck, U., and Sandor, C.
ClonAR: Rapid Redesign of Real-World Objects
Poster in Proceedings of IEEE International Symposium on Mixed and Augmented Reality, pages 277--278, Atlanta, USA, November, 2012. Best Poster Honourable Mention
PDF | poster | video | bib

BurnAR: Feel the Heat Weir, P., Sandor, C., Swoboda, M., Nguyen, T., Eck, U., Reitmayr, G., and Dey, A.
BurnAR: Feel the Heat
Poster in Proceedings of IEEE International Symposium on Mixed and Augmented Reality, pages 331--332, Atlanta, USA, November, 2012.
PDF | poster | video | bib

2007
Exploring Visuo-Haptic Mixed Reality Sandor, C., Kuroki, T., Uchiyama, S., and Yamamoto, H.
Exploring Visuo-Haptic Mixed Reality
Technical Report PRMU2006-199 IEICE, Tokyo, Japan, 2007.
PDF | bib

Visuo-Haptic Systems: Half-Mirrors Considered Harmful Sandor, C., Uchiyama, S., and Yamamoto, H.
Visuo-Haptic Systems: Half-Mirrors Considered Harmful
WHC '07: Proceedings of the Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, pages 292--297, Tsukuba, Japan, IEEE Computer Society, 2007.
PDF | video | bib

Theses

2012
Evaluation of Psychophysical Effects in High Quality Augmented Reality Environments Weir, P.
Evaluation of Psychophysical Effects in High Quality Augmented Reality Environments
October, 2012.
bib

2011
HARP: A Framework for Visuo-Haptic Augmented Reality Research Projects Eck, U.
HARP: A Framework for Visuo-Haptic Augmented Reality Research Projects
October, 2011.
bib

Patents

2006
Kuroki, T., Sandor, C., and Uchiyama, S.
Information Processing Method and Apparatus. Japanese patent 2006118446 (Filed 5/2006).
May, 2006.
bib

Sandor, C., Kuroki, T., and Uchiyama, S.
Information processing method and device for presenting haptics received from a virtual object. Japanese patent 2006117732 (Filed 4/2006). Patent in China, Europe, and US 8,378,997 (Filed 19 April 2007). Available online at google patent search.
April, 2006.
bib

Sandor, C., Kuroki, T., and Uchiyama, S.
Information Processing Method and Apparatus. Japanese patent application P2007-293412A (Filed 4/2006)
April, 2006.
bib

Demos

2011
BurnAR: Feel the Heat Swoboda, M., Nguyen, T., Eck, U., Reitmayr, G., Hauswiesner, S., Ranftl, R., and Sandor, C.
BurnAR: Feel the Heat
Demo at IEEE International Symposium on Mixed and Augmented Reality, Basel, Switzerland, October., 2011. Best Demo Award
PDF | bib