DeforMe: Visualization of Tangential Surface Deformation using Invisible Textures

Abstract

We propose a projection-based mixed reality system that visualizes the tangential deformation of a nonrigid surface by superimposing graphics directly onto the surface by projected imagery. The superimposed graphics is deformed according to the surface deformation. To achieve this goal, we develop a computer vision technique that estimates the tangential deformation by measuring the frame-by-frame movement of an infrared (IR) texture on the surface. IR ink, which can be captured by an IR camera under IR light but is invisible to the human eye, is used to provide surface texture. Consequently, the textures do not degrade the image quality of the augmented graphics. The proposed technique measures surface motion between two successive frames individually. Therefore, it does not suffer from occlusions caused by interactions and allows various interactions (e.g., touching, pushing, pulling, and pinching). The moving least squares (MLS) technique interpolates the measured result to estimate denser surface deformation. The proposed method relies only on the apparent motion measurement; thus, it is not limited to a specific deformation characteristic, but is flexible for multiple deformable materials, such as viscoelastic and elastic materials. Experiments confirm that the proposed method can visualize surface deformation of various materials by projected illumination even when a user’s hand occludes the surface from the camera.

Projection-based mixed reality (MR) refers to techniques in whichcomputer graphics are projected on to real objects, to create a realistic appearance without requiring refabrication or user-worn equip-ment. These techniques allow us to generate the illusion of virtualmodification of the physical surface, and create a realistic repre-sentation of a virtual reality in the real world. Moreover, with aflexible surface (e.g., deformable surfaces, malleable objects), theyallow the user to interact with the projected graphics while obtain-ing tactile feedback. Several investigations have explored meth-ods of projecting graphics on deformable displays or tangible ob-jects, by measuring the surface deformation using, for example, adepth sensor. However, they are not appli-cable to tangential deformations that can appear on a surface in ahorizontal direction during interactions such as touching, pinching,and pulling, even by the force of gravity. To address this limitation, we propose a new projection-based MRsystem to reveal the deformation of the surface as realistically as possible, calledDeforMe. With our system, the projected graphicscan be deformed according to the deformation of the real object. Unlike previous attempts, which utilized the physical characteris-tics of the materials, or depth information, our system has the advantage of performingreal-time estimations of the tangential deformation flows with onlya small number of feature points, and is not limited to the charac-teristics of the physical models. It can, therefore, extend the application field of projection-based MR.

Related Publications

Parinya Punpongsanon, Daisuke Iwai, and Kosuke Sato. Projection-based Visualization of Tangential Deformation of Nonrigid Surface by Deformation Estimation Using Infrared Texture. Virtual Reality, Springer London, Vol. 19(1), pp. 45-56. March 2015.

Parinya Punpongsanon, Daisuke Iwai, and Kosuke Sato. DeforMe: projection-based visualization of deformable surfaces using invisible textures. In Proceedings of ACM SIGGRAPH Asia 2013: Emerging Technologies (e-Tech). Article 8, Hongkong, China, November 2013. Demonstration

Parinya Punpongsanon, Daisuke Iwai, and Kosuke Sato. Projection-based Mixed Reality for Deformable Surfaces. Talked in the 6th Korea-Japan Workshop on Mixed Reality (KJMR2013), Okinawa, Japan, February 2013. Best Presentation Award

Parinya Punpongsanon, Daisuke Iwai, and Kosuke Sato. 表面変形のビジュアルトラッキングによる粘弾性曲面上での射影に基づく複合現実感. 第88回パターン計測シンポジウム, 計測自動制御学会, ページ 35-40, 2012年12月.