A project funding provided by:

grant-jsps-kakenhi

Publication

P. Punpongsanon, D. Iwai, and K. Sato.
Projection-based Visualization of Tangential Deformation of Nonrigid Surface by Deformation Estimation Using Infrared Texture.
Springer VR ’15.

DOI     Paper    

P. Punpongsanon, D. Iwai, and K. Sato.
DeforMe: Visualization of Tangential Surface Deformation
using Invisible Textures.
ACM SIGGRAPH Asia ’13 (e-tech).

DOI     Paper    

P. Punpongsanon, D. Iwai, and K. Sato.
Projection-based Mixed Reality for Deformable Surfaces.
KJMR ’13 Best Presentation Award

Slides

2013-siggraphasia-Deforme-Slide1 2013-siggraphasia-Deforme-Slide2 2013-siggraphasia-Deforme-Slide3 2013-siggraphasia-Deforme-Slide4 2013-siggraphasia-Deforme-Slide5 2013-siggraphasia-Deforme-Slide6 2013-siggraphasia-Deforme-Slide7 2013-siggraphasia-Deforme-Slide8 2013-siggraphasia-Deforme-Slide9 2013-siggraphasia-Deforme-Slide10 2013-siggraphasia-Deforme-Slide11 2013-siggraphasia-Deforme-Slide12 2013-siggraphasia-Deforme-Slide13 2013-siggraphasia-Deforme-Slide14

DeforMe:
Visualization of Tangential Surface Deformation using Invisible Textures




We propose a projection-based mixed reality system that visualizes the tangential deformation of a nonrigid surface by superimposing graphics directly onto the surface by projected imagery. The superimposed graphics is deformed according to the surface deformation. To achieve this goal, we develop a computer vision technique that estimates the tangential deformation by measuring the frame-by-frame movement of an infrared (IR) texture on the surface. IR ink, which can be captured by an IR camera under IR light but is invisible to the human eye, is used to provide surface texture. Consequently, the textures do not degrade the image quality of the augmented graphics. The proposed technique measures surface motion between two successive frames individually. Therefore, it does not suffer from occlusions caused by interactions and allows various interactions (e.g., touching, pushing, pulling, and pinching). The moving least squares (MLS) technique interpolates the measured result to estimate denser surface deformation. The proposed method relies only on the apparent motion measurement; thus, it is not limited to a specific deformation characteristic, but is flexible for multiple deformable materials, such as viscoelastic and elastic materials. Experiments confirm that the proposed method can visualize surface deformation of various materials by projected illumination even when a user's hand occludes the surface from the camera.

deforme-deformation-projection
Figure 1: With DeforMe, projected graphics are realistically deformed to match the deformed surfaces: (a) infrared-based textures are used to estimate the deformation of the surfaces, (b)(c) and (d) various deformable surfaces can be used with the system, (e) DeforMe can also be used for the case of a self-dilating object.

Projection-based mixed reality (MR) refers to techniques in whichcomputer graphics are projected on to real objects, to create a realistic appearance without requiring refabrication or user-worn equip-ment. These techniques allow us to generate the illusion of virtualmodification of the physical surface, and create a realistic repre-sentation of a virtual reality in the real world. Moreover, with aflexible surface (e.g., deformable surfaces, malleable objects), theyallow the user to interact with the projected graphics while obtain-ing tactile feedback. Several investigations have explored meth-ods of projecting graphics on deformable displays or tangible ob-jects, by measuring the surface deformation using, for example, adepth sensor. However, they are not appli-cable to tangential deformations that can appear on a surface in ahorizontal direction during interactions such as touching, pinching,and pulling, even by the force of gravity. To address this limitation, we propose a new projection-based MRsystem to reveal the deformation of the surface as realistically as possible, calledDeforMe. With our system, the projected graphicscan be deformed according to the deformation of the real object. Unlike previous attempts, which utilized the physical characteris-tics of the materials, or depth information, our system has the advantage of performingreal-time estimations of the tangential deformation flows with onlya small number of feature points, and is not limited to the charac-teristics of the physical models. It can, therefore, extend the application field of projection-based MR.