A project funding provided by:



P. Punpongsanon, D. Iwai, and K. Sato.
SoftAR: Visually Manipulating Haptic Softness Perception
in Spatial Augmented Reality.
IEEE TVCG ’15 (Proceedings of IEEE ISMAR ’15).

DOI     Paper     Video     Poster    

P. Punpongsanon, D. Iwai, and K. Sato.
A Preliminary Study on Altering Surface Softness Perception
using Augmented Color and Deformation.

DOI     Paper    


2015-tvcg-softar-slide-1 2015-tvcg-softar-slide-2 2015-tvcg-softar-slide-3 2015-tvcg-softar-slide-4 2015-tvcg-softar-slide-5 2015-tvcg-softar-slide-6 2015-tvcg-softar-slide-7 2015-tvcg-softar-slide-8 2015-tvcg-softar-slide-9 2015-tvcg-softar-slide-10 2015-tvcg-softar-slide-11 2015-tvcg-softar-slide-12 2015-tvcg-softar-slide-13 2015-tvcg-softar-slide-14 2015-tvcg-softar-slide-15 2015-tvcg-softar-slide-16 2015-tvcg-softar-slide-17 2015-tvcg-softar-slide-18 2015-tvcg-softar-slide-19 2015-tvcg-softar-slide-20 2015-tvcg-softar-slide-21 2015-tvcg-softar-slide-22 2015-tvcg-softar-slide-23 2015-tvcg-softar-slide-24 2015-tvcg-softar-slide-25 2015-tvcg-softar-slide-26 2015-tvcg-softar-slide-27 2015-tvcg-softar-slide-28 2015-tvcg-softar-slide-29 2015-tvcg-softar-slide-30 2015-tvcg-softar-slide-31 2015-tvcg-softar-slide-32 2015-tvcg-softar-slide-33 2015-tvcg-softar-slide-34 2015-tvcg-softar-slide-35 2015-tvcg-softar-slide-36 2015-tvcg-softar-slide-37 2015-tvcg-softar-slide-38 2015-tvcg-softar-slide-39 2015-tvcg-softar-slide-40 2015-tvcg-softar-slide-41 2015-tvcg-softar-slide-42 2015-tvcg-softar-slide-43 2015-tvcg-softar-slide-44 2015-tvcg-softar-slide-45 2015-tvcg-softar-slide-46 2015-tvcg-softar-slide-47 2015-tvcg-softar-slide-48 2015-tvcg-softar-slide-49 2015-tvcg-softar-slide-50 2015-tvcg-softar-slide-51 2015-tvcg-softar-slide-52 2015-tvcg-softar-slide-53 2015-tvcg-softar-slide-54 2015-tvcg-softar-slide-55 2015-tvcg-softar-slide-56 2015-tvcg-softar-slide-57 2015-tvcg-softar-slide-58 2015-tvcg-softar-slide-59

Visually Manipulating Haptic Softness Perception in Spatial Augmented Reality

We present SoftAR, a novel spatial augmented reality (AR) technique based on a pseudo-haptics mechanism that visually manipulates the sense of softness perceived by a user pushing a soft physical object. Considering the limitations of projection-based approaches that change only the surface appearance of a physical object, we propose two projection visual effects, i.e., surface deformation effect (SDE) and body appearance effect (BAE), on the basis of the observations of humans pushing physical objects. The SDE visualizes a two-dimensional deformation of the object surface with a controlled softness parameter, and BAE changes the color of the pushing hand. Through psychophysical experiments, we confirm that the SDE can manipulate softness perception such that the participant perceives significantly greater softness than the actual softness. Furthermore, fBAE, in which BAE is applied only for the finger area, significantly enhances manipulation of the perception of softness. We create a computational model that estimates perceived softness when SDE+fBAE is applied. We construct a prototype SoftAR system in which two application frameworks are implemented. The softness adjustment allows a user to adjust the softness parameter of a physical object, and the softness transfer allows the user to replace the softness with that of another object.

Figure 1: Overview of SoftAR application. The user can manipulate an object’s softness by projecting the surface deformation and body appearance effect.

The sense of softness is an essentialhapticcue that significantly af-fects the impressions of soft objects as well as the assessment of theirmaterial qualities. Softness properties should be designed carefullyfor various soft products, particularly those that must be comfortablefor users, such as furniture (e.g., cushions and sofas), clothes (e.g.,hats and shoes), and plush toys (e.g., dolls). For soft products that useimitation materials (e.g., artificial leather and fake fur), manufacturershave pursued not only textures but also softness of products that aresimilar to those of real materials to achieve high-quality products atlow prices. The food industry also attempts to optimize the softness offood products, which can significantly affect the taste of the products.In social human-robot interaction, the softness of a robot’s skin playsan important role because it determines the close physical interactionbetween robots and humans, and the impressions of the robot.

The softness properties of the above mentioned products are gen-erally optimized through trial-and-error processes in which designerscompare different soft materials through touch. However, the numberof materials that designers can investigate is generally limited becauseof budget and space constraints. Augmented reality (AR) researchershave tackled this issue, thus allowing designers to explore a larger soft-ness parameter space with a small number of real materials by manip-ulating the perceived softness of materials based on either physicalhaptic feedback or pseudo-haptics. The latter approach vi-sually changes the degree of deformation of a soft object while a usertouches it, and this visual information is displayed on a head-mounteddisplay (HMD) worn by the user. However, such approaches requireusers to wear or hold dedicated equipment that prevents comfortableand natural interactions.

Figure 2: Body appearance effects (BAEs): (a) without BAEs (SDE only), (b) color change of fingernail (nBAE), (c) color change of finger (fBAE), (d)color change of blood vessels (vBAE), and (e) color change of entire hand area (hBAE)..

In this paper, we introduce SoftAR, a novel spatial AR technique thatvisually manipulates the sense of softness perceived by a user touch-ing a physical soft object without requiring any special devices. Weuse a consumer-grade projector to modify the appearance of a user’shand and a touched object to change the perception of softness byleveraging multi-sensory integration of visual andhapticsensations inhuman brain. Such visual stimulation affectshapticperception; how-ever, the flexibility of the visual representation of spatial AR is limited.In particular, the proposed method controls only the surface appearance of projected objects. It cannot change the surface shape three-dimensionally similar to previous work using a video see-through approach. Therefore, the main challenge is to determine a visual ef-fect that satisfies both of the following conditions: (1) it significantlyalters softness perception, and (2) it can be displayed in spatial AR.The central contribution of this work is not only the determination ofsuch a visual effect through a psychophysical study, but also that wederive a computational model that can estimate a user’s softness per-ception from the physical softness of a touched object and the visualeffect parameter. Based on the model, we have built a prototype sys-tem called SoftAR that manipulates a user’s sense of softness usingprojected imagery when the user touches a soft object. Furthermore,we have conducted user studies using the prototype to validate the pro-posed model. Note that the real objects used in this study are assumed to exhibit elastic properties.