Learning Human-environment Interactions using Conformal Tactile Textiles

Yiyue Luo     Yunzhu Li     Pratyusha Sharma     Wan Shou     Kui Wu     Michael Foshey     Beichen Li
NE2021
Schematic of the textile-based tactile learning platform. a, Scalable manufacturing of tactile sensing textiles using a customized coaxial piezoresistive fiber fabrication system and digital machine knitting. b, Digitally designed and automatically knitted full-sized tactile sensing wearables: i, artificial robot skin, ii, vest, iii, sock, and iv, glove. c, Examples of tactile frames collected during human-environment interactions and their applications explored using machine learning techniques.
Abstract
Recording, modelling, and understanding tactile interactions is important in the study of human behaviour and in the development of applications in healthcare and robotics. However, such studies remain challenging because existing wearable sensory interfaces are limited in terms of performance, flexibility, scalability, and cost. Here we report a textile-based tactile learning platform that enables recording, monitoring, and learning of human–environment interactions. The tactile textiles are created via digital machine knitting of inexpensive piezoresistive fibres, and can conform to arbitrary three-dimensional geometries. To ensure that our system is robust against variations in individual sensors, we use machine learning techniques for sensing correction and calibration. Using the platform, we capture diverse human–environment interactions (more than a million tactile frames), and show that the artificial intelligence-powered sensing textiles can classify humans' sitting poses, motions, and other interactions with the environment. We also show that the platform can recover dynamic whole-body poses, reveal environmental spatial information, and discover biomechanical signatures.
Paper

Learning Human-environment Interactions using Conformal Tactile Textiles
Y.Luo, Y.Li, P. Sharma, W. Shou, K. Wu, M. Foshey, B. Li, T. Palacios, A. Torralba, W. Matusik
Nature Electronics, 4, 193–201(2021) [Paper (External link)] [Dataset] [Code]

Selected as Nature Electronics, Volume 4 Issue 3 (March 2021) Cover Image [Link].

Video
Press
Related Project

IntelligentCarpet: Inferring 3D Human Pose from Tactile Signals
Y.Luo, Y.Li, M. Foshey, W. Shou, P. Sharma, T. Palacios, A. Torralba, W. Matusik
CVPR 2021 [Project page]

KnitUI: Fabricating Interactive and Sensing Textiles with Machine Knitting
Y.Luo*, K. Wu*, T. Palacios, W. Matusik
ACM CHI 2021 [Project page]

Learning the Signatures of the Human Grasp Using a Scalable Tactile Glove
S.Sundaram, P. Kellnhofer, Y.Li, J. Zhu, A. Torralba, W. Matusik
Nature, 569 (7758), 2019 [Project page]