Tactile Genesis: Simulating Tactile Sensors at Scale to Explore Dexterous Task Learning

Abstract

Tactile sensing is critical for contact-rich dexterous manipulation, yet it remains unclear which tactile abstractions a policy needs and when richer tactile fields justify their hardware cost. This is hard to study empirically: changing sensors can alter hand geometry, compliance, and calibration, while systematic hardware comparisons are costly and difficult to control.

We present Tactile Genesis, a GPU-parallel tactile sensor simulation platform that exposes binary contact, contact depth, per-taxel kinematic force/torque, elastomer marker displacement, and proximity sensing under a common interface, with configurable placement, resolution, and models of common measurement imperfections (drift, hysteresis, dead taxels, crosstalk). It scales past 16,384 parallel environments and 1,000 taxels on a single GPU, improving throughput by 3 to 20 times over previous tactile simulators. Our elastomer model yields lower marker-displacement error than FOTS and HydroShear for both dilation and shear.

We use the platform to compare 8 tactile representations and 3 force/torque imperfection levels across three hand-task pairs. We find that richer force information is most useful for contact-rich dexterous manipulation, while depth/pressure cues are sufficient for grasp-and-lift. Sensitivity to measurement imperfections is likewise task-dependent, showing that tactile sensor design should be matched to the manipulation regime.

Platform Overview

**Overview of tactile simulation features.** (a) The sensor models approximate selected mechanical readouts, including 6-axis force/torque, elastomer displacement, and proximity signals. (b) Our sensor implementations are highly parallelized and support heterogeneous objects and randomization. (c) The simulated tactile sensors can be applied on any robot hardware surface and (d) the placement can be of arbitrary shape and resolution.

Sensor and observation interface

Sensor abstraction	Data output (N probes)	Policy observation types
`SurfaceDistanceProbe`	distance to tracked surfaces	reward metric only
`ContactProbe`	contact bits, N	`bool`, per-link `agg_bool`
`ContactDepthProbe`	penetration depths, N	`depth`, per-link `agg_force`
`KinematicTaxel`	force and torque, N×6	`force`, `force_torque`
`ProximityTaxel`	proximity-derived force and torque, N×6	`proximity`
`ElastomerTaxel`	marker displacement, N×3	`elastomer`

Five tactile abstractions and an auxiliary surface-distance query share configurable probe layouts. The eight policy representations use per-taxel fields or compact per-link readouts; a proprioception-only baseline (none) completes the comparison.

Sensors

Six readouts share one probe layout and one configurable interface. A layout is a set of positions, normals and sensing radii on any link of any robot; a readout decides what those probes report. Swapping --sensors=actual-hand/force_torque for .../elastomer changes what the policy feels without touching the hand.

`ContactDepthProbe`

1 depth per probe

Penetration depth per taxel, queried from the object's signed distance field. Thresholding it with a Schmitt trigger gives the binary contact readout, and a vendor-specific tactile image can be rendered from the depth map.

`KinematicTaxel`

3 or 6 per probe

A spring-damper normal force with shear from the relative surface velocity, and optionally the local torque, which is what resolves twist about a contact patch.

`ElastomerTaxel`

3 per probe

Marker displacement on a gel pad: dilation under indentation blended with a long-range term, and anchored shear that latches to the substrate and slips.

`ProximityTaxel`

6 per probe

Force and twist derived from a sampled point cloud of nearby objects, so the field responds before contact as well as through it. It models geometric proximity rather than a device's electrical response.

`SurfaceDistanceProbe`

1 distance per probe

The nearest distance from each probe to the tracked surfaces. Not a policy observation in this study; it shapes the reward that encourages contact.

`TemperatureGrid`

voxel field per link

A degrees-Celsius field over a voxelized link, integrating contact conduction through a harmonic-mean interface conductivity, spectral diffusion, and optional generation and radiation.

Measurement imperfections

Every readout supports the same imperfections: read delay and jitter, noise, bias, drift, quantization, sensing-radius and gain variation, dead taxels, spatial crosstalk, and single-Maxwell hysteresis. The clean, mild and strong profiles are spelled out in conf/sensor/tactile_params.yaml, and a policy can be trained and evaluated under any of them.

The clip shows a kinematic force field under the strong profile: the same contact, read through a degraded sensor.

Performance

Each sensor type benchmarked in a fixed scene of ten stacked cubes, so the measurement isolates sensing cost from task cost. Switch panel, metric and GPU; hover a point for its value.

Throughput and memory benchmark for each simulated sensor type — **Performance benchmark for each simulated sensor type.** The sensors parallelize beyond 16,384 environments on a single RTX 5090, at over 600,000 environment steps per second. Adding more sensors or noise parameters costs about 8% throughput for most sensors; the elastomer sensor costs 38%, since local displacement effects are computed per sensor. With environments fixed at 1024, throughput and memory hold up past 10,000 taxels per hand. The elastomer sensor tracks every object point in contact and so scales less well with point-cloud size, but point clouds larger than about 6,000 are not typically needed for dexterous tasks.

Comparison with Previous Simulators

Prior-work values are the numbers each paper reports; ours are measured from the logs committed in this repository.

Comparison with TacSL, HydroShear and TacMap — **Comparison with previous work.** (a,b) Our `KinematicTaxel` force field and performance compared with TacSL at 10×10 and 100×100 taxels; our throughput is about 3× higher, and our implementation runs at 16,384 environments without exhausting memory. (c) A trained hand policy with five 5×4 elastomer arrays (100 taxels) reaches 1.6× HydroShear's reported throughput at 1024 environments, where HydroShear uses one 7×9 array (63 taxels). (d) With five sensors and more than 1,000 taxels each, our depth-query throughput is 20× TacMap's and uses 7× less GPU memory per environment.

Elastomer Marker Motion

Real GelSight marker motion compared with simulated elastomer sensors

Relative RMSE against the real GelSight (lower is better)
	FOTS	HydroShear	Ours
Dilate	0.514	0.403	0.329
Shear	0.210	0.217	0.174

Elastomer marker motion comparison. Marker displacement fields on a real GelSight under dilation and shear, compared with FOTS, HydroShear, and our ElastomerTaxel. Each simulator's parameters are fitted before the error is measured.

Hands and Layouts

The same readouts attach to any hand. Each hand carries a layout derived from its real device alongside synthetic whole-hand layouts at roughly 128 and 512 probes, so taxel budget is an experimental variable rather than a hardware commitment. Pick a hand and a layout.

Rendered placements

Radii are half the median nearest-neighbour spacing per link: a simulation footprint, not a vendor calibration. Every layout records its region counts, grid shapes and source hashes in layout_metadata.json, and regenerates with scripts/generate_synthetic_tactile.py.

Teacher–Student Distillation

Privileged teacher and tactile student architecture — A privileged teacher trains on object state and proprioception with PPO. Each tactile student is then distilled from that same teacher, seeing only proprioception, the goal, and one tactile representation, so the only thing that changes across a comparison is what the student can feel.

Tactile Representation Study

Three deliberately different hand-task pairs act as design probes: reposing with the Wuji hand, continuous in-finger rotation with Sharpa, and grasp-and-lift with Allegro. Holding the policy, task and hand fixed and varying only the tactile abstraction is what the platform is for; the sweep below is one instance of it.

What it shows is that the useful tactile signal moves with the manipulation regime rather than with the richness of the sensor. Grasp-and-lift is best served by depth cues, reposing by a compact aggregate force readout, and continuous in-finger rotation by taxel-resolved force/torque, while on grasp-and-lift the kinematic force representations fall below the proprioception-only baseline that depth beats. Sensitivity to measurement imperfection is task-dependent in the same way. A single sensing recommendation would not have transferred across these three pairs.

**Tactile representation and imperfection study across three hand-task pairs.** Privileged teachers for Wuji reposing, Sharpa rotation, and Allegro grasp-and-lift supervise separate students for each observation condition. Taxel layouts show example fields; plots compare representations with proprioception and clean, mild, and strong force/torque imperfections. Tactile observations provide modest gains for reposing and substantial gains for rotation. For grasp-and-lift the benefit depends strongly on representation: depth, aggregate force, and elastomer exceed proprioception, whereas kinematic force/torque substantially underperforms it.

Representation and Imperfection Curves

Every tactile student is distilled from the same privileged teacher for its hand and task, so the only thing that changes across a curve is what the student can feel. Both views show checkpoint performance over distillation iterations: settled-goal success for repose and lift, and retained axial speed for continuous rotation. The first compares tactile types; the second holds the type at force_torque and varies the sensor imperfection profile. Wuji elastomer uses 250 internal surface-query samples instead of the default 500, without changing its taxel layout or observation dimension. Pick a task, hover a mark for its value, or show the table. Lift is scored here over the 17 training objects, so these curves sit on a different scale from the paper's 50-unseen-variant panel.

Beyond Contact Mechanics

The temperature sensor returns a degrees-Celsius field over a voxelized link, integrating contact conduction through a harmonic-mean interface conductivity, spectral diffusion, and optional generation and radiation. Material identity is not observable from contact forces alone, but heat flow depends on conductivity and heat capacity, so the two cues are close to independent and combining them is what separates the materials.

Confusion matrices for material classification from force, temperature, and both — **Material classification from simulated fingertip slides across eight materials** (1.5 s per trial, 64 trials per material). An SVM is trained on force features, temperature features, or both, evaluated with stratified five-fold cross-validation. Neither modality alone separates the materials; together they reach 73.6%.

Contact Audio Demonstration

Spectrogram of simulated contact audio across materials — **Contact audio sensor output.** As the ball bounces and rolls along the different materials (wood, metal, glass), the contact audio sensor synthesizes sound based on contact physics and the material properties, which the rigid-body engine alone cannot capture. Turn your sound on. The contact-audio sensor is a prototype extension and is not part of the released Genesis v1.4.1 build, so this demonstration ships as a recording.