Two-stage imitation learning → autonomous policy switching

Robotics case study // 004

Vision-Based Towel Folding with Imitation Learning

A real-robot cloth-manipulation system that uses separate ACT policies to flatten and fold a towel. An interpretable RGB-D terminal classifier decides when the towel is ready to switch from preparation to folding.

Cloth state changes with every contact.

Self-contact, friction, wrinkles, and occlusion make cloth difficult to model and leave a single end-to-end controller with a long, fragile horizon.

The solution

Decompose the task, then learn each skill.

The simulation establishes a controlled folding baseline. On hardware, separate ACT policies learn flattening and folding from demonstrations, while TTCC selects the correct policy from geometric RGB-D measurements.

Physical system

Built around real robot demonstrations.

A 6-DoF OpenManipulator-Y performs the task using a wrist-mounted RealSense D405 and an external D415 RGB-D camera. Leader–follower teleoperation records demonstrations as LeRobot episodes for ACT training.

OpenManipulator-YRealSense D405/D415ROS2 JazzyLeRobot

OpenManipulator-Y towel folding hardware setup with wrist and external RGB-D cameras — The physical setup used for teleoperated collection, ACT inference, TTCC state estimation, and robot execution.

System architecture

Perception selects the skill; ACT generates the motion.

The wrist camera and robot state feed the learned policies, while the external RGB-D camera independently measures whether the towel is ready for the fold. ROS2 services connect perception, policy inference, and trajectory execution.

Policy observations

Wrist RealSense D405RGB image

→

Robot feedbackJoint state

→

LeRobot inference serverACT policy

→

ROS2 control clientAction chunk

→

OpenManipulator-YExecute motion

Policy selection

External RealSense D415RGB-D frame

→

3D preprocessingPlane removal + towel mask

→

TTCCRfit + height statistics

→

Not readyFLATTEN → repeat

ReadyFOLD → handoff

Two-stage imitation learning

Two ACT policies shorten the learning horizon.

Leader–follower teleoperation records wrist images, follower joint states, and leader trajectories as LeRobot observation–action episodes. Each policy learns one coherent skill, avoiding the ambiguity of asking one network to both prepare and fold the towel.

01

60 episodes · DAgger

Flattening policy

An ACT policy reduces wrinkles and pulls the towel toward a rectangular, foldable configuration.

02

RGB-D geometry

TTCC transition

An interpretable classifier checks rectangularity and height variation, then emits FLATTEN or FOLD.

03

30 episodes

Folding policy

A second ACT policy executes the half-fold only after the towel reaches a suitable terminal state.

Implementation details

From demonstration to closed-loop execution.

The system separates data collection, training, inference, and visual policy switching into testable ROS2 components.

01 · Collect

Leader–follower demonstrations

The leader arm commands the follower while the system synchronizes wrist images, follower joint states, leader actions, and episode-control events into LeRobot-format trajectories.

02 · Train

Skill-specific ACT models

The flattening model uses 60 episodes including DAgger corrections; the folding model uses 30 focused episodes. Each transformer predicts a short action chunk instead of a single joint command.

03 · Infer

ROS2 policy service

inference_server loads the selected checkpoint and returns action chunks. kkm_control_client requests actions and publishes the resulting robot trajectories.

04 · Switch

Interpretable visual feedback

realsense_towel_metrics estimates the tabletop, segments the towel, computes geometric metrics, and publishes /towel/decision to keep flattening or hand control to the folding policy.

Tunable terminal condition classification

A transparent switch instead of a black box.

TTCC measures rectangularity fit R_fit, residual-height standard deviation σ_h, and robust height range Δh. Thresholds can be tightened for precise folding or relaxed in noisier environments.

FOLD if Rfit ≥ τrect
and σh ≤ τstd
and Δh ≤ τrange

Hardware results

Switching quality determines long-horizon success.

Autonomous TTCC switching reached 80.0% full-task success, compared with 63.3% using manual transition timing. The result isolates policy orchestration—not only individual policy accuracy—as a central design problem.

76.7%

Flattening only

93.3%

Folding only

63.3%

Manual switch

80.0%

TTCC autonomous

Wrinkled towel classified as flatten — **Keep flattening.** Rectangularity is close, but height variance and range exceed the strict thresholds.

Flat towel classified as fold — **Ready to fold.** A rectangular mask and low residual-height variation satisfy all three conditions.

Crumpled towel classified as flatten under relaxed thresholds — **Robust rejection.** Relaxed thresholds still reject a strongly non-rectangular towel.

Mostly flat towel classified as fold under relaxed thresholds — **Earlier transition.** Relaxed thresholds tolerate moderate wrinkles while preserving a foldable global shape.

Qualitative results

Real robot flattening and folding rollouts.

These rollouts show the OpenManipulator-Y executing learned cloth-manipulation actions. The first-stage policy prepares the towel; once TTCC detects a sufficiently flat, rectangular state, control passes to the folding policy.

OpenManipulator-Y learned towel manipulation rollout one — **Learned execution.** Camera observations and robot state condition action chunks generated by the ACT policy.

OpenManipulator-Y learned towel manipulation rollout two — **Policy handoff.** TTCC makes the flatten-or-fold decision from interpretable RGB-D shape metrics.

What remains difficult

Generalization beyond the demonstrated workspace.

The learned policies handle varied demonstrated towel states, but unseen materials, sizes, severe occlusion, and out-of-distribution grasps remain open problems.

Next experiment

Close the loop with recovery data.

Expand DAgger collection around failure states, randomize cloth properties and camera pose, and learn recovery behaviors while retaining TTCC as an interpretable safety gate.

Next case study

RoboMemory

Persistent agent memory for robot experiment debugging and future data collection.

View project arrow_forward