Why a Grasp That Works in Isaac or MuJoCo Still Fails on Clear and Soft Objects

There is not one reality gap

A single sim-to-real number hides three different physics problems. A rigid opaque part fails when the pose is wrong. A transparent part fails when the depth camera reports the background instead of the surface. A soft or thin part fails when the contact patch grows, slips, or buckles in a way the contact model never had. Training longer on the first problem does not remove the second or the third. [1] [2] [3]

This site already explains sim-to-real transfer, MuJoCo, Isaac, and optical tactile sensors as separate subjects. The narrow question here is the grasp itself when the object is not a rigid opaque block. Domain randomization, a point-contact simulator, and a depth camera each cover one slice. None of them is a general fix for the other slices. [3] [2]

Transfer methods in general are already discussed. Read the sim-to-real overviewThis page stays on contact with clear and soft objects.

Randomized images transfer a pose, not a squeeze

Tobin and colleagues showed that a detector trained only on simulated RGB, with randomized textures, can localize rigid objects in the real world to about 1.5 cm in their experiment, including with distractors and partial occlusion. They then used those detections to grasp in clutter. The result is about where the object is. It is not a measurement of normal force, friction, or the moment when the object starts to slip. [3]

Randomizing color, lighting and texture changes the pixels. It does not change the compliance of a bag, the thickness of a blister pack, or the refraction through a bottle. If the simulated grasp succeeds because the object is a rigid mesh with a fixed friction coefficient, the real grasp can fail while the pose error stays inside that 1.5 cm. Quoting the localization accuracy as a picking accuracy mixes two outputs. [3]

Isaac-style synthetic rendering can feed the same kind of detector. The limit travels with the method. A photorealistic image of a bottle still needs a depth or tactile channel that agrees with the glass surface. If that channel is wrong, a higher-resolution render makes a more convincing picture of the wrong geometry. [1] [3]

Clear objects break the depth that the grasp plan trusts

ClearGrasp states the sensor failure directly. Transparent objects are common, and standard 3D sensors often return a noisy or distorted version of the surfaces behind them. The method estimates surface normals, a mask of transparent regions, and occlusion boundaries from one RGB-D frame, then repairs the depth on those regions. The training set they built has more than 50,000 synthetic RGB-D images. The real benchmark has 286 RGB-D images with geometry ground truth. [1]

Those 286 images are a vision benchmark, not a robot success rate. A repaired depth map can still be too late, too smooth, or wrong on a shape that was not in the synthetic set. The grasp that follows has to use the repaired surface, the residual error, and a gripper that may itself be larger than the error. A paper that improves depth is not a paper that measured pickup rate on a production bin. [1]

The practical failure is easy to misread on a video. The gripper closes where the depth said the object was. If the depth was the back wall of the bin, the fingers close on empty space or on the rim. Domain randomization of the label “bottle” does not move that false surface. The fix, when there is one, is a depth repair or a sensor that does not rely on the returned range alone, plus a trial count on transparent objects reported apart from opaque ones. [1] [3]

MuJoCo contact is a point, unless you build the soft body yourself

MuJoCo’s contact model is a point between two geometries, with a frame at that point. The contact can include normal force, tangential friction, and, with extra dimensions, torsional and rolling friction. The documentation notes that torsional friction is useful for modeling soft fingers and can improve the stability of a simulated grasp. That is a friction torque at a point. It is not a measured pressure image across a fingertip. [2]

Soft constraints in MuJoCo can be used to model ropes, cloth and deformable volumes. The documentation is explicit that this needs a large collection of ordinary bodies, joints, tendons and constraints acting together. A single rigid mesh with a soft contact parameter is not that collection. If the real object is a bag, a sleeve, or a cable, the simulated grasp that pinches one rigid body does not reproduce the way the real contact area grows as the material folds. [2]

Tactile sensors add a further gap. An optical fingertip images the deformation of a gel. A formed resistive or capacitive film, of the kind discussed in the Tesla application note on this site, reports an electrical change at conductor overlaps. Neither signal is the MuJoCo contact force unless someone calibrates them on the same geometry, after the cover and the forming process. Randomizing the visual domain does not perform that calibration. [2]

The simulator’s role in locomotion is a different page. MuJoCo for humanoid modelsContact for grasping is the narrower claim here.

Camera-in-the-finger sensors are explained separately. Optical tactile sensingThey do not remove the point-contact limit in simulation.

A bottle and a pouch fail in different lines of the log

Take a clear bottle in a bin of boxes. The depth channel, as ClearGrasp describes for transparent surfaces, may report the cardboard behind the glass or a noisy mix of both. A pose network trained with randomized textures can still draw a box around the bottle, because the RGB edges are visible. The grasp pose then aims at a point that is not on the glass. The fingers close, the bottle stays put or tips, and the log records a miss. That miss is a geometry error in the range image. Adding more randomized colors to the training set does not move the false surface. [1] [3]

Take a pouch or a thin bag beside it. The depth may be fine. The mesh in simulation, if it is one rigid body, does not fold when the fingers close. MuJoCo will happily solve a point contact with friction, including extra torsional friction if the model asks for a softer finger. The real pouch collapses, the contact area jumps, and the friction assumption that was a single coefficient becomes a moving patch. The log records a slip or a drop after the fingers have already “succeeded” in simulation. That miss is a contact-model error. Repairing depth, which was the right work for the bottle, does not build the chain of bodies the documentation says a cloth-like object needs. [2] [1]

A third number sometimes enters from tactile hardware. An optical gel or a formed electrical film can show that contact happened. Neither one is the simulator’s contact force unless the cell publishes the calibration. The Tesla application note on this site is a reminder of how large that gap is for a film: the public text does not include a force curve at all. A grasp stack that closes the loop on an uncalibrated taxel can stop on a threshold that means “something changed electrically,” which is not “the part is held at N newtons.” [2]

The film’s missing calibration is documented separately. Tesla tactile-array applicationDo not import it as a sim-to-real success.

The reporting habit that matches these mechanisms is dull and sufficient. One table, three rows, same gripper, same lighting, same bin furniture. Columns for raw depth, repaired depth, rigid-body simulation, and real pickup. If a cell cannot fill the transparent row and the pouch row, it does not yet have a picking result. It has a rigid-object result with extra objects in the video. [1] [3] [2]

A simulator parameter is not a calibration certificate

Teams often answer the gap by tuning one friction coefficient until the simulated success rate matches a small real set. That fit can be real for the objects in the set. It does not travel to a bottle whose depth is wrong, because the fingers never reach the surface the coefficient describes. It does not travel to a pouch whose contact area changes during the close, because a single coefficient has no area. ClearGrasp and the MuJoCo contact chapter are about different inputs. Tuning friction touches neither the false depth nor the missing chain of bodies. [1] [2]

The log line that prevents the self-deception is the input signature of the trial. Raw depth or repaired depth. Rigid mesh or a multi-body soft model. Opaque, clear, or folding. If two rows share a success rate, they are not two rows. Tobin’s 1.5 cm result stays in the opaque rigid pose column. It can inform how large a grasp pad must be for that class. It cannot be averaged with a transparent miss. [3] [1]

Count the failures by object class

A useful picking report splits at least three rows: rigid opaque, transparent or specular, and deformable or thin. Each row needs the same gripper, the same bin, and a stated trial count. A combined success rate lets the easy rigid parts carry the average. ClearGrasp’s 286-image benchmark is the right kind of separation for vision. The robot still needs the same separation for pickup, slip, and crush. [1]

[3] [1] [2]

• Publish the transparent and deformable rows even when they are worse. Hiding them inside one average is how a cell is mis-sized.
• State whether depth was raw or repaired. A grasp on repaired depth is a different system.
• If tactile feedback is in the loop, publish the calibration object and the force at which slip started.
• Do not import the 1.5 cm figure into a claim about squeeze force. It was a position error in one study.

[3] [1] [2]