[sfink]

I don't know anything about this space, but damn, this looks impressive!

Could it be used to sort trash and recycling? Could it recalibrate if gunk got on it, or as it aged? (I guess silicon is probably pretty resistant to aging.) Can it wash and de-stem a tomato?

I think I want a trackpad made out of this. How much resolution could it get? I suppose I wouldn't want to sacrifice a lot of resolution for the pressure, tilt, etc. that I am assuming this would provide.

(I said "think", because I might find out that it feels like running my finger over skin, and I'm wondering how creepy that might feel. I don't really want my laptop to have a fleshy part.)

[kibitzor]

I've worked in this space in automation with industrial grade robots and more bespoke end effectors that don't look like mainstream robots, but fulfil specific needs. Responding to some of your questions with how I could see the above touch sensor helping:

Trash sort and recycling: Not many robots here, majority of sorting takes advantage of object material properties. Some companies tried to add delta robots to keep up with the high rates required to even approach profitability, but they weren't good enough. Maybe some municipalities or universities that have lots of funding could justify adding robots, but it's just hard to financially justify.

Recalibration: I'm curious what the developers have for handling reduced magnetic fields over time along with gunk. Silicone is washdown rated, but anything soft at high throughput with parts will start to wear out and change pickup characteristics.

Washing and destemming a tomato is more of a problem to solve now that will need another 10+ years of price reductions in robot+end effector costs and increased efficiency before it beats bulk washing and hand-destemming (or crude machine work). Maybe it'll be a grad-student's project for a theoretical future home-bot

The Lenovo TrackPoint is likely already 95% of what you'd need from a trackpad, but this touch sensor is likely not even focused at that market.

Things I see useful for this robot touch sensor:

* Simpler version that detects part presence, is just a Boolean feedback of "part detected" which can stick on existing end effectors. This is often handled by load calculations of the robot to detect if it has a part, but could also detect if a part has substantially "moved" while it's been gripped, sending a signal to the robot to pause

* Harder to suggest items for food as soft grippers (inflatable fingers) will grip at the precise pressure that they're inflated, reducing the need for sensitive feedback. The application for this touch sensor would be food that needs a combination of different pressures to properly secure something, can't think of a great example

* Hard to also suggest places where this sensor would help with fine alignment, as major manufacturers have motor and arm feedback with WAY more sensitivity than the average person would realize, google Fanuc " Touch Sensing". But, this could help when the end effector is longer and it's harder for the joints to detect position

* Fabric manipulation. Fabric is just a hard problem for robots, adding in more information about the "part" should be helpful. Unlocking more automations for shoe manufacturing at reasonable prices is a big wall

[raunaqmb]

This is a very insightful summary, thank you! A few things to add about AnySkin that might be relevant:

- AnySkin expressly handles wear and gunk by being replaceable. So if it wears out, and you have a heuristic or learned model for the old skin, it will work pretty well on the new skin! We verify this through an analysis of the raw signal consistency across skins, as well as through visuotactile policies learned using behavior cloning. We found swapping skins to work for some pretty precise tasks like inserting USBs and swiping credit cards.

- Could definitely be used for part motion detection

- Soft, inflatable grippers are effective, but often passive. AnySkin is not just soft, but also offers contact information from the interaction to actively ensure that blueberry doesn't get squished!

- This sensor would be key for robots that seek to use learned ML policies in cluttered environments. Robots are very likely to encounter scenarios where they see an object they must interact with, but the object is occluded either by their own end-effector(s) or by other objects. Touch, and an understanding of touch in relation to vision becomes critical to manipulate objects in these settings.

- Industrial robots do have very sensitive motor and arm feedback. However, these systems are bulky and unsafe to integrate into household robotic technologies. Sensors like AnySkin could be used as a powerful, lightweight solution in these scenarios, potentially by integrating with some exciting recent household robotics models like Robot Utility Models.

- ReSkin, the predecessor to AnySkin, has previously been used quite effectively for fabric manipulation! (see work from David Held's group at CMU). AnySkin is more reliable as well as more consistent and could potentially improve the performance seen in prior work.

[kaibee]

> - Industrial robots do have very sensitive motor and arm feedback. However, these systems are bulky and unsafe to integrate into household robotic technologies. Sensors like AnySkin could be used as a powerful, lightweight solution in these scenarios, potentially by integrating with some exciting recent household robotics models like Robot Utility Models.

I bet having good touch sense would let you get away with much cheaper mechanical systems for the robots.

[MaxikCZ]

Industrial robots are mainly bulky because they need to be very robust and precise at almost incredible speeds. (I work with those). Its not uncommon to have a 500+ kg robot on 500kg rail (totaling 7 axes) to actuate a 1mm wide, 5cm long nozzle, and moving it at speeds of 1+ meter/second, while navigating it in a gap where it has 0.5mm space on each side of the nozzle. Consistently, all day every day.

Lots of industrial robots arent even meant to touch their work piece, yet the robustness is the only way to make the whole assembly rigid enough.

I can imagine a touch-sense equiped arm could be made way smaller (less rigidity being compensated by quick enough feedback loop), but the speeds would probably have to decrease quite a bit. Not a problem for home robots tho.

[sfink]

> Washing and destemming a tomato is more of a problem to solve now that will need another 10+ years of price reductions in robot+end effector costs and increased efficiency before it beats bulk washing and hand-destemming (or crude machine work). Maybe it'll be a grad-student's project for a theoretical future home-bot

Heh, fair. I wasn't thinking of this as a practical usage, it was just the first thing to come to mind when imagining a task requiring a lot of pressure sensitivity and a range of forces.

Then again, now that I've said it, I believe the current approach to this is to breed really hard, tasteless tomatoes and then agitate them in a vat. Perhaps we can eventually get tastier produce if robots can handle more fragile things!

Hm... or you could invert things and make a glove, then use it as a controller. (VR, or just a richer set of control dimensions for eg photo editing or something.) I guess that needs to generalize across hand shapes and sizes, not just swapping out the glove, but I'd be up for a calibration/training phase.

> * Harder to suggest items for food as soft grippers (inflatable fingers) will grip at the precise pressure that they're inflated, reducing the need for sensitive feedback. The application for this touch sensor would be food that needs a combination of different pressures to properly secure something, can't think of a great example

How do you know the right pressure without feedback? A lot of foods vary in firmness over time and ripeness. Lemons, for example. I guess most don't, as long as you're sticking to a single type of food.