[quantumduck]

It's easy to simulate the robotic manipulators - the dynamics are well known and they are rigid objects.

On the other hand, simulating say skin/muscle/blood or anything that closely resembles human body is near impossible. Without that, a simulator is pretty much useless and it'll probably easier to train the surgeons on real robot + some animal like pig

[kingkawn]

Not impossible, there are some good new algorithms for soft tissue deformation that aren't CPU killers like even a few years ago. Most of the surgical sim software out now is either so basic that it isn't good for much beyond cursory anatomical review, or it requires some major hardware installed in a permanent space with limited use-cases that nobody ever bothers to use because at the end of a resident shift they just want to go home and sleep.

I'm working for a small company that is trying to bridge the gap and make something good that can run on the consumer VR/AR hardware that's coming out in the next year or three. Lots of interesting problems to solve.

[iancmceachern]

It's a solved problem

https://www.intuitive.com/en-us/products-and-services/da-vin...

[lostdog]

Do you believe the marketing of this company?

In my experience in a related field, simulators are about 2% as useful as advertised. The cousin thread explains how difficult building a flesh simulator is, so I wouldn't expect surgical experience on a simulator to be very useful.

[iancmceachern]

I've used it, it's pretty good.

[catgary]

I’m not even thinking of anything as grandiose as a full simulation of the human, it could be as simple as mimicking the movements from a previous surgery using an overlay of the simulated robot arms.

[deelowe]

Is a physical object even required? Why can't it be like a flight sim and just be controls and some screens?

[quantumduck]

What do you mean? In a flight simulator you have your controls controlling a plane (right body, dynamics are well known) in the sky (simple environment, dynamics are reasonably well known, even with different atmospheric conditions like wind). The output is visualized on the screens as a rendering of the scene.

In the case of robotic surgery simulator, you are using the controls to control the arm, but to interact with what? If you just want to move the arm around and maybe interact with some rigid objects sure that's easy. Would that add any training value to the surgeon? Probably not. You can get value only when the simulator includes a simulation of something the surgeon would have to face eventually - organic mass of the human body. Simulating that is hard, and I doubt anyone would invest much into it when you can train surgeons on alternative physical objects like pigs.

[soperj]

it would control a software arm operating on a software body, just like a flight sim controls a software plane.

[sebmellen]

It is nearly impossible to accurately simulate a “software body“.

[manmal]

Why? Couldn’t one model the various tissues and bones, fluids, surface tensions, resistance to cutting etc?

[jpeloquin]

Yes, you can produce a model incorporating nonlinear elasticity, viscosity, plasticity, frictional contact, subfailure damage, and fracture / cutting. Repeat for each tissue involved. Some of those methods are well-developed, others are not. Damage and failure in particular is poorly developed, and simulating deformable body contacts can be tricky. Then you have to estimate parameters for all of that, validate that the model produces realistic outcomes, and get it to run in real time. You can ignore most of the complexity and make approximations, like treating everything as an isotropic linear elastic material, setting cutting resistance to a fixed value per tissue, treating the scalpel like a lightsaber, and accepting that contact will be imperfectly enforced (surfaces may interpenetrate). A crude simulation can still be good enough to be useful as a learning tool though, but it tends to work more like a video game than a true physics-based simulation. We'll still eventually get the kind of accurate simulation you're asking for, probably.

[RandomLensman]

Very very many variables... For example, burnt tissue, crushed tissue, injection trauma, infected tissue plus variable anatomy.

[musingsole]

But don't we now have thousands of hours of real-use? Surely in the age of Dall-e, a system to use that as input to ape if not fully simulate the surgical environment is within reach.

[quantumduck]

While DALL-E might seem like something out of the world, what it outputs at the end of the day is pretty basic: a 2D image.

To run a high-fidelity simulator of human body that is useful for surgeons, you need a LOT more. And I doubt it can be data-driven. Data-driven simulators for things like autonomous cars are just coming up, and these are way simple as the agent (the autonomous car) doesn't directly change the environment when it's driving around (as in, you don't have to simulate the car colliding into a traffic pole and then breaking it, etc.)

To simulate a human body, you need to be able to capture the material properties of different layers, some of which are fluids, and then also the interaction between them and how different organs react to a surgical operation. It's a very hard problem.

Why do all that when you have animal corpses readily available? There's no need to create a problem that doesn't exist.

[hansword]

Content note: surgery gore

a) thousands of hours is far far far too little data. This is a total ballpark estimate, with only knowledge of the ML and basically zero knowledge of the surgery side, but I would expect three to six orders of magnitude greater, i.e. millions to billions of hours necessary to train a machine learning model to do something like that, to a standard.

b) the big problem is not the procedures themselves, but edge cases and things going a little bit wrong.

c) you are fine with tiny flukes in your generated images, but you won't be very happy with an ML model whose tiny flukes lead to internal bleeding.

d) unfortunately (with contemporary models) we can't easily explore latent space, which might possibly contain regions corresponding to catastrophic robot arm movements. the chance isn't that high, but it's not a chance most'd be willing to take at this point.

[ska]

I think this overestimates what Dall-E is actually doing, and underestimates the complexity of the task.