A machine a thousand times better than the best surgeon will still fail one day. Who answers for it?
On robot surgeons, responsibility, and the line we should draw before we have to
Imagine the machine, because it is coming in some form: a surgical system a thousand times more capable than the best human surgeon who has ever lived, available at a fraction of the cost of the worst one. Not slightly better. Not better on average with footnotes. Better in the way a calculator is better than a room full of accountants, operating on a different level of the activity entirely, steadier than any hand, more patient than any temperament, with a memory of every procedure ever recorded.
Now imagine that it fails.
Not often. Rarely, in fact, far more rarely than human surgeons fail. But it fails, and a patient is harmed, and a family sits in a room with a lawyer asking the question our legal system will eventually have to answer cleanly: whose fault was this?
The answer we reach for first will be the wrong one, because we will reach for the human categories. Malpractice. Negligence. The standard of care. Did the machine exercise the judgment a reasonable practitioner would have exercised? These words feel serious and familiar, and they are built for a world of persons, beings who can know better, who can be trained, shamed, licensed, and struck off. A machine cannot be negligent in that sense, any more than a bridge that collapses was careless. When the bridge falls, we do not put the bridge on trial. We ask who designed it, who built it, who inspected it, who signed.
That body of law already exists. It is called product liability, and it is the correct home for machine failure, for as long as the machine is a product.
Here is the line I would draw, and I would draw it now, while it is still cheap to draw: until a system is self-aware, its failures are product liability, not malpractice. Responsibility follows the chain of human decisions that produced, deployed, and profited from the tool. The manufacturer answers for the design. The hospital answers for the deployment. The physician who chose the tool and supervised its use answers for that choice the way he answers for choosing any instrument. What no one should be permitted is the moral disappearing act in the middle, the shrug that says the algorithm decided, as though a decision had occurred somewhere no human was responsible for building, buying, or switching on.
The shrug is the real danger. Not the machine. Institutions love a responsibility gap the way water loves a crack, and advanced systems are going to offer history's most elegant one: a process too complex to follow, producing outcomes no individual chose, harmful in ways everyone regrets and no one owns. If we allow "the system did it" to become an acceptable sentence, we will have invented something worse than a dangerous machine. We will have invented a machine for dissolving accountability, and every large organization on earth will buy one.
Self-awareness is the line because self-awareness is where the category changes. Malpractice, blame, moral responsibility, these presuppose a subject: someone home inside the act, capable in principle of knowing what they were doing and doing otherwise. We can argue about where that threshold lies and how we would ever test it; that argument belongs to another essay and probably to another generation. What matters for law and for conscience is that we do not hand the privileges of personhood, or its liabilities, to things that are, by our own best understanding, extraordinarily sophisticated tools. A tool's failure indicts its makers and its users. That is not a loophole. Used honestly, it is a stricter regime than malpractice, because it cannot be settled with an apology and a seminar. Products get recalled.
There is a temptation, especially among the religious, to skip the legal categories and go straight to the soul. Resist it, in this order. The surgeon's soul is not the question when the instrument fails. The question is the older, duller, more biblical one: who is answerable? Scripture's moral world runs on answerability, the steward gives an account, the watchman answers for the warning he did not sound, the master asks what was done with what was entrusted. An age of intelligent tools does not retire that pattern. It tests whether we still believe it when the tool is impressive enough to hide behind.
So let the machines come, even the miraculous ones. Let them operate, drive, diagnose, and build. And let every one of them stand in the legal and moral order as what it is: a product, with makers, sellers, and owners who answer for it, until the day something looks back at us and the categories themselves have to change.
On that day we will need different arguments. Until that day, we need this one, and we need it written down before the first family sits in the first room being told that nobody is to blame.