11. Where Responsibility Lives
The Problem: After an Incident, Only the Logs Remain
At a distribution center, an automated sorter repeatedly sends packages to the wrong destinations. Boxes pass beneath a camera, the sorting model predicts where each one should go, and the control system sends it down either the right or the left rail. One day, a series of boxes printed with addresses from a particular region are sent down the wrong rail in succession. The logistics team complains about delivery delays, while the development team checks the model's accuracy metrics. The model had scored high enough on the test set.
A log is displayed on the conference-room screen. A request came in, a prediction was produced, and a command was executed. But the log does not say why that threshold was chosen, why no human review took place, whether an operator saw the warning, or whether the same error had already been recurring in another region. The system recorded countless events, but it did not record the path of responsibility.
When an incident occurs, we habitually look for one person. We point to the developer who built the model, the operator who pressed the button, the product manager who wrote the requirements, or the administrator who approved the system. But in complex software, an outcome does not come from a single line of code. Defaults, data pipelines, deployment procedures, monitoring screens, and an organization's schedules and incentives overlap to produce a single action. Responsibility is not stored in one place in the code. Even so, the path of responsibility can be designed into the code and the system.
The Philosophical Question: Responsibility Is a Bridge Between Intention and Outcome
Responsibility is not simply a matter of asking, “Who caused this?” Causes can be traced backward in physical terms, but responsibility asks who knew what, what could have been anticipated, and where someone had the authority to intervene. These questions build a bridge between intention and outcome.
The fact that a developer did not intend to create a bug matters. But responsibility does not disappear merely because the outcome was unintended. Its weight changes depending on whether the developer knew about the risk and skipped testing anyway, ignored signals of failure, or released an interface that users could reasonably misunderstand. Conversely, placing all responsibility on the operator who pressed the final button may also be unjust. If the operator was not given enough explanation, had no authority to cancel the action, and had no channel through which to report a failure, then the action already took place inside a narrow corridor built by the system.
The engineer's task, then, is not to leave responsibility as an abstract code of ethics. Responsibility must be translated into system structure. Who approved the model? What data and version were used? What happens when the model's confidence is low? Can someone challenge an automated decision? When an incident occurs, does the system leave behind only logs, or does it also preserve the basis for the decision and the options that were available at the time?
The philosophy of responsibility is also present in the way work is broken into small tickets. A ticket titled “Implement recommendations” easily becomes a sentence that no one truly owns. Write instead, “Display the basis for each recommendation to the user, and allow them to use the service even when they reject the recommendation,” and the scope of responsibility changes. Breaking work down is a technique for making tasks smaller, but it is also a technique for drawing boundaries so responsibility does not scatter or disappear.
An Engineering Case: Designing Incident Response as a Map of Responsibility
Suppose a medical appointment system automatically determines the priority of patients' appointments. The model considers symptoms, hospital capacity, waiting time, and other factors to recommend an order. One day, a patient's urgency is rated too low, delaying their treatment. Saying afterward that “the model was wrong” does not restore anything. What is needed is a reconstruction of what inputs the system received at the time, which model version calculated which probabilities, and which rules converted those results into the final order.
That requires recording the model version and the provenance of the input data. Recommendations and confirmed decisions must be kept distinct. If a human modifies the model's recommendation, the system should preserve the before-and-after states and the reason for the change. When confidence is low or certain conditions are met, automated processing should stop and route the case to expert review. The important thing is not to accumulate more logs. It is to design the meaning of the record so that, when someone later asks, “Why was that decision made?” the system can answer.
Incident response procedures are part of responsibility too. If people try to find the cause before anything else when an outage occurs, they may delete logs or hide inconvenient facts. If the organization instead establishes, in advance, an order of operations—contain the harm, notify users, preserve data, and conduct an independent review—it can choose recovery before blame. A responsible system is not one in which mistakes never happen. It is one in which, when a mistake does happen, it is clear who must do what.
Objection and Tension: If Responsibility Is Divided, Does No One Remain Responsible?
There is a danger in saying that responsibility should be distributed across several layers. Developers point to the data team, the data team points to product, product points to customer demands, and responsibility eventually disperses like fog. This is the moment when saying that everyone bears a little responsibility turns into saying that no one bears any.
Distributed responsibility therefore needs an ultimate owner. Each team's contribution should be recorded clearly, but someone with the authority to operate and stop the system as a whole must exist. Responsibility also means more than punishment. It includes the authority to make corrections, the duty to explain, the resources to repair harm, and the time needed to change the system so the same error does not recur. To store responsibility in code does not mean replacing people with code. It means returning to people the traces of judgment that are so easily lost between code and organization.
Questions to Leave With
If your system caused an incident tomorrow, could you look at its logs and explain “what happened”? More importantly, could you say “who knew what, and at what point could something have been changed”? Have you embedded, in your work tickets, code reviews, and deployment procedures, a structure that keeps responsibility from being dumped on one individual while also ensuring that no one can escape it entirely?