07. What We Can Do Is Not the Same as What We Should Do
The Problem: The Door Opened by a Single Button
A strange silence settles over the room just before a new feature is deployed. The developer brings up a performance graph and says, “We can process it in real time now.” The product manager shows a forecast for conversion rates. “Once we add this feature, we can reduce churn.” The operations lead has already set the deployment time and rollback procedure. On one side of the screen is a green button: Deploy.
Technically, it is possible. The data is there, the model is there, and the servers can handle the load. But is that enough? The fact that we can build something does not lead directly to the conclusion that we should build it. In engineering, the most dangerous sentence is sometimes the one spoken with the greatest confidence: “Because we can.”
Philosophy pauses in front of that sentence. What is the relationship between what is possible and what ought to be done? Who needs to enter the space between them? Should that space include users, workers, victims whose harm will be discovered too late, and people who do not yet have a voice?
The Philosophical Question: The Moment We Leap from Facts to Values
We often mix facts and values within a single sentence. From the fact that a model is more accurate than a person, we derive the claim that the model should replace people. From the fact that automation can reduce costs, we arrive at the conclusion that we should automate. But accuracy does not tell us what, exactly, is being predicted, and efficiency does not explain who bears the cost of that efficiency.
Engineering may look like the act of taking requirements as input and producing a system as output. In practice, it is the act of fixing choices in place. We decide which errors to tolerate, whose convenience to prioritize, and who will explain what went wrong—and then inscribe those decisions into database fields, permission structures, notification rules, and interfaces. No system is free of value judgments. There are only systems that have failed to document their value judgments.
The important question is not whether a feature is good or evil. A more uncomfortable, and more precise, question is this: In the order created by this feature, what choices do people lose? The moment a system begins deciding on someone’s behalf, the user is not merely saving time. They are handing over the opportunity to decide.
An Engineering Scene: The Person Behind the Hiring Model’s Accuracy
Suppose a company builds a model that automatically sorts job applications. Trained on the résumés and interview results of past hires, the model organizes candidates faster and more consistently than the HR team. The initial validation results are also promising. Recruiters spend less time on the work, and variation in résumé screening has decreased.
Several months later, however, a developer reviewing the logs notices a strange pattern. Graduates of certain universities and applicants trying to return to work after a career break are receiving unusually low scores. The model did not learn a rule called discrimination. It learned whom the company had hired in the past, and how the company had recorded those outcomes. Past bias had returned wearing the face of numbers.
At this point, many responses are technically available. The team could adjust the threshold, remove some features, or add fairness metrics. But what ought to be done cannot be determined by technical adjustments alone. Should recruiters be prohibited from treating the model’s recommendation as binding? Should rejected applicants have a way to appeal? Should the company stop using the model for the time being? If accuracy improvement is the only goal, the most important problem is missed: a structure remains in which no one can explain why anyone was rejected.
Counterargument and Tension: We Cannot Stop Everything
Of course, we cannot halt technology whenever there is a risk. Automated systems can genuinely protect people’s lives in areas such as medical image analysis, disaster forecasting, and assistive technology for people with disabilities. If we wait for perfect certainty, we may leave suffering that could be reduced today untouched. Doing nothing is also a choice, and that choice also passes a cost on to someone.
What we need, then, is not a rejection of possibility but the addition of both brakes and direction. Before launch, we should write down plausible harm scenarios, test the system within a limited scope, and leave users a way to refuse or reverse its actions. Even making automated decisions available for human review is not always enough. We must also design for whether that person has genuine authority, sufficient time, and the responsibility to explain the decision.
An engineer is not someone who predicts the future. An engineer is someone who pushes one of several possible futures into reality. That is why code review should consider not only security vulnerabilities but also shifts in power. Who becomes more powerful because of this feature? Who will be forced into silence when something goes wrong? Questions like these belong in the space between being able to press the deployment button and deciding that it should be pressed.
A Question to End With
What does the system you are building make possible? And, more importantly, because of that possibility, what might someone no longer be able to refuse? In this deployment, should we be checking not only whether the feature works, but also which choices people can still claim as their own after it does?