06. Does data tell the truth?

The question: How does a number testify?

The interviewer turns the laptop screen around. "This model was trained on historical hiring data, and its prediction accuracy is 92 percent." Heads nod around the conference table. There is a great deal of data, the calculations are automated, and the result is carried out to two decimal places. The explanation follows that the system was built to reduce human bias. But what if the historical data is a record of having hired more people from certain schools for many years? Instead of eliminating bias, the model will reproduce the traces bias has left behind—faster and more consistently.

Data is often used as if it were another name for fact. The phrase "the data tells us" carries an image of data opening its own mouth while humans merely write down what it says. But data is not a voice that rises spontaneously from nature. It is the result of someone's decisions about what to record and what not to record, which categories and labels to apply, and which questions to ask. Data can convey truth, but it does not guarantee truth.

The philosophical point: Recorded facts and facts about the world

Data has at least two layers. One is the fact that an event occurred; the other is the fact that the event was recorded in a particular system. If a hospital record contains a pain score of 3, that does not mean the patient's pain was objectively 3. It also reflects whether the patient understood the question, whether the nurse was busy, whether there was a cultural habit of describing pain as less severe, and what kinds of answers the form allowed.

Data's gaps are data too. To determine whether an area with few reports is safe or whether its residents do not trust the institution to which they would report, we have to examine the conditions behind what was not recorded. If the behavior of visitors who were not logged in is excluded from the analysis, the system can speak only about logged-in users. The moment we forget that limitation, a partial sample begins to stand in for humanity as a whole.

This is an epistemological problem. What are we entitled to say we know? A model's output is not a verdict on the future, but an estimate calculated under particular data and assumptions. It matters that the estimate achieved high accuracy, but we also need to know whom it gets wrong and who bears the cost when it does. Average accuracy can conceal how responsibility and harm are distributed.

An engineering case: Automated risk scores

Suppose a financial service automates loan screening. The model predicts the likelihood of delinquency and requests additional documents when the risk score is high. A developer may believe the system is fair because gender and race were removed from its direct inputs. Yet variables such as ZIP code, transaction times, the type of mobile phone, and the branch a person used in the past may already be speaking on behalf of social conditions. Where someone lives and which financial institution they have dealt with can become channels of disadvantage unrelated to their willingness to repay.

The more complicated problem is that past outcomes are used as if they were causes. If a particular group received fewer loans in the past, that may be because the group was more risky—but it may also be because banks offered them fewer loans in the first place. When the model learns historical approval records, a lack of access to opportunity is transformed into a lack of creditworthiness. The system appears to say, "This person is risky," when it may really be saying, "People like this person had fewer opportunities in the past."

In this situation, the engineering response is not to make the model more complicated. We need to trace how the data was produced, document the institutional judgments embedded in the labels, and examine error and rejection rates by group. We should design a channel for appeals and a human-review process instead of treating the prediction as a final decision. We should also observe how real people adapt after the model is deployed. Someone denied a loan will change their behavior in their next application, and that change will become new data. Data does not merely reflect the world; it also helps create the next scene in that world.

Counterpoint and tension: Isn't data better than people?

The objection that "human reviewers are biased too, but at least data and models are consistent" cannot simply be dismissed. Automation can reduce the fluctuations caused by fatigue and mood, and it can apply the same rules to many people. A well-designed system may indeed be fairer than human judgment.

But consistency is not the same as justice. Applying a wrong standard to everyone equally is not fair. And when human bias does not disappear but moves into code and data pipelines, those harmed by it may find it even harder to identify who is responsible. The sentence "The model made that decision" sounds like an explanation, but it can easily become a way of erasing accountability.

Engineers should therefore disclose the conditions of data rather than claim that data is neutral. They should state which population it was collected from, what omissions and measurement errors it contains, and in which situations it must not be used. Building a model is not the same as building a machine that produces the right answer. It is the work of managing a particular uncertainty in a particular way.

A question to leave with

Whose experiences does this data record, and whose does it silence? When the model is wrong, who can discover the error and challenge it, and can the system carry that voice into its next version? When we say "the data speaks," what human choices and institutions are we actually making it speak for?