A DOOR FROM OTHERWISE · VOLUME II
Does prediction change the people it predicts?
Accuracy can rise for two different reasons: a model can learn the world, or the world can learn how to become legible to the model.
Recommendations had changed what they encountered; rankings changed what they attempted; schools trained toward measurable outcomes; applicants learned the categories; creators learned the recommender. The population had not merely been understood. It had been trained into greater legibility. 'Did the model learn the world,' Noor asked, 'or did the world learn the model?' No one could answer from accuracy alone.
The prediction had improved partly because the predicted population had been trained to resemble the model.
“First we predict the animal,” she said. “Then we breed for the prediction.” Nobody liked the sentence. That was why it stayed.
The prediction had improved partly because the predicted population had been trained to resemble the model.
“First we predict the animal,” she said. “Then we breed for the prediction.” Nobody liked the sentence. That was why it stayed.
A useful test is to hide the score from part of the population. If the visible-score group becomes easier to predict, some of the model's apparent intelligence may be coming from the people learning the measurement.
ENTER OTHERWISEExcerpt: OTHERWISE, Volume II. The current ARC invitation begins with Volume I.