What happened
Researchers Addison J. Wu, Ryan Liu, Xuechunzi Bai, and Thomas L. Griffiths, from Princeton University and the University of Chicago, published a paper titled "Large Language Models Develop Novel Social Biases Through Adaptive Exploration," presented as an oral session at ICML (the International Conference on Machine Learning) in Seoul in July 2026. MIT Technology Review reported on the findings on July 20, 2026. The design, adapted from an existing psychology study on how humans form stereotypes, put each model in the role of a hiring consultant for a fictional city. Over 40 rounds, the model filled 20 different jobs, including doctors, lawyers, child-care aides, and janitors, from four fictional demographic groups given made-up names (Tufa, Aima, Reku, and Weki). After every hire, the model learned whether that hire succeeded and moved to the next round. Unknown to the model, every group had an identical, equal chance of succeeding at every job. There was no real demographic data anywhere in the setup, no names tied to any real ethnicity, and no bias built into the training scenario. The models still developed stereotypes. MIT Technology Review reported that when a model observed one fictional group failing at a high-status role like doctor, it began routing that entire group toward lower-status roles like janitor, a pattern the paper's authors describe as jobs split by perceived warmth and competence. On the study's 0 to 2 segregation scale, where 2 means every group has been sorted into its own separate job category, the human participants in the original psychology experiment scored 0.84. The AI models scored roughly 65% higher on average, and OpenAI's reasoning model o3 scored 1.83, close to the maximum possible score. DeepSeek's R1 also showed strong stratification. Notably, the paper found the effect got worse, not better, in newer models with stronger reasoning capability.
Why it matters for business owners
Most small and medium businesses are not running a custom AI hiring simulation, but a growing number use AI tools that do something structurally similar: screen resumes, rank leads, score vendor bids, or triage support tickets, and improve or adjust based on the outcomes they observe over time. Any tool built to "learn" from what happened last time, whether that is explicit fine-tuning, a memory feature, or just an operator repeatedly telling the tool what worked and what did not, is running the same basic loop the researchers tested: make a decision, observe an outcome, adjust the next decision. The useful, verified finding here is not that AI can be biased. Every business owner evaluating AI tools already assumes that is possible. The finding is where the bias can come from: not only from a skewed training dataset the vendor controls, but from ordinary use inside your own business, generated fresh from your own outcomes, with no biased data anywhere in the system to begin with. A vendor's fairness testing at launch cannot catch a pattern that has not been created yet.
What owners should not misunderstand
This is not proof that any specific commercial hiring tool, applicant tracking system, or CRM scoring feature is doing this today. The study is a controlled lab simulation with fictional groups and a simplified 40-round game, not an audit of a real product used by real applicants. It demonstrates a mechanism the researchers could reproduce reliably in a lab, not a documented incident inside any named company's software. It is also not a reason to assume newer or more capable AI models are safer for this kind of task by default. The paper found the opposite: models with stronger reasoning, like o3 and R1, stratified candidates more than earlier, less capable models, not less. Do not treat a model's benchmark scores on coding or reasoning tasks as evidence about its fairness in a repeated decision-making role. And this is not only a hiring story. Griffiths and coauthors' broader argument is about any AI system making repeated decisions with real-world feedback, loan approvals, insurance pricing, lead scoring, vendor selection, wherever an AI tool observes outcomes and adjusts. Hiring is simply the setting the researchers tested directly.
The operational lesson
Telling an AI system to "be fair" is not a reliable fix. MIT Technology Review reported that a direct fairness instruction had little effect on the models' behavior in this study, one model (per the paper) even refused to respond to a fairness-focused system prompt rather than change its pattern. What did work, according to the paper, was changing what the model was actually optimizing for: adding an explicit diversity objective to the reward changed behavior far more reliably than asking nicely. Coauthor Ryan Liu told MIT Technology Review that promising models a bonus for diverse hiring outcomes made them far less biased, while a plain fairness instruction largely did not change the underlying tendency. The researchers also found that giving the model more individuating information about each candidate, real, relevant detail rather than group membership, reduced stereotyping, while irrelevant detail did not help and could even reinforce it. That is a concrete, testable design principle: an AI decision tool that only sees a thin profile is more likely to fall back on pattern-matching by group than one that sees the specific, job-relevant facts about a specific person.
What a serious business should do next
List every AI tool your business uses for a repeated selection or ranking decision, not just hiring: resume screening, lead scoring, applicant ranking, vendor shortlisting, credit or payment risk flags, customer prioritization. For each one, find out from the vendor, in writing, whether the tool's scoring changes over time based on outcomes it observes inside your account, or whether it is a fixed model that does not update from your usage. A static, well-tested model and a continuously adapting one carry different risk profiles, and most owners cannot currently answer which one they have. For any tool that does adapt or has memory of past outcomes, ask what objective it is actually optimizing for, and whether fairness or diversity considerations are part of that objective or bolted on afterward as an instruction. Per this research, the difference matters more than most vendors' marketing suggests. Build in periodic human review of AI-assisted hiring or screening decisions, not as a one-time compliance check at rollout, but as a recurring spot check, since a pattern like this can emerge gradually over weeks or months of normal use, not all at once. Do not wait for a discrimination complaint to be the first time anyone looks at the pattern of who the tool has been recommending.
The Atlacis view
Atlacis helps owners look past a vendor's fairness claims at launch and ask the harder question: what does this tool actually optimize for once it is running inside your business, and who is checking its pattern of decisions after the first ninety days. A tool can pass every fairness benchmark a vendor publishes and still drift once it starts learning from your specific outcomes, which is a workflow design question, not a one-time procurement question. Before recommending any AI tool that screens, ranks, or scores people, Atlacis maps whether it adapts from experience, what it is actually rewarded for, and what a reasonable human review cadence looks like for that specific use, rather than treating a vendor's launch-day audit as the last word.
The short version
- A Princeton and University of Chicago study, presented at ICML in Seoul in July 2026 and reported by MIT Technology Review on July 20, found that AI models invent new demographic stereotypes purely from their own sequential hiring decisions, with no biased training data and no real demographic information in the setup.
- The models stereotyped more than human participants in the same test. On a 0 to 2 segregation scale, humans scored 0.84; the AI models scored roughly 65% higher on average, and OpenAI's o3 scored 1.83, near the maximum.
- Newer, higher-reasoning models (o3, DeepSeek's R1) showed stronger bias, not weaker, which means model capability is not a proxy for fairness in repeated decision-making roles.
- Telling a model to "be fair" had little effect in the study. Adding an explicit diversity objective to what the model was optimizing for, and giving it more individuating detail about each candidate, worked better, per the paper and coauthor Ryan Liu.
- This is a lab finding about a mechanism, not proof any specific commercial hiring tool is doing this today. The operational lesson: inventory any AI tool that adapts from your business's own outcomes, ask what it optimizes for, and build in recurring human review rather than a one-time fairness check at purchase.
Where ATLACIS can help
Sources
- MIT Technology Review: AI is more likely than humans to form biases when hiring (July 20, 2026)
- ICML 2026: Large Language Models Develop Novel Social Biases Through Adaptive Exploration (Oral session, Seoul, July 2026)
- Wu, Liu, Bai, and Griffiths: Large Language Models Develop Novel Social Biases Through Adaptive Exploration (full paper)