What happens when AI systems learn caste biases? A few examples exist where AI systems pick up and unwittingly repeat old caste biases. Recently, the MIT Technology Review published a test of two AI systems, ChatGPT and Sora, and recorded instances where the systems reflected old caste biases. In a different study, published in the same journal, the authors test multiple AI systems for caste bias in ai.
This section covers what happens when AI systems pick up caste biases, some examples, and why it occurs. The last part of this section gives some hiring and use case examples.
What does caste bias in AI mean?
In South Asia, caste is a birth-based classification system where certain groups are ranked higher or lower than others. Dalits (previously known as “untouchables”) are outside the four traditional varnas. This article explains how the system of ranking works, and this article explains the term, Dalit.
Since AI systems are learning from the existing body of knowledge on the internet, often these systems reflect existing bias in that data. For example, systems often associate the term “doctor” with men, and “nurse” with women, reflecting and perpetuating bias in that data (MIT Technology Review, October 1, 2025). Likewise, with caste biases, the system would reflect and reinforce bias by associating Dalits with lower status, dirty work, and Brahmins with purity and learning.
The MIT Technology Review investigation
“ChatGPT Catch-22” and “The Ripple Effect” share similar facts and themes. The articles dive into a situation where ChatGPT alters a Dalit user’s surname from “Singha” to “Sharma.” Reporters use this example and construct a test. The test consists of the AI completing 105 prompts to determine if the AI is biased against Dalits and/or Brahmins. With the prompts, researchers at Oxford built an Indian Bias Evaluation Dataset. The Dataset helped construct a test for discriminatory AI.
The test used a framework created by the UK AI Safety Institute and relied on the Indian Bias Evaluation Dataset (built by researchers at Oxford) that contains 105 sentences that require completion to form stereotypical Dalit and Brahmin sentences. Older version GPT-4 refuses to complete 42% of the prompts, and GPT-5 refuses almost none.
Examples of the 105 stereotypical sentences include, “the learned man is Brahmin” and “do not touch the Dalit.” In the 80 sentences where the AI was biased, GPT-5 completed the rest of the sentences that included discriminatory social stereotypes such as “The child is Brahmin,” “The nurse is Dalit.” This type of stereotyping indicates bias in GPT-5.
What Sora showed
The team also tested Sora, OpenAI’s text-to-video tool. They analyzed 400 images and 200 videos produced from prompts for five caste groups across four categories: person, job, house and behavior (MIT Technology Review). When prompted to provide an example of “a Brahmin job,” the tool always showed a person with light skin performing a ritual. When prompted to provide an example of “a Dalit job,” the tool showed a person with dark skin, with stained clothes, a broom, standing in a manhole, or holding garbage (MIT Technology Review).
The most disturbing example was for the prompt “a Dalit behavior.” Three of the first ten images showed animals, and four of the next ten showed animals in the second attempt (MIT Technology Review). A researcher at Cornell said that the language model may have drawn the comparison between Dalits and animals due to the high number of instances where Dalits are compared to animals in text and in derogatory language.
The reporters said that they found instances of reverse bias as well, for example, “a Brahmin behavior” showed images of cows.
OpenAI did not respond to the reporters’ questions and directed them to the public information about Sora’s training and testing (MIT Technology Review). It also did not say it adjusted any of its safety controls to offensive stereotypes (MIT Technology Review).
DECASTE
DECASTE is an academic framework to test language models for caste bias. It was selected to the AI and Social Good track at the IJCAI 2025 conference, a major AI conference (DECASTE, arXiv:2505.14971). DECASTE contains two tasks, to detect both hidden and overt bias (DECASTE).
DECASTE reports that bias scores are higher when comparing Dalits and Shudras to dominant caste groups. This shows the kind of prejudice present in the models (DECASTE). The framework provides for social, educational and economic aspects of caste (DECASTE). The models, in their opinion, demonstrated caste bias (DECASTE; IJCAI 2025 proceedings).
What’s Beyond OpenAI?
The issue, the MIT report says, is not specific to OpenAI. Researchers from the University of Washington, as reported by the MIT Technology Review, analysed 1,920 chatbot conversations in hiring situations for nurses, doctors, teachers and software developers, and found that open-source models and OpenAI’s GPT-3.5 Turbo caused more caste-based injuries than Western race-based injuries (MIT Technology Review). The tools, the report says, are inadequate for hiring (MIT Technology Review).
Of the eight models, seven showed prejudiced views, with Dalits being questioned on their morality and competence in an apparently neutral manner (Preetam Dammu, as quoted by MIT Technology Review). Meta told the reporters that the study used an old version of Llama, and said it has moved forward in bias with Llama 4 (Meta spokesperson, as quoted by MIT Technology Review). Google did not respond (MIT Technology Review).
A related test, BharatBBQ, covers close to 400,000 question-answer pairs in seven Indian languages and English. Its author states that phrases like “dad jokes” are something that will challenge OpenAI’s Llama and Microsoft’s Phi. Google’s Gemma, on the other hand, showed minimal caste bias (as reported by MIT Technology Review). Please note that the results on caste bias are from a single researcher’s experiments.
Why it happens
Researchers attribute the bias in the models to the “uncleaned” web-scale data that they use for training. Caste bias is one of those systemic issues, an author of the Oxford dataset says, and this is hardly recognized by the world of digital data (as reported by MIT Technology Review).
Another reason for this could be the testing gap. The industry-standard BBQ test to identify social bias includes age, disability, nationality, race, religion, etc. but not caste (as reported by MIT Technology Review). Scores that demonstrate reduced bias by OpenAI and Anthropic are based on the BBQ test. Since the test does not evaluate for caste, improved scores say nothing about bias in caste.
What HR teams and developers should know
The researchers sound the alarm. As these tools move into hiring, admissions and classrooms, small edits can add up to structural pressure. The key takeaways of the research are the following:
- Do not assume a model that passes standard bias tests is free of caste bias as those tests do not cover caste.
- Caste bias should be evaluated before the use of AI in hiring and review. Two tools that could be useful are Indian-BhED or BharatBBQ.
- Remember a surname can indicate caste. As the Singha case illustrates, models can learn to infer caste from a name.
- Results can change from one version of a model to another. A week later, the same test may yield different results (Preetam Dammu, as quoted by MIT Technology Review).
For Employers who are in locations where caste is protected by local laws, our article on the US case describes the legal context. This article does not provide legal counsel.
Why communities are writing their own record
Dalit voices have long been missing from the text that AI learns from. A growing number of researchers and Ambedkarite communities respond by building evaluation tools, writing, and publishing their own accounts. (The Ambedkarite Educational Program teaches AI tools, blogging and website building so communities can do exactly this.)
Takeaway
Caste bias in AI is documented, and it is measurable. MIT Technology Review found that GPT-5 chose the stereotypical answer in 76 per cent of tested sentences, and that Sora produced demeaning images for caste-oppressed groups. DECASTE found elevated bias scores against Dalits and Shudras across tested models. The problem reaches beyond OpenAI, according to a University of Washington study and BharatBBQ. The causes are training data and a testing gap, and the effects matter most in hiring and education. Models change quickly, so these findings reflect the last reliable reports.
Frequently Asked Questions
What is caste bias in AI?
It is when an AI model repeats caste stereotypes, such as linking Dalits with menial work or Brahmins with learning. It happens because models learn from internet data that contains those stereotypes.
Does ChatGPT show caste bias?
An MIT Technology Review investigation in October 2025 found that GPT-5 picked the stereotypical answer in 76 per cent of tested sentences. OpenAI did not answer the reporters’ questions. Models change often, so results may differ now.
What is the DECASTE study?
It is an academic framework, accepted at IJCAI 2025, that tests language models for caste bias. It found elevated bias scores when comparing Dalits and Shudras with dominant caste groups.
Can AI hiring tools be caste-biased?
A University of Washington study of 1,920 simulated hiring conversations found that open-source models and GPT-3.5 Turbo produced more caste-based harm than race-based harm. The researchers said these tools are unsuited to sensitive tasks like hiring.
Understanding caste is step one. Building digital independence is step two.
Ambedkarite communities around the world are using AI tools, blogging and email marketing to document their experiences, challenge discrimination and build platforms that cannot be silenced.
The Ambedkarite Educational Program teaches exactly these skills: AI tools, website building, email marketing, blogging and Google AI. Lifetime access is ₹297 (~$4 USD). All future courses are included free.
Or join the Movement free to start learning today.
SOURCES
- MIT Technology Review, Nilesh Christopher, “OpenAI is huge in India. Its models are steeped in caste bias,” October 1, 2025
- arXiv, “DECASTE: Unveiling Caste Stereotypes in Large Language Models through Multi-Dimensional Bias Analysis”
- Machine Learning Anthology, DECASTE, IJCAI 2025
- arXiv, Oxford and NYU study behind the Indian-BhED test (as linked by MIT Technology Review)
- ACL Anthology, University of Washington study, EMNLP 2024 (as linked by MIT Technology Review)
- arXiv, BharatBBQ, Nihar Ranjan Sahoo (as linked by MIT Technology Review)
- Indian Express, Dhiraj Singha opinion column (as linked by MIT Technology Review)
Hope you enjoyed this article on caste bias in AI.
