Opportunities › Aaj ki opportunities › AI & Software

Trustworthy AI in Indian languages: testing and grading Indic AI models

AI & Software14 Nov 2026Score 76/100Confidence MediumPublished5 sources · 10 sourced facts

India now has many home-grown AI models, but few fair tests that check them in Indian languages and reward an honest 'I don't know'.

Seedhi baat: Model bahut ban rahe hain, par Indian bhashaon me unhe sahi tarah jaanchne wale test kam hain. Yahi kaam ek mauka hai.

Fact sourced, with link and check date (click [id]). Analysis NitiMarg's reading, not a fact.

1. The world's development challenge

  • FactMILU, a benchmark from AI4Bharat (IIT Madras) and IBM Research India, tests AI models in 11 Indian languages across 41 subjects. The best model (GPT-4o) averaged about 74%, and language-specific Indic models did only slightly better than random guessing.[S1]
  • FactThe same study found models do worse in low-resource languages, and in culture-specific areas such as Arts & Humanities and Law & Governance than in STEM.[S1]
  • FactStanford's 2026 AI Index reports that documented AI incidents rose to 362, up from 233 in 2024, and that reporting on responsible-AI benchmarks remains spotty.[S2]

2. The real technology challenge

  • FactA 2025 paper from OpenAI and Georgia Tech argues that language models 'hallucinate' because training and evaluation reward guessing over admitting uncertainty. Most popular benchmarks give no credit for 'I don't know'.[S3]
  • FactThe same paper points to exams with negative marking, including India's JEE, NEET and GATE, as the model for fixing this: tell the model the penalty for a wrong answer, so guessing stops paying.[S3]
  • FactThe 2026 AI Index calls today's AI a 'jagged frontier': a model won an International Mathematical Olympiad gold medal, yet the top model reads analog clocks correctly only 50.1% of the time.[S2]

3. Where scientists are stuck stage today

Research: the problem is understood, the fix is not standard yet

  • FactThe MILU authors found that adapting English models with small Indian-language datasets gave minimal gains, and that training data lacks culture-specific Indian knowledge.[S1]
  • AnalysisSo the stuck point is not 'more models'. It is reliable, India-specific test sets and grading rules that buyers trust.

4. What researchers are trying now

  • FactResearchers propose 'confidence targets': each test question states how much a wrong answer costs, so a model is scored for knowing when to abstain.[S3]
  • FactThe IndiaAI Mission (outlay Rs 10,371.92 crore, approved 7 March 2024) has picked 20 indigenous foundation-model proposals from 506 applications, sanctioned 93.18 lakh GPU hours to 237 projects, and selected 13 'Safe & Trusted AI' projects on bias, machine unlearning, privacy, algorithm auditing and explainability.[S4]
  • FactOn 3 November 2026, IndiaAI released Soket's EKA-1 (26.3 billion parameters, about 7.5 trillion training tokens, all 22 scheduled languages plus English) and ZenteiQ's open-source Dhir 2B and 5B models (English plus 10 Indian languages, can run offline).[S5]

5. India's angle

  • AnalysisIndia is building many models quickly (see facts above). Banks, state governments and startups that deploy them will need proof that a model answers correctly in Marathi, Tamil or Odia, about Indian schemes and laws, and says 'I don't know' when unsure.
  • AnalysisIndia already understands negative-marking exams at mass scale. That habit, plus a large pool of language graduates, is a natural base for building and grading such tests.

6. The opportunity for India

Indic AI testing lab: India-specific test sets and 'honest grading' for AI models

Who can do it: Small teams (2-6 people) with language experts plus one ML engineer; also existing data-annotation firms.

  • AnalysisBuild domain test sets in Indian languages (for example government schemes, crop advice, consumer law), grade models with negative marking and credit for 'I don't know', and sell audits and dashboards to model builders and organisations that deploy chatbots.

First steps

  • AnalysisPick one domain and two languages; write 500 checked questions from public, licence-clear material.
  • AnalysisScore 5 open models (including IndiaAI-supported ones) and publish a free summary to earn trust.
  • AnalysisOffer paid private audits to one bank or state department pilot.

7. Risks and what could go wrong

  • AnalysisBig AI labs or IndiaAI may release free benchmarks that reduce demand.
  • AnalysisCopyright: question material must be licence-clear; MILU itself was collected from public exam portals, which a commercial product cannot copy freely.
  • AnalysisGrading open answers needs paid human experts; quality slips if this is cut.

Score 76 of 100 NitiMarg analysis (andaaza), same 5-rating formula as the opportunity list.

Problem proof4/5Demand lasts4/5Low competition3/5Easy entry3/5India fit5/5

Formula: round((0.25*P + 0.25*D + 0.25*C + 0.10*E + 0.15*I) / 5 * 100). Confidence: Medium.

Sources all opened on 14 Nov 2026

  1. [S1] MILU: A Multi-task Indic Language Understanding Benchmark
    AI4Bharat / IIT Madras / IBM Research (arXiv; NAACL 2025) · published 2024-11 (NAACL 2025) · opened and checked 14 Nov 2026
  2. [S2] The 2026 AI Index Report: Top takeaways
    Stanford HAI · published 2026 · opened and checked 14 Nov 2026
  3. [S3] Why Language Models Hallucinate
    Kalai, Nachum, Vempala, Zhang (OpenAI, Georgia Tech) - arXiv · published 2025-09-04 · opened and checked 14 Nov 2026
  4. [S4] Government Expands Sovereign AI Infrastructure Through IndiaAI Mission and Semiconductor Initiatives
    PIB (MeitY, Lok Sabha reply) · published 2026-08-06 · opened and checked 14 Nov 2026
  5. [S5] IndiaAI Mission released indigenous foundation models EKA-1 and Dhir 2B & 5B
    PIB (MeitY) · published 2026-11-03 · opened and checked 14 Nov 2026
Educational, not investment, medical or business advice. Numbers are as published by each source on the date shown; NitiMarg did not re-compute them. Back to Aaj ki opportunities · Research frontier