OnFinance AIneogpt-asr

Benchmarks

neogpt-asr against Sarvam, AWS Transcribe, Speechmatics, AssemblyAI and OpenAI's Whisper. Same phone-quality recordings, same scoring for every system.

83%
word accuracy on real Hindi phone calls, 1st of 8
6 of 6
Indian languages where it ranks first
22
Indian languages measured, 8 above 80%
8
systems on the same recordings

Best on real Hindi phone calls

Real customer-style calls recorded over Indian phone lines, with background noise and rural accents. neogpt-asr gets 83.4% of words right, level with Sarvam's newest model and ahead of every global provider.

Hover or tap a bar for error rates and rank.

First in Bengali, Gujarati, Kannada, Marathi, Tamil and Telugu

Global providers fall apart outside Hindi: some write Bengali in the wrong script or translate Telugu into English. neogpt-asr leads in all six (tied in Kannada), with its widest lead in Tamil: 6.2 points ahead of AWS Transcribe.

Switch languages above the chart.

Account numbers and IDs

Hindi callers reading out phone, account and ID numbers. neogpt-asr gets 90.4% of words right, the most of any system.

English with an Indian accent

English is where we have the most room to grow. neogpt-asr ranks 5th of 7 here, 5.7 points behind AssemblyAI.

Every scheduled Indian language

neogpt-asr on phone-quality recordings in all 22 languages, best first.

Every result

Word accuracy for each system and test set. Green is the best in each column.

SystemHindi, phone callsHindi, numbers and IDsBengaliGujaratiKannadaMarathiTamilTeluguEnglish, Indian accent
neogpt-asr83.4%90.4%90.0%79.8%63.7%86.6%76.5%70.7%89.5%
Sarvam saaras:v483.3%88.3%88.8%79.5%63.7%81.5%68.0%67.3%91.8%
Sarvam saaras:v381.5%84.6%85.3%75.3%55.3%80.7%67.0%62.0%Not run
Speechmatics80.7%79.4%74.8%Not supportedNot supported58.5%47.7%Not supported91.4%
AWS Transcribe80.1%82.9%86.5%79.2%51.5%84.4%70.3%68.1%84.4%
AssemblyAI73.4%71.8%0.0%0.0%8.4%4.0%31.0%0.0%95.2%
Whisper large-v344.5%62.9%26.1%33.6%0.0%0.0%5.2%0.0%90.6%
Whisper large-v3-turbo37.2%50.1%0.0%0.0%0.0%0.0%0.0%0.0%89.5%

How we tested

Every system heard the same recordings and was scored by the same code.

Phone-quality audio

Real 8 kHz phone recordings for Hindi, and public test recordings passed through a phone codec for the other languages.

Same scoring for everyone

Word accuracy against careful human transcripts, with punctuation and spelling variants evened out. "9840" and the spoken digits count the same.

Out-of-the-box settings

Each system was told the call's language and run with its standard settings, through its public API or unmodified model.

Systems
  • neogpt-asrOnFinance
  • Sarvam saaras:v4 and v3API
  • AWS TranscribeAPI
  • SpeechmaticsAPI
  • AssemblyAIAPI
  • Whisper large-v3Open weights
  • Whisper large-v3-turboOpen weights
Data
  • GramVaaniOpenSLR 118
  • IndicVoicesAI4Bharat
  • IndicVoices number tasksAI4Bharat
  • FLEURSGoogle

Try neogpt-asr on your own calls

Tell us which languages your customers speak and what you use today. We will run it on audio like yours and walk you through the results.