Benchmarks · updated 9 October 2026
Nirnoy comes in two sizes: Nirnoy-flash, small and fast, and Nirnoy 12B, the most accurate on harder tasks. We asked both and five other models the same questions, with the same answer options, mostly on human-labelled Bangla and Banglish data. Scores are macro-F1 (%), higher is better; ★ marks the best model in each category, and any within 0.1 points of it, which count as tied for first: the same hosted model moved that much between two runs (Jev scored 37.4 on emotion on 6 October 2026 and 37.5 on emotion on 9 October 2026).
| Category | Nirnoy-flash | Nirnoy 12B | Jev | Clef | Clef-flash | Gemma 12B | Gemma E4B |
|---|---|---|---|---|---|---|---|
| Everyday tasks | |||||||
| Assistant commands: intent | 85.6 ★ | 82.8 | 76.6 | 79.8 | 79.5 | – | – |
| Assistant commands: scenario | 92.3 ★ | 90.8 | 84.3 | 86.9 | 88.5 | – | – |
| Hate speech in Bangla and Banglish | 84.1 ★ | 82.5 | 70.2 | 61.6 | 58.1 | – | – |
| Code-mixed sentiment | 76.2 | 77.1 ★ | 63.8 | 54.7 | 51.5 | – | – |
| News topic | 88.1 | 88.3 ★ | 88.1 | 84.3 | 79.7 | – | – |
| Emotion | 36.1 | 37.4 Tied for first: 37.4 is within 0.1 points of the best score, 37.5. The same hosted model, run again on another day, moves this much: Jev scored 37.4 on emotion on 6 October 2026 and 37.5 on emotion on 9 October 2026. So gaps this small count as a tie. | 37.5 ★ | 34.1 | 31.6 | – | – |
| Scam detection | |||||||
| SMS scam detection | 82.8 | 90.1 ★ | 88.4 | 85.3 | 83.8 | 88.6 | 83.1 |
| Scam messages, private set | 86.8 | 87.3 ★ | – | 67.3 | 59.3 | 73.0 | 82.6 |
| Decision scenarios, synthetic (accuracy) | |||||||
| Decision scenarios | 77.2 | 80.1 ★ | 75.7 | 74.6 | 67.7 | 70.8 | 67.8 |
A human-labelled test of 11,063 questions, every option shown. Nirnoy trained on other examples from the first four datasets; topic and emotion are new to it. The base Gemma models were not run on this test with every option shown.
Pick the right one of 60 intents · 1,500 commands
Data: MASSIVE (Bengali) (CC BY 4.0)
কাল সকাল সাতটায় একটা অ্যালার্ম দিয়ে দাও
Which of 60 intents? → set an alarm
Pick the right one of 18 areas · 1,500 commands
Data: MASSIVE (Bengali) (CC BY 4.0)
আজ ঢাকায় বৃষ্টি হবে কি?
Which of 18 areas? → weather
Is a comment hateful? · 3,966 comments
Data: BanHate (MIT) and BanTH (MIT)
tor moto faltu lok ar dekhini, chup thak
Is this hateful? → yes
Banglish comments, including mixed feelings · 2,024 comments
Data: BnSentMix (MIT)
Movie ta overall bhalo, but ending ta ektu boring laglo.
Positive, negative, neutral or mixed? → mixed
Pick one of 7 topics (not in Nirnoy's training) · 204 headlines
Data: SIB-200 (Bengali) (CC BY-SA 4.0)
বিশ্বকাপ বাছাইপর্বে শেষ মিনিটের গোলে জিতল বাংলাদেশ
Which topic? → sports
Pick one of 6 emotions (not in Nirnoy's training) · 1,869 texts
Data: EmoNoBa (CC BY 4.0)
রেজাল্ট দেখে চোখে পানি চলে এলো, এত খুশি আগে কখনো লাগেনি!
Which emotion? → joy
Real messages Nirnoy never trained on: a public set of Bengali SMS (scored on a held-back half, once) and a private set of messages. Jev was not run on the private set, because it is never sent to outside services; with 110 messages, gaps under about 8 points are within noise.
Public set of real Bengali SMS (MIT licence): scam, spam or legitimate · 701 messages
Data: Bengali SMS smishing dataset (MIT)
অভিনন্দন! আপনার নম্বর ৫০,০০০ টাকা জিতেছে। টাকা পেতে এখনই এই লিংকে ঢুকে আপনার মোবাইল ব্যাংকিং পিন দিন।
Scam, spam or legitimate? → scam
Real messages, with the question in Bangla and in English · 110 messages
Data: A private set of real messages; not published.
আপনার মোবাইল ব্যাংকিং একাউন্ট আজ বন্ধ হয়ে যাবে। চালু রাখতে OTP কোডটি এই নম্বরে পাঠান।
এই মেসেজটি কি প্রতারণা, স্প্যাম নাকি বৈধ? / Scam, spam or legitimate? → scam
10,000 everyday decisions of ten kinds, written in Bangla by a language model, which also set the answers. Useful for breadth, but the answers are not human-checked.
Routing, safety checks, answer checks, record matching and six more kinds of decision; accuracy (%) · 10,000 rows
Data: A private synthetic set written with a language model; not published.
আমার অর্ডার এখনো আসেনি, টাকা ফেরত চাই।
Route to: sales, refunds, technical support or delivery? → refunds
Same questions for every model. Each model saw the same text, question and full list of answer options. Jev, Clef and Clef-flash answered through their public APIs on OpenRouter (October 2026). The private scam messages never leave our machines, so there Clef and Clef-flash ran from their released weights on our GPUs and Jev was not run. The Gemma 4 base models are untrained; Nirnoy-flash is built on Gemma 4 E4B and Nirnoy 12B on Gemma 4 12B.
Macro-F1 averages the F1 score over the answer classes, so a model cannot score well by always giving the most common answer.
Held-back half. The public SMS set is split in two by record. We tuned only on one half; the score here is from the other half, scored once.
What Nirnoy saw in training. Neither model was trained on the scam sets or the decision scenarios. For the everyday tasks they were trained on other examples from the intent, scenario, hate and sentiment datasets; topic and emotion are new to them.
Examples on this page are written for illustration and are not taken from the test sets.