Nirnoy নির্ণয়

Showcase · real test questions

Real questions Nirnoy gets right and the others miss

35 questions from public, human-labelled Bangla and Banglish test sets, shown word for word. On each one a Nirnoy model (Nirnoy-flash, Nirnoy 12B or both) gives the dataset's answer and Jev, Clef and Clef-flash all give another. We picked them by hand, choosing questions where the dataset's answer is clear.

How often this happens

Hand-picked examples are not a score, so here is the whole test. Of 11,063 everyday-task questions, a Nirnoy model was right while Jev, Clef and Clef-flash were all wrong on 1017. The reverse (all three right, both Nirnoy models wrong) happened on 239.

TaskQuestionsOnly Nirnoy rightOnly the other three right
Voice-assistant intent1,500495
Voice-assistant area1,500301
Emotion1,8699319
Hate speech3,966588183
Sentiment2,02425028
Topic20473

The everyday-tasks test from the benchmarks, with every answer option shown. “Only Nirnoy right” counts a question when Nirnoy-flash or Nirnoy 12B is right and Jev, Clef and Clef-flash are all wrong. Their answers come from their public APIs (OpenRouter, 9 October 2026), the same requests for every question.

The questions

Last scene tah joss chilo

“The last scene was awesome.”

What is the overall sentiment of this comment? · 4 options
Dataset answer: positive

  • ✓Nirnoy-flashpositive96%
  • ✓Nirnoy 12Bpositive95%
  • ✗Jevnegative41%
  • ✗Clefneutral89%
  • ✗Clef-flashneutral83%
  • ✗Gemma 12Bneutral99%
  • ✗Gemma E4Bneutral99%

“Joss” is Bangladeshi slang for great.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "Last scene tah joss chilo",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "What is the overall sentiment of this comment?",
      "criteria": {
        "positive": "praise, thanks, satisfaction or happiness",
        "negative": "complaint, insult, disappointment or anger",
        "neutral": "a question, a request, a factual statement, or no clear opinion",
        "mixed": "clearly positive and clearly negative at the same time"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "negative",
      "probabilities": {
        "positive": 0.39,
        "negative": 0.41,
        "neutral": 0.19,
        "mixed": 0.01
      },
      "confidence": 0.21
    }
  },
  "usage": {
    "input_tokens": 376,
    "output_tokens": 46,
    "cost": 0.000015792
  },
  "id": "gen-dec-1791517790-0RXzyrRgK2cPVuIT1NLZ",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.0325,
        "negative": 0.0511,
        "neutral": 0.8849,
        "mixed": 0.0315
      },
      "confidence": 0.8465
    }
  },
  "usage": {
    "input_tokens": 203,
    "output_tokens": 0,
    "cost": 0.000008526
  },
  "id": "gen-dec-1791517795-MwtPUFy3bMuLRPVu0mlA",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.0428,
        "negative": 0.0664,
        "neutral": 0.834,
        "mixed": 0.0568
      },
      "confidence": 0.7787
    }
  },
  "usage": {
    "input_tokens": 203,
    "output_tokens": 0,
    "cost": 0.000004263
  },
  "id": "gen-dec-1791517800-CPrJtiIhiKKrWZd5MwXf",
  "provider": "PrimeIntellect"
}

offer valo dia na

“They don't give good offers.”

What is the overall sentiment of this comment? · 4 options
Dataset answer: negative

  • ✓Nirnoy-flashnegative91%
  • ✓Nirnoy 12Bnegative91%
  • ✗Jevneutral43%
  • ✗Clefneutral72%
  • ✗Clef-flashneutral82%
  • ✓Gemma 12Bnegative95%
  • ✗Gemma E4Bneutral99%

“Valo” (good) is cancelled by “na” at the end.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "offer valo dia na",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "What is the overall sentiment of this comment?",
      "criteria": {
        "positive": "praise, thanks, satisfaction or happiness",
        "negative": "complaint, insult, disappointment or anger",
        "neutral": "a question, a request, a factual statement, or no clear opinion",
        "mixed": "clearly positive and clearly negative at the same time"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.35,
        "negative": 0.21,
        "neutral": 0.43,
        "mixed": 0.01
      },
      "confidence": 0.24
    }
  },
  "usage": {
    "input_tokens": 373,
    "output_tokens": 46,
    "cost": 0.000015666
  },
  "id": "gen-dec-1791517791-n2Z4aTnPKXlpyHCsddPt",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.0171,
        "negative": 0.2272,
        "neutral": 0.7165,
        "mixed": 0.0392
      },
      "confidence": 0.622
    }
  },
  "usage": {
    "input_tokens": 200,
    "output_tokens": 0,
    "cost": 0.0000084
  },
  "id": "gen-dec-1791517797-bmgEvYAH55Gx734sfnJQ",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.0687,
        "negative": 0.0597,
        "neutral": 0.8173,
        "mixed": 0.0543
      },
      "confidence": 0.7564
    }
  },
  "usage": {
    "input_tokens": 200,
    "output_tokens": 0,
    "cost": 0.0000042
  },
  "id": "gen-dec-1791517802-h6u67Q7ObjRADTl8gSgL",
  "provider": "PrimeIntellect"
}

Biyer jhinnik batti bhalo lagenaaa

“I don't like wedding fairy lights.”

What is the overall sentiment of this comment? · 4 options
Dataset answer: negative

  • ✓Nirnoy-flashnegative93%
  • ✓Nirnoy 12Bnegative89%
  • ✗Jevpositive90%
  • ✗Clefpositive82%
  • ✗Clef-flashpositive50%
  • ✓Gemma 12Bnegative100%
  • ✓Gemma E4Bnegative100%

“bhalo lagenaaa” (don't like) begins with “bhalo” (good); the negation comes at the end.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "Biyer jhinnik batti bhalo lagenaaa",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "What is the overall sentiment of this comment?",
      "criteria": {
        "positive": "praise, thanks, satisfaction or happiness",
        "negative": "complaint, insult, disappointment or anger",
        "neutral": "a question, a request, a factual statement, or no clear opinion",
        "mixed": "clearly positive and clearly negative at the same time"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "positive",
      "probabilities": {
        "positive": 0.9,
        "negative": 0.08,
        "neutral": 0,
        "mixed": 0.02
      },
      "confidence": 0.87
    }
  },
  "usage": {
    "input_tokens": 383,
    "output_tokens": 46,
    "cost": 0.000016086
  },
  "id": "gen-dec-1791517792-U4Ka0dJzhd2CkwWC4yNl",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "positive",
      "probabilities": {
        "positive": 0.8189,
        "negative": 0.0276,
        "neutral": 0.1108,
        "mixed": 0.0427
      },
      "confidence": 0.7585
    }
  },
  "usage": {
    "input_tokens": 210,
    "output_tokens": 0,
    "cost": 0.00000882
  },
  "id": "gen-dec-1791517798-TpqBZKfpTZ3NEGHAyo2g",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "positive",
      "probabilities": {
        "positive": 0.5025,
        "negative": 0.0331,
        "neutral": 0.4166,
        "mixed": 0.0478
      },
      "confidence": 0.3367
    }
  },
  "usage": {
    "input_tokens": 210,
    "output_tokens": 0,
    "cost": 0.00000441
  },
  "id": "gen-dec-1791517803-gOrfxAOmZkWTO5sXJsfJ",
  "provider": "PrimeIntellect"
}

Onk Balo Apps Ata

“This is a very good app.”

What is the overall sentiment of this comment? · 4 options
Dataset answer: positive

  • ✓Nirnoy-flashpositive97%
  • ✓Nirnoy 12Bpositive95%
  • ✗Jevneutral35%
  • ✗Clefneutral75%
  • ✗Clef-flashneutral87%
  • ✗Gemma 12Bnegative95%
  • ✗Gemma E4Bneutral100%

“Balo” is a common spelling of “valo” (good).

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "Onk Balo Apps Ata",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "What is the overall sentiment of this comment?",
      "criteria": {
        "positive": "praise, thanks, satisfaction or happiness",
        "negative": "complaint, insult, disappointment or anger",
        "neutral": "a question, a request, a factual statement, or no clear opinion",
        "mixed": "clearly positive and clearly negative at the same time"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.29,
        "negative": 0.35,
        "neutral": 0.35,
        "mixed": 0.01
      },
      "confidence": 0.13
    }
  },
  "usage": {
    "input_tokens": 375,
    "output_tokens": 46,
    "cost": 0.00001575
  },
  "id": "gen-dec-1791517790-ER4CHEqcKKrnSrBM02cl",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by Cloudflare · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.0634,
        "negative": 0.1187,
        "neutral": 0.7461,
        "mixed": 0.0718
      },
      "confidence": 0.4399
    }
  },
  "usage": {
    "input_tokens": 202,
    "output_tokens": 0,
    "cost": 0.00004848
  },
  "id": "gen-dec-1791517795-oni9ai378mar4zVmUyAo",
  "provider": "Cloudflare"
}

Clef-flash's response, exactly as returned · served by Cloudflare · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.0333,
        "negative": 0.0565,
        "neutral": 0.8709,
        "mixed": 0.0393
      },
      "confidence": 0.6857
    }
  },
  "usage": {
    "input_tokens": 202,
    "output_tokens": 0,
    "cost": 0.00001818
  },
  "id": "gen-dec-1791517801-FVhGTR5vAjCvE1Cz4RCY",
  "provider": "Cloudflare"
}

Ai bajete sera phn

“The best phone in this budget.”

What is the overall sentiment of this comment? · 4 options
Dataset answer: positive

  • ✓Nirnoy-flashpositive86%
  • ✓Nirnoy 12Bpositive89%
  • ✗Jevneutral41%
  • ✗Clefneutral64%
  • ✗Clef-flashneutral84%
  • ✗Gemma 12Bnegative68%
  • ✗Gemma E4Bneutral100%

“Sera” means best.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "Ai bajete sera phn",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "What is the overall sentiment of this comment?",
      "criteria": {
        "positive": "praise, thanks, satisfaction or happiness",
        "negative": "complaint, insult, disappointment or anger",
        "neutral": "a question, a request, a factual statement, or no clear opinion",
        "mixed": "clearly positive and clearly negative at the same time"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.17,
        "negative": 0.41,
        "neutral": 0.41,
        "mixed": 0.01
      },
      "confidence": 0.21
    }
  },
  "usage": {
    "input_tokens": 375,
    "output_tokens": 46,
    "cost": 0.00001575
  },
  "id": "gen-dec-1791517792-6yCs8naKKpxxMztth4qm",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.0471,
        "negative": 0.2604,
        "neutral": 0.6371,
        "mixed": 0.0554
      },
      "confidence": 0.5161
    }
  },
  "usage": {
    "input_tokens": 202,
    "output_tokens": 0,
    "cost": 0.000008484
  },
  "id": "gen-dec-1791517797-UWczlxJPO3bVu0E4ya85",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.0269,
        "negative": 0.0767,
        "neutral": 0.8437,
        "mixed": 0.0527
      },
      "confidence": 0.7916
    }
  },
  "usage": {
    "input_tokens": 202,
    "output_tokens": 0,
    "cost": 0.000004242
  },
  "id": "gen-dec-1791517802-4N4SmoneLdU5PcIPipyO",
  "provider": "PrimeIntellect"
}

Deshi brand deser gorvo

“A local brand is the country's pride.”

What is the overall sentiment of this comment? · 4 options
Dataset answer: positive

  • ✓Nirnoy-flashpositive96%
  • ✓Nirnoy 12Bpositive95%
  • ✗Jevneutral50%
  • ✗Clefneutral56%
  • ✗Clef-flashneutral66%
  • ✗Gemma 12Bnegative73%
  • ✓Gemma E4Bpositive99%

“Gorvo” (pride) carries the praise.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "Deshi brand deser gorvo",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "What is the overall sentiment of this comment?",
      "criteria": {
        "positive": "praise, thanks, satisfaction or happiness",
        "negative": "complaint, insult, disappointment or anger",
        "neutral": "a question, a request, a factual statement, or no clear opinion",
        "mixed": "clearly positive and clearly negative at the same time"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.28,
        "negative": 0.21,
        "neutral": 0.5,
        "mixed": 0.01
      },
      "confidence": 0.33
    }
  },
  "usage": {
    "input_tokens": 375,
    "output_tokens": 46,
    "cost": 0.00001575
  },
  "id": "gen-dec-1791517791-uTAQGX3OhHDsI0AgdcEF",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.2214,
        "negative": 0.1075,
        "neutral": 0.5653,
        "mixed": 0.1058
      },
      "confidence": 0.4204
    }
  },
  "usage": {
    "input_tokens": 203,
    "output_tokens": 0,
    "cost": 0.000008526
  },
  "id": "gen-dec-1791517796-gQ7KqXvnyUecTEi1clx3",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.0658,
        "negative": 0.1904,
        "neutral": 0.6593,
        "mixed": 0.0845
      },
      "confidence": 0.5457
    }
  },
  "usage": {
    "input_tokens": 203,
    "output_tokens": 0,
    "cost": 0.000004263
  },
  "id": "gen-dec-1791517802-N0OwuvkkL5LFe2nyZtSX",
  "provider": "PrimeIntellect"
}

Gramer barite thakar mojai alada

“Staying at the village home is a special kind of fun.”

What is the overall sentiment of this comment? · 4 options
Dataset answer: positive

  • ✓Nirnoy-flashpositive83%
  • ✓Nirnoy 12Bpositive92%
  • ✗Jevneutral42%
  • ✗Clefneutral65%
  • ✗Clef-flashneutral45%
  • ✓Gemma 12Bpositive100%
  • ✓Gemma E4Bpositive100%

Fondness without any explicit praise word.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "Gramer barite thakar mojai alada",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "What is the overall sentiment of this comment?",
      "criteria": {
        "positive": "praise, thanks, satisfaction or happiness",
        "negative": "complaint, insult, disappointment or anger",
        "neutral": "a question, a request, a factual statement, or no clear opinion",
        "mixed": "clearly positive and clearly negative at the same time"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.22,
        "negative": 0.34,
        "neutral": 0.42,
        "mixed": 0.02
      },
      "confidence": 0.23
    }
  },
  "usage": {
    "input_tokens": 379,
    "output_tokens": 46,
    "cost": 0.000015918
  },
  "id": "gen-dec-1791517792-ytKjkprTAqsmLo2acmon",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.1449,
        "negative": 0.0928,
        "neutral": 0.6494,
        "mixed": 0.1129
      },
      "confidence": 0.5325
    }
  },
  "usage": {
    "input_tokens": 206,
    "output_tokens": 0,
    "cost": 0.000008652
  },
  "id": "gen-dec-1791517797-fP93DECe0nrZvH9ZJcmo",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.061,
        "negative": 0.401,
        "neutral": 0.4456,
        "mixed": 0.0923
      },
      "confidence": 0.2608
    }
  },
  "usage": {
    "input_tokens": 206,
    "output_tokens": 0,
    "cost": 0.000004326
  },
  "id": "gen-dec-1791517802-XDyEXw3xoIICgFfcI2pD",
  "provider": "PrimeIntellect"
}

Eto valo app jibone dehi nai... amare keo dhor..anonde moira gelam

“Never seen such a good app in my life... someone hold me, I'm dying of joy.”

What is the overall sentiment of this comment? · 4 options
Dataset answer: positive

  • ✓Nirnoy-flashpositive76%
  • ✓Nirnoy 12Bpositive90%
  • ✗Jevnegative73%
  • ✗Clefneutral38%
  • ✗Clef-flashnegative40%
  • ✗Gemma 12Bnegative100%
  • ✗Gemma E4Bnegative100%

“Dying of joy” is praise, not distress.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "Eto valo app jibone dehi nai... amare keo dhor..anonde moira gelam",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "What is the overall sentiment of this comment?",
      "criteria": {
        "positive": "praise, thanks, satisfaction or happiness",
        "negative": "complaint, insult, disappointment or anger",
        "neutral": "a question, a request, a factual statement, or no clear opinion",
        "mixed": "clearly positive and clearly negative at the same time"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "negative",
      "probabilities": {
        "positive": 0.19,
        "negative": 0.73,
        "neutral": 0.01,
        "mixed": 0.07
      },
      "confidence": 0.64
    }
  },
  "usage": {
    "input_tokens": 391,
    "output_tokens": 46,
    "cost": 0.000016422
  },
  "id": "gen-dec-1791517792-0wHucVwmeSwAHSyydeD9",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.1752,
        "negative": 0.2951,
        "neutral": 0.376,
        "mixed": 0.1537
      },
      "confidence": 0.168
    }
  },
  "usage": {
    "input_tokens": 219,
    "output_tokens": 0,
    "cost": 0.000009198
  },
  "id": "gen-dec-1791517797-jD5MVdruuH1Ag5y8xXRl",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "negative",
      "probabilities": {
        "positive": 0.0769,
        "negative": 0.4027,
        "neutral": 0.2946,
        "mixed": 0.2259
      },
      "confidence": 0.2036
    }
  },
  "usage": {
    "input_tokens": 219,
    "output_tokens": 0,
    "cost": 0.000004599
  },
  "id": "gen-dec-1791517802-NAATKeOoU6FIuXKGetnr",
  "provider": "PrimeIntellect"
}

Onak mojar akta natok chilo

“It was a very fun drama.”

What is the overall sentiment of this comment? · 4 options
Dataset answer: positive

  • ✗Nirnoy-flashnegative45%
  • ✓Nirnoy 12Bpositive95%
  • ✗Jevneutral59%
  • ✗Clefneutral76%
  • ✗Clef-flashneutral80%
  • ✓Gemma 12Bpositive100%
  • ✗Gemma E4Bneutral100%

“Mojar” means fun. Only Nirnoy 12B gets this one.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "Onak mojar akta natok chilo",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "What is the overall sentiment of this comment?",
      "criteria": {
        "positive": "praise, thanks, satisfaction or happiness",
        "negative": "complaint, insult, disappointment or anger",
        "neutral": "a question, a request, a factual statement, or no clear opinion",
        "mixed": "clearly positive and clearly negative at the same time"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.25,
        "negative": 0.14,
        "neutral": 0.59,
        "mixed": 0.02
      },
      "confidence": 0.45
    }
  },
  "usage": {
    "input_tokens": 379,
    "output_tokens": 46,
    "cost": 0.000015918
  },
  "id": "gen-dec-1791517790-EvZAE3hTv3Y6WRCnKMbh",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.0548,
        "negative": 0.0862,
        "neutral": 0.7627,
        "mixed": 0.0962
      },
      "confidence": 0.6836
    }
  },
  "usage": {
    "input_tokens": 206,
    "output_tokens": 0,
    "cost": 0.000008652
  },
  "id": "gen-dec-1791517795-c6lfdzNTCkBNd0rqoZap",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.053,
        "negative": 0.0847,
        "neutral": 0.7974,
        "mixed": 0.0649
      },
      "confidence": 0.7299
    }
  },
  "usage": {
    "input_tokens": 206,
    "output_tokens": 0,
    "cost": 0.000004326
  },
  "id": "gen-dec-1791517800-UoMJwuYacXIy2CCjCehR",
  "provider": "PrimeIntellect"
}

valo kintu Network nai

“Good, but there's no network.”

What is the overall sentiment of this comment? · 4 options
Dataset answer: mixed

  • ✓Nirnoy-flashmixed96%
  • ✓Nirnoy 12Bmixed95%
  • ✗Jevnegative49%
  • ✗Clefnegative48%
  • ✗Clef-flashnegative68%
  • ✓Gemma 12Bmixed100%
  • ✗Gemma E4Bnegative98%

Praise and a complaint in one line: the label is mixed.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "valo kintu Network nai",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "What is the overall sentiment of this comment?",
      "criteria": {
        "positive": "praise, thanks, satisfaction or happiness",
        "negative": "complaint, insult, disappointment or anger",
        "neutral": "a question, a request, a factual statement, or no clear opinion",
        "mixed": "clearly positive and clearly negative at the same time"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "negative",
      "probabilities": {
        "positive": 0.13,
        "negative": 0.49,
        "neutral": 0.2,
        "mixed": 0.18
      },
      "confidence": 0.32
    }
  },
  "usage": {
    "input_tokens": 374,
    "output_tokens": 46,
    "cost": 0.000015708
  },
  "id": "gen-dec-1791517789-L9Ky6VtvU3N3C08oLLH5",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "negative",
      "probabilities": {
        "positive": 0.0165,
        "negative": 0.4815,
        "neutral": 0.4541,
        "mixed": 0.0479
      },
      "confidence": 0.3087
    }
  },
  "usage": {
    "input_tokens": 202,
    "output_tokens": 0,
    "cost": 0.000008484
  },
  "id": "gen-dec-1791517795-Y5qRLeW0j9TjBPAf92qh",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "negative",
      "probabilities": {
        "positive": 0.0055,
        "negative": 0.6811,
        "neutral": 0.2605,
        "mixed": 0.0529
      },
      "confidence": 0.5748
    }
  },
  "usage": {
    "input_tokens": 202,
    "output_tokens": 0,
    "cost": 0.000004242
  },
  "id": "gen-dec-1791517800-e2GUascUNDZHrmOkaSyA",
  "provider": "PrimeIntellect"
}

Stocks vlo player but sakib er level er nh

“Stokes is a good player, but not at Shakib's level.”

What is the overall sentiment of this comment? · 4 options
Dataset answer: mixed

  • ✓Nirnoy-flashmixed79%
  • ✓Nirnoy 12Bmixed89%
  • ✗Jevnegative47%
  • ✗Clefneutral60%
  • ✗Clef-flashneutral49%
  • ✗Gemma 12Bnegative96%
  • ✗Gemma E4Bnegative98%

Cricket talk with praise and a put-down together: mixed.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "Stocks vlo player but sakib er level er nh",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "What is the overall sentiment of this comment?",
      "criteria": {
        "positive": "praise, thanks, satisfaction or happiness",
        "negative": "complaint, insult, disappointment or anger",
        "neutral": "a question, a request, a factual statement, or no clear opinion",
        "mixed": "clearly positive and clearly negative at the same time"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "negative",
      "probabilities": {
        "positive": 0.05,
        "negative": 0.47,
        "neutral": 0.01,
        "mixed": 0.47
      },
      "confidence": 0.29
    }
  },
  "usage": {
    "input_tokens": 380,
    "output_tokens": 46,
    "cost": 0.00001596
  },
  "id": "gen-dec-1791517791-q3alnTCLRrwKsMdAjyIK",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.034,
        "negative": 0.2041,
        "neutral": 0.5999,
        "mixed": 0.1621
      },
      "confidence": 0.4665
    }
  },
  "usage": {
    "input_tokens": 207,
    "output_tokens": 0,
    "cost": 0.000008694
  },
  "id": "gen-dec-1791517797-sOZWOy0K4AFISCVO71wm",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.0371,
        "negative": 0.2575,
        "neutral": 0.4886,
        "mixed": 0.2168
      },
      "confidence": 0.3181
    }
  },
  "usage": {
    "input_tokens": 207,
    "output_tokens": 0,
    "cost": 0.000004347
  },
  "id": "gen-dec-1791517802-UygeOka9QhpQKU4MZWnJ",
  "provider": "PrimeIntellect"
}

Football match ta onek intense chilo, kintu amar favorite team ta haray khub kharap laglo.

“The football match was very intense, but my favourite team lost and that felt really bad.”

What is the overall sentiment of this comment? · 4 options
Dataset answer: mixed

  • ✓Nirnoy-flashmixed97%
  • ✓Nirnoy 12Bmixed93%
  • ✗Jevnegative91%
  • ✗Clefnegative74%
  • ✗Clef-flashnegative79%
  • ✗Gemma 12Bnegative100%
  • ✗Gemma E4Bnegative92%

Excitement and disappointment in one line: the label is mixed.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "Football match ta onek intense chilo, kintu amar favorite team ta haray khub kharap laglo.",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "What is the overall sentiment of this comment?",
      "criteria": {
        "positive": "praise, thanks, satisfaction or happiness",
        "negative": "complaint, insult, disappointment or anger",
        "neutral": "a question, a request, a factual statement, or no clear opinion",
        "mixed": "clearly positive and clearly negative at the same time"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "negative",
      "probabilities": {
        "positive": 0,
        "negative": 0.91,
        "neutral": 0,
        "mixed": 0.09
      },
      "confidence": 0.88
    }
  },
  "usage": {
    "input_tokens": 393,
    "output_tokens": 46,
    "cost": 0.000016506
  },
  "id": "gen-dec-1791517790-tgNkvmxDM0VZdNC3RYnR",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "negative",
      "probabilities": {
        "positive": 0.0112,
        "negative": 0.7379,
        "neutral": 0.0369,
        "mixed": 0.2139
      },
      "confidence": 0.6505
    }
  },
  "usage": {
    "input_tokens": 222,
    "output_tokens": 0,
    "cost": 0.000009324
  },
  "id": "gen-dec-1791517795-x24jIBSJkPI5DEM96NV6",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by Cloudflare · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "negative",
      "probabilities": {
        "positive": 0.0356,
        "negative": 0.7935,
        "neutral": 0.0462,
        "mixed": 0.1247
      },
      "confidence": 0.5315
    }
  },
  "usage": {
    "input_tokens": 222,
    "output_tokens": 0,
    "cost": 0.00001998
  },
  "id": "gen-dec-1791517801-ownfItb5xtpSnATbCJn1",
  "provider": "Cloudflare"
}

Aajke cycling korlam, kintu jokhon basha phire aslam bike er tyre puncture hoye galo.

“I went cycling today, but when I got home the bike's tyre was punctured.”

What is the overall sentiment of this comment? · 4 options
Dataset answer: mixed

  • ✓Nirnoy-flashmixed90%
  • ✓Nirnoy 12Bmixed88%
  • ✗Jevnegative85%
  • ✗Clefnegative61%
  • ✗Clef-flashnegative71%
  • ✗Gemma 12Bnegative99%
  • ✗Gemma E4Bnegative96%

A good day with a bad ending: mixed.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "Aajke cycling korlam, kintu jokhon basha phire aslam bike er tyre puncture hoye galo.",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "What is the overall sentiment of this comment?",
      "criteria": {
        "positive": "praise, thanks, satisfaction or happiness",
        "negative": "complaint, insult, disappointment or anger",
        "neutral": "a question, a request, a factual statement, or no clear opinion",
        "mixed": "clearly positive and clearly negative at the same time"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "negative",
      "probabilities": {
        "positive": 0,
        "negative": 0.85,
        "neutral": 0.03,
        "mixed": 0.12
      },
      "confidence": 0.8
    }
  },
  "usage": {
    "input_tokens": 397,
    "output_tokens": 46,
    "cost": 0.000016674
  },
  "id": "gen-dec-1791517792-ZVSnYhzkdoQKQE1WGndo",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "negative",
      "probabilities": {
        "positive": 0.015,
        "negative": 0.605,
        "neutral": 0.2305,
        "mixed": 0.1494
      },
      "confidence": 0.4733
    }
  },
  "usage": {
    "input_tokens": 224,
    "output_tokens": 0,
    "cost": 0.000009408
  },
  "id": "gen-dec-1791517798-OscbO8S8WNpdUcWUV07D",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "negative",
      "probabilities": {
        "positive": 0.0063,
        "negative": 0.7062,
        "neutral": 0.1908,
        "mixed": 0.0967
      },
      "confidence": 0.6083
    }
  },
  "usage": {
    "input_tokens": 224,
    "output_tokens": 0,
    "cost": 0.000004704
  },
  "id": "gen-dec-1791517803-Q0gB7bfAplTCuSPnbvxG",
  "provider": "PrimeIntellect"
}

Class gula onk helpful tobe ar aktu agee dile valo hoy

“The classes are very helpful, but it would be better if they came a bit earlier.”

What is the overall sentiment of this comment? · 4 options
Dataset answer: mixed

  • ✓Nirnoy-flashmixed87%
  • ✓Nirnoy 12Bmixed83%
  • ✗Jevpositive72%
  • ✗Clefpositive59%
  • ✗Clef-flashneutral61%
  • ✗Gemma 12Bpositive50%
  • ✗Gemma E4Bpositive100%

Praise with a request: mixed, not simply positive.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "Class gula onk helpful tobe ar aktu agee dile valo hoy",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "What is the overall sentiment of this comment?",
      "criteria": {
        "positive": "praise, thanks, satisfaction or happiness",
        "negative": "complaint, insult, disappointment or anger",
        "neutral": "a question, a request, a factual statement, or no clear opinion",
        "mixed": "clearly positive and clearly negative at the same time"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "positive",
      "probabilities": {
        "positive": 0.72,
        "negative": 0.01,
        "neutral": 0.03,
        "mixed": 0.24
      },
      "confidence": 0.63
    }
  },
  "usage": {
    "input_tokens": 383,
    "output_tokens": 46,
    "cost": 0.000016086
  },
  "id": "gen-dec-1791517791-4fQhHMajpPlecWuuTg1m",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "positive",
      "probabilities": {
        "positive": 0.5866,
        "negative": 0.018,
        "neutral": 0.2603,
        "mixed": 0.1351
      },
      "confidence": 0.4488
    }
  },
  "usage": {
    "input_tokens": 210,
    "output_tokens": 0,
    "cost": 0.00000882
  },
  "id": "gen-dec-1791517796-qmV0ys3ao8hwCl1yA5me",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.1971,
        "negative": 0.0189,
        "neutral": 0.6046,
        "mixed": 0.1794
      },
      "confidence": 0.4728
    }
  },
  "usage": {
    "input_tokens": 210,
    "output_tokens": 0,
    "cost": 0.00000441
  },
  "id": "gen-dec-1791517802-J4BD6CyLQCF4w4Sr0ZM8",
  "provider": "PrimeIntellect"
}

Good. Tbe amr package golo oto tao vlo na. 7 days package hle beshi vlo hoii

“Good. But my package wasn't that good; a 7-day package would be better.”

What is the overall sentiment of this comment? · 4 options
Dataset answer: mixed

  • ✓Nirnoy-flashmixed87%
  • ✓Nirnoy 12Bmixed68%
  • ✗Jevpositive65%
  • ✗Clefpositive38%
  • ✗Clef-flashpositive74%
  • ✗Gemma 12Bpositive98%
  • ✗Gemma E4Bnegative75%

It opens with “Good”, then complains: mixed.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "Good. Tbe amr package golo oto tao vlo na. 7 days package hle beshi vlo hoii",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "What is the overall sentiment of this comment?",
      "criteria": {
        "positive": "praise, thanks, satisfaction or happiness",
        "negative": "complaint, insult, disappointment or anger",
        "neutral": "a question, a request, a factual statement, or no clear opinion",
        "mixed": "clearly positive and clearly negative at the same time"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "positive",
      "probabilities": {
        "positive": 0.65,
        "negative": 0.09,
        "neutral": 0.01,
        "mixed": 0.25
      },
      "confidence": 0.53
    }
  },
  "usage": {
    "input_tokens": 394,
    "output_tokens": 46,
    "cost": 0.000016548
  },
  "id": "gen-dec-1791517791-fXYDFoamIwAsbOMFGfbE",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "positive",
      "probabilities": {
        "positive": 0.3791,
        "negative": 0.0724,
        "neutral": 0.2486,
        "mixed": 0.2999
      },
      "confidence": 0.1721
    }
  },
  "usage": {
    "input_tokens": 222,
    "output_tokens": 0,
    "cost": 0.000009324
  },
  "id": "gen-dec-1791517796-VmdUa1VnoEZEJspc56Qd",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "positive",
      "probabilities": {
        "positive": 0.7393,
        "negative": 0.0045,
        "neutral": 0.1538,
        "mixed": 0.1024
      },
      "confidence": 0.6524
    }
  },
  "usage": {
    "input_tokens": 222,
    "output_tokens": 0,
    "cost": 0.000004662
  },
  "id": "gen-dec-1791517801-BWZJiReJQLMvaHkQAUfE",
  "provider": "PrimeIntellect"
}

Motamoti valo Add money hoina

“Fairly good, but Add Money doesn't work.”

What is the overall sentiment of this comment? · 4 options
Dataset answer: mixed

  • ✗Nirnoy-flashnegative63%
  • ✓Nirnoy 12Bmixed91%
  • ✗Jevnegative93%
  • ✗Clefnegative52%
  • ✗Clef-flashnegative63%
  • ✗Gemma 12Bnegative100%
  • ✗Gemma E4Bnegative100%

Mild praise plus a complaint: mixed. Only Nirnoy 12B gets this one.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "Motamoti valo Add money hoina",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "What is the overall sentiment of this comment?",
      "criteria": {
        "positive": "praise, thanks, satisfaction or happiness",
        "negative": "complaint, insult, disappointment or anger",
        "neutral": "a question, a request, a factual statement, or no clear opinion",
        "mixed": "clearly positive and clearly negative at the same time"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "negative",
      "probabilities": {
        "positive": 0.02,
        "negative": 0.93,
        "neutral": 0.04,
        "mixed": 0.01
      },
      "confidence": 0.91
    }
  },
  "usage": {
    "input_tokens": 377,
    "output_tokens": 46,
    "cost": 0.000015834
  },
  "id": "gen-dec-1791517792-yKVEUDG9kpAyaQf2RKEz",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "negative",
      "probabilities": {
        "positive": 0.0843,
        "negative": 0.5165,
        "neutral": 0.3133,
        "mixed": 0.086
      },
      "confidence": 0.3553
    }
  },
  "usage": {
    "input_tokens": 205,
    "output_tokens": 0,
    "cost": 0.00000861
  },
  "id": "gen-dec-1791517798-MiHt7a1sec9kelN0JhBF",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "negative",
      "probabilities": {
        "positive": 0.026,
        "negative": 0.6292,
        "neutral": 0.2685,
        "mixed": 0.0763
      },
      "confidence": 0.5056
    }
  },
  "usage": {
    "input_tokens": 205,
    "output_tokens": 0,
    "cost": 0.000004305
  },
  "id": "gen-dec-1791517803-lytSE9y6kqKaxxTWR6sg",
  "provider": "PrimeIntellect"
}

Whiteboard e dekhale bhalo hoy

“It would be better to show it on a whiteboard.”

What is the overall sentiment of this comment? · 4 options
Dataset answer: neutral

  • ✓Nirnoy-flashneutral76%
  • ✓Nirnoy 12Bneutral91%
  • ✗Jevpositive84%
  • ✗Clefpositive84%
  • ✗Clef-flashpositive66%
  • ✓Gemma 12Bneutral100%
  • ✗Gemma E4Bpositive100%

A suggestion, not praise: the label is neutral.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "Whiteboard e dekhale bhalo hoy",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "What is the overall sentiment of this comment?",
      "criteria": {
        "positive": "praise, thanks, satisfaction or happiness",
        "negative": "complaint, insult, disappointment or anger",
        "neutral": "a question, a request, a factual statement, or no clear opinion",
        "mixed": "clearly positive and clearly negative at the same time"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "positive",
      "probabilities": {
        "positive": 0.84,
        "negative": 0.07,
        "neutral": 0.09,
        "mixed": 0
      },
      "confidence": 0.79
    }
  },
  "usage": {
    "input_tokens": 380,
    "output_tokens": 46,
    "cost": 0.00001596
  },
  "id": "gen-dec-1791517793-eOn69TyT1Ca26YpDfplG",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "positive",
      "probabilities": {
        "positive": 0.8444,
        "negative": 0.0169,
        "neutral": 0.1082,
        "mixed": 0.0305
      },
      "confidence": 0.7925
    }
  },
  "usage": {
    "input_tokens": 206,
    "output_tokens": 0,
    "cost": 0.000008652
  },
  "id": "gen-dec-1791517798-YL4HvPw1sw2mxZ8ElFDB",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "positive",
      "probabilities": {
        "positive": 0.6604,
        "negative": 0.0079,
        "neutral": 0.2953,
        "mixed": 0.0364
      },
      "confidence": 0.5472
    }
  },
  "usage": {
    "input_tokens": 206,
    "output_tokens": 0,
    "cost": 0.000004326
  },
  "id": "gen-dec-1791517803-5GH6wvNEQi2QEuVJZGib",
  "provider": "PrimeIntellect"
}

age appsta valo silo akhon onek kharap

“The app used to be good, now it's very bad.”

What is the overall sentiment of this comment? · 4 options
Dataset answer: mixed

  • ✓Nirnoy-flashmixed95%
  • ✓Nirnoy 12Bmixed92%
  • ✗Jevnegative66%
  • ✗Clefnegative85%
  • ✗Clef-flashnegative93%
  • ✓Gemma 12Bmixed98%
  • ✗Gemma E4Bnegative100%

Praise for how it was and a complaint about now: mixed.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "age appsta valo silo akhon onek kharap",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "What is the overall sentiment of this comment?",
      "criteria": {
        "positive": "praise, thanks, satisfaction or happiness",
        "negative": "complaint, insult, disappointment or anger",
        "neutral": "a question, a request, a factual statement, or no clear opinion",
        "mixed": "clearly positive and clearly negative at the same time"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 04:06 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "negative",
      "probabilities": {
        "positive": 0.01,
        "negative": 0.66,
        "neutral": 0,
        "mixed": 0.33
      },
      "confidence": 0.55
    }
  },
  "usage": {
    "input_tokens": 380,
    "output_tokens": 46,
    "cost": 0.00001596
  },
  "id": "gen-dec-1791518777-A3GgGxq63uoGr65YZ38d",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 04:06 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "negative",
      "probabilities": {
        "positive": 0.0276,
        "negative": 0.851,
        "neutral": 0.0548,
        "mixed": 0.0667
      },
      "confidence": 0.8013
    }
  },
  "usage": {
    "input_tokens": 209,
    "output_tokens": 0,
    "cost": 0.000008778
  },
  "id": "gen-dec-1791518788-IL4Mk9HFBwxrhisNPlsB",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 04:06 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "negative",
      "probabilities": {
        "positive": 0.0045,
        "negative": 0.9289,
        "neutral": 0.0344,
        "mixed": 0.0323
      },
      "confidence": 0.9052
    }
  },
  "usage": {
    "input_tokens": 209,
    "output_tokens": 0,
    "cost": 0.000004389
  },
  "id": "gen-dec-1791518795-r81r6ykbszD30MRuMzGE",
  "provider": "PrimeIntellect"
}

Onnk valo laglo vai

“Really enjoyed it, bro.”

What is the overall sentiment of this comment? · 4 options
Dataset answer: positive

  • ✓Nirnoy-flashpositive98%
  • ✓Nirnoy 12Bpositive96%
  • ✗Jevnegative73%
  • ✗Clefneutral60%
  • ✗Clef-flashneutral86%
  • ✓Gemma 12Bpositive100%
  • ✓Gemma E4Bpositive97%

Plain Banglish praise: “valo laglo” means “liked it”.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "Onnk valo laglo vai",
  "questions": {
    "sentiment": {
      "type": "choice",
      "instructions": "What is the overall sentiment of this comment?",
      "criteria": {
        "positive": "praise, thanks, satisfaction or happiness",
        "negative": "complaint, insult, disappointment or anger",
        "neutral": "a question, a request, a factual statement, or no clear opinion",
        "mixed": "clearly positive and clearly negative at the same time"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 04:11 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "negative",
      "probabilities": {
        "positive": 0.15,
        "negative": 0.73,
        "neutral": 0.11,
        "mixed": 0.01
      },
      "confidence": 0.64
    }
  },
  "usage": {
    "input_tokens": 375,
    "output_tokens": 46,
    "cost": 0.00001575
  },
  "id": "gen-dec-1791519078-1c865y32DDdZIG7lBaTk",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 04:11 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.1725,
        "negative": 0.1601,
        "neutral": 0.5996,
        "mixed": 0.0678
      },
      "confidence": 0.4661
    }
  },
  "usage": {
    "input_tokens": 202,
    "output_tokens": 0,
    "cost": 0.000008484
  },
  "id": "gen-dec-1791519081-grr9fh2xPyLFVryukfzW",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 04:11 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "sentiment": {
      "type": "choice",
      "choice": "neutral",
      "probabilities": {
        "positive": 0.0635,
        "negative": 0.0385,
        "neutral": 0.8634,
        "mixed": 0.0345
      },
      "confidence": 0.8179
    }
  },
  "usage": {
    "input_tokens": 202,
    "output_tokens": 0,
    "cost": 0.000004242
  },
  "id": "gen-dec-1791519085-UrvULQiJ7vN872xibcVA",
  "provider": "PrimeIntellect"
}
Voice-assistant areaMASSIVE bn-BD · CC-BY-4.0

রেডিওতে কি গান আছে

“What song is on the radio?”

Which area does this request to a voice assistant belong to? · 18 options
Dataset answer: music

  • ✓Nirnoy-flashmusic75%
  • ✓Nirnoy 12Bmusic94%
  • ✗Jevplay80%
  • ✗Clefplay78%
  • ✗Clef-flashplay99%

Asking about a song is a music question, not a request to play something.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "রেডিওতে কি গান আছে",
  "questions": {
    "scenario": {
      "type": "choice",
      "instructions": "Which area does this request to a voice assistant belong to?",
      "criteria": {
        "alarm": "alarms",
        "audio": "device volume",
        "calendar": "calendar events and reminders",
        "cooking": "recipes and cooking",
        "datetime": "date, time and time zones",
        "email": "email and contacts",
        "general": "greetings, jokes and chit-chat",
        "iot": "smart-home devices: lights, plugs, vacuum, coffee machine",
        "lists": "to-do and shopping lists",
        "music": "music preferences and settings",
        "news": "news",
        "play": "play music, radio, podcasts, audiobooks or games",
        "qa": "factual questions, definitions, maths, stocks, currency",
        "recommendation": "recommendations for places, events or movies",
        "social": "social-media posts and complaints",
        "takeaway": "food delivery and takeaway",
        "transport": "taxis, tickets, traffic and trains",
        "weather": "weather"
      }
    },
    "intent": {
      "type": "choice",
      "instructions": "What does the user want the voice assistant to do?",
      "criteria": {
        "alarm_query": "ask about alarms that are set",
        "alarm_remove": "delete an alarm",
        "alarm_set": "set an alarm",
        "audio_volume_down": "lower the volume",
        "audio_volume_mute": "mute or silence the device",
        "audio_volume_other": "other volume request",
        "audio_volume_up": "raise the volume",
        "calendar_query": "ask about calendar events",
        "calendar_remove": "delete a calendar event",
        "calendar_set": "add an event or reminder",
        "cooking_query": "cooking question",
        "cooking_recipe": "ask for a recipe",
        "datetime_convert": "convert time or time zone",
        "datetime_query": "ask the date or time",
        "email_addcontact": "add an email contact",
        "email_query": "check or read emails",
        "email_querycontact": "ask about a contact's details",
        "email_sendemail": "send or reply to an email",
        "general_greet": "greeting",
        "general_joke": "ask for a joke",
        "general_quirky": "chit-chat or an odd request",
        "iot_cleaning": "start the robot vacuum",
        "iot_coffee": "make coffee",
        "iot_hue_lightchange": "change light colour",
        "iot_hue_lightdim": "dim the lights",
        "iot_hue_lightoff": "turn lights off",
        "iot_hue_lighton": "turn lights on",
        "iot_hue_lightup": "brighten the lights",
        "iot_wemo_off": "turn a smart plug off",
        "iot_wemo_on": "turn a smart plug on",
        "lists_createoradd": "create a list or add to it",
        "lists_query": "ask what is on a list",
        "lists_remove": "remove a list or an item",
        "music_dislikeness": "say a song or artist is disliked",
        "music_likeness": "say a song or artist is liked",
        "music_query": "ask what music is playing",
        "music_settings": "shuffle, repeat or other music settings",
        "news_query": "ask for news",
        "play_audiobook": "play an audiobook",
        "play_game": "play a game",
        "play_music": "play music",
        "play_podcasts": "play a podcast",
        "play_radio": "play the radio",
        "qa_currency": "currency exchange question",
        "qa_definition": "ask for a definition",
        "qa_factoid": "factual question",
        "qa_maths": "maths question",
        "qa_stock": "stock price question",
        "recommendation_events": "recommend events",
        "recommendation_locations": "recommend places",
        "recommendation_movies": "recommend movies",
        "social_post": "post on social media or complain to a company",
        "social_query": "ask about social-media updates",
        "takeaway_order": "order takeaway food",
        "takeaway_query": "ask about a takeaway order or restaurant",
        "transport_query": "ask about trains or transport",
        "transport_taxi": "book a taxi",
        "transport_ticket": "book a ticket",
        "transport_traffic": "ask about traffic",
        "weather_query": "ask about the weather"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "play",
      "probabilities": {
        "alarm": 0,
        "audio": 0,
        "calendar": 0,
        "cooking": 0,
        "datetime": 0,
        "email": 0,
        "general": 0,
        "iot": 0,
        "lists": 0,
        "music": 0.04,
        "news": 0,
        "play": 0.8,
        "qa": 0.16,
        "recommendation": 0,
        "social": 0,
        "takeaway": 0,
        "transport": 0,
        "weather": 0
      },
      "confidence": 0.79
    },
    "intent": {
      "type": "choice",
      "choice": "music_query",
      "probabilities": {
        "alarm_query": 0,
        "alarm_remove": 0,
        "alarm_set": 0,
        "audio_volume_down": 0,
        "audio_volume_mute": 0,
        "audio_volume_other": 0,
        "audio_volume_up": 0,
        "calendar_query": 0,
        "calendar_remove": 0,
        "calendar_set": 0,
        "cooking_query": 0,
        "cooking_recipe": 0,
        "datetime_convert": 0,
        "datetime_query": 0,
        "email_addcontact": 0,
        "email_query": 0,
        "email_querycontact": 0,
        "email_sendemail": 0,
        "general_greet": 0,
        "general_joke": 0,
        "general_quirky": 0,
        "iot_cleaning": 0,
        "iot_coffee": 0,
        "iot_hue_lightchange": 0,
        "iot_hue_lightdim": 0,
        "iot_hue_lightoff": 0,
        "iot_hue_lighton": 0,
        "iot_hue_lightup": 0,
        "iot_wemo_off": 0,
        "iot_wemo_on": 0,
        "lists_createoradd": 0,
        "lists_query": 0,
        "lists_remove": 0,
        "music_dislikeness": 0,
        "music_likeness": 0,
        "music_query": 0.84,
        "music_settings": 0,
        "news_query": 0,
        "play_audiobook": 0,
        "play_game": 0,
        "play_music": 0.01,
        "play_podcasts": 0,
        "play_radio": 0.14,
        "qa_currency": 0,
        "qa_definition": 0,
        "qa_factoid": 0.01,
        "qa_maths": 0,
        "qa_stock": 0,
        "recommendation_events": 0,
        "recommendation_locations": 0,
        "recommendation_movies": 0,
        "social_post": 0,
        "social_query": 0,
        "takeaway_order": 0,
        "takeaway_query": 0,
        "transport_query": 0,
        "transport_taxi": 0,
        "transport_ticket": 0,
        "transport_traffic": 0,
        "weather_query": 0
      },
      "confidence": 0.84
    }
  },
  "usage": {
    "input_tokens": 1680,
    "output_tokens": 719,
    "cost": 0.00007056
  },
  "id": "gen-dec-1791517790-2izB8e8xL5CueIRio80B",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "play",
      "probabilities": {
        "alarm": 0.0008,
        "audio": 0.0022,
        "calendar": 0.001,
        "cooking": 0.0007,
        "datetime": 0.0006,
        "email": 0.0008,
        "general": 0.003,
        "iot": 0.001,
        "lists": 0.0007,
        "music": 0.2054,
        "news": 0.0021,
        "play": 0.7752,
        "qa": 0.0019,
        "recommendation": 0.0015,
        "social": 0.0009,
        "takeaway": 0.0006,
        "transport": 0.0007,
        "weather": 0.0009
      },
      "confidence": 0.762
    },
    "intent": {
      "type": "choice",
      "choice": "music_query",
      "probabilities": {
        "alarm_query": 0.0023,
        "alarm_remove": 0.0002,
        "alarm_set": 0.0002,
        "audio_volume_down": 0.0003,
        "audio_volume_mute": 0.0004,
        "audio_volume_other": 0.0005,
        "audio_volume_up": 0.0003,
        "calendar_query": 0.0012,
        "calendar_remove": 0.0002,
        "calendar_set": 0.0003,
        "cooking_query": 0.0005,
        "cooking_recipe": 0.0006,
        "datetime_convert": 0.0002,
        "datetime_query": 0.0006,
        "email_addcontact": 0.0002,
        "email_query": 0.0006,
        "email_querycontact": 0.0006,
        "email_sendemail": 0.0003,
        "general_greet": 0.0003,
        "general_joke": 0.0004,
        "general_quirky": 0.0022,
        "iot_cleaning": 0.0003,
        "iot_coffee": 0.0003,
        "iot_hue_lightchange": 0.0002,
        "iot_hue_lightdim": 0.0002,
        "iot_hue_lightoff": 0.0002,
        "iot_hue_lighton": 0.0002,
        "iot_hue_lightup": 0.0002,
        "iot_wemo_off": 0.0002,
        "iot_wemo_on": 0.0003,
        "lists_createoradd": 0.0003,
        "lists_query": 0.0045,
        "lists_remove": 0.0003,
        "music_dislikeness": 0.0012,
        "music_likeness": 0.0011,
        "music_query": 0.6575,
        "music_settings": 0.0066,
        "news_query": 0.004,
        "play_audiobook": 0.0021,
        "play_game": 0.0006,
        "play_music": 0.0392,
        "play_podcasts": 0.0063,
        "play_radio": 0.2535,
        "qa_currency": 0.0003,
        "qa_definition": 0.0005,
        "qa_factoid": 0.0013,
        "qa_maths": 0.0003,
        "qa_stock": 0.0004,
        "recommendation_events": 0.0004,
        "recommendation_locations": 0.0003,
        "recommendation_movies": 0.0004,
        "social_post": 0.0004,
        "social_query": 0.0007,
        "takeaway_order": 0.0002,
        "takeaway_query": 0.0006,
        "transport_query": 0.0005,
        "transport_taxi": 0.0003,
        "transport_ticket": 0.0003,
        "transport_traffic": 0.0004,
        "weather_query": 0.0007
      },
      "confidence": 0.6517
    }
  },
  "usage": {
    "input_tokens": 1843,
    "output_tokens": 0,
    "cost": 0.000077406
  },
  "id": "gen-dec-1791517795-w0MTalQmLbr1GuSyidud",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "play",
      "probabilities": {
        "alarm": 0.0004,
        "audio": 0.0004,
        "calendar": 0.0006,
        "cooking": 0.0005,
        "datetime": 0.0006,
        "email": 0.0004,
        "general": 0.0012,
        "iot": 0.0008,
        "lists": 0.0005,
        "music": 0.0015,
        "news": 0.001,
        "play": 0.9874,
        "qa": 0.0016,
        "recommendation": 0.001,
        "social": 0.0005,
        "takeaway": 0.0006,
        "transport": 0.0008,
        "weather": 0.0005
      },
      "confidence": 0.9867
    },
    "intent": {
      "type": "choice",
      "choice": "play_radio",
      "probabilities": {
        "alarm_query": 0.0009,
        "alarm_remove": 0.0008,
        "alarm_set": 0.0006,
        "audio_volume_down": 0.0007,
        "audio_volume_mute": 0.0008,
        "audio_volume_other": 0.0006,
        "audio_volume_up": 0.0008,
        "calendar_query": 0.001,
        "calendar_remove": 0.0008,
        "calendar_set": 0.0007,
        "cooking_query": 0.0008,
        "cooking_recipe": 0.0008,
        "datetime_convert": 0.0008,
        "datetime_query": 0.0009,
        "email_addcontact": 0.0007,
        "email_query": 0.0009,
        "email_querycontact": 0.0009,
        "email_sendemail": 0.0009,
        "general_greet": 0.0006,
        "general_joke": 0.0007,
        "general_quirky": 0.0023,
        "iot_cleaning": 0.0009,
        "iot_coffee": 0.0007,
        "iot_hue_lightchange": 0.0007,
        "iot_hue_lightdim": 0.0008,
        "iot_hue_lightoff": 0.0008,
        "iot_hue_lighton": 0.001,
        "iot_hue_lightup": 0.0009,
        "iot_wemo_off": 0.0009,
        "iot_wemo_on": 0.0011,
        "lists_createoradd": 0.0013,
        "lists_query": 0.0016,
        "lists_remove": 0.001,
        "music_dislikeness": 0.0009,
        "music_likeness": 0.0009,
        "music_query": 0.0648,
        "music_settings": 0.0011,
        "news_query": 0.002,
        "play_audiobook": 0.0048,
        "play_game": 0.0084,
        "play_music": 0.022,
        "play_podcasts": 0.0183,
        "play_radio": 0.8333,
        "qa_currency": 0.0006,
        "qa_definition": 0.0008,
        "qa_factoid": 0.0016,
        "qa_maths": 0.0006,
        "qa_stock": 0.0006,
        "recommendation_events": 0.0008,
        "recommendation_locations": 0.0007,
        "recommendation_movies": 0.0009,
        "social_post": 0.0011,
        "social_query": 0.0007,
        "takeaway_order": 0.0007,
        "takeaway_query": 0.001,
        "transport_query": 0.0009,
        "transport_taxi": 0.0008,
        "transport_ticket": 0.0008,
        "transport_traffic": 0.0008,
        "weather_query": 0.0009
      },
      "confidence": 0.8305
    }
  },
  "usage": {
    "input_tokens": 1843,
    "output_tokens": 0,
    "cost": 0.000038703
  },
  "id": "gen-dec-1791517801-0QaTditychxjIHCywwvX",
  "provider": "PrimeIntellect"
}
Voice-assistant areaMASSIVE bn-BD · CC-BY-4.0

অলি আজ রাত নয়টায় নিউ মার্কেট একটা ভালো ব্যান্ড বাজছে আমি যেতে চাই

“Olly, a good band is playing at New Market at nine tonight, I want to go.”

Which area does this request to a voice assistant belong to? · 18 options
Dataset answer: calendar

  • ✓Nirnoy-flashcalendar84%
  • ✗Nirnoy 12Brecommendation78%
  • ✗Jevrecommendation91%
  • ✗Clefrecommendation83%
  • ✗Clef-flashrecommendation50%

New Market is in Dhaka; a time and a plan to go belong in the calendar, not a request for suggestions.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "অলি আজ রাত নয়টায় নিউ মার্কেট একটা ভালো ব্যান্ড বাজছে আমি যেতে চাই",
  "questions": {
    "scenario": {
      "type": "choice",
      "instructions": "Which area does this request to a voice assistant belong to?",
      "criteria": {
        "alarm": "alarms",
        "audio": "device volume",
        "calendar": "calendar events and reminders",
        "cooking": "recipes and cooking",
        "datetime": "date, time and time zones",
        "email": "email and contacts",
        "general": "greetings, jokes and chit-chat",
        "iot": "smart-home devices: lights, plugs, vacuum, coffee machine",
        "lists": "to-do and shopping lists",
        "music": "music preferences and settings",
        "news": "news",
        "play": "play music, radio, podcasts, audiobooks or games",
        "qa": "factual questions, definitions, maths, stocks, currency",
        "recommendation": "recommendations for places, events or movies",
        "social": "social-media posts and complaints",
        "takeaway": "food delivery and takeaway",
        "transport": "taxis, tickets, traffic and trains",
        "weather": "weather"
      }
    },
    "intent": {
      "type": "choice",
      "instructions": "What does the user want the voice assistant to do?",
      "criteria": {
        "alarm_query": "ask about alarms that are set",
        "alarm_remove": "delete an alarm",
        "alarm_set": "set an alarm",
        "audio_volume_down": "lower the volume",
        "audio_volume_mute": "mute or silence the device",
        "audio_volume_other": "other volume request",
        "audio_volume_up": "raise the volume",
        "calendar_query": "ask about calendar events",
        "calendar_remove": "delete a calendar event",
        "calendar_set": "add an event or reminder",
        "cooking_query": "cooking question",
        "cooking_recipe": "ask for a recipe",
        "datetime_convert": "convert time or time zone",
        "datetime_query": "ask the date or time",
        "email_addcontact": "add an email contact",
        "email_query": "check or read emails",
        "email_querycontact": "ask about a contact's details",
        "email_sendemail": "send or reply to an email",
        "general_greet": "greeting",
        "general_joke": "ask for a joke",
        "general_quirky": "chit-chat or an odd request",
        "iot_cleaning": "start the robot vacuum",
        "iot_coffee": "make coffee",
        "iot_hue_lightchange": "change light colour",
        "iot_hue_lightdim": "dim the lights",
        "iot_hue_lightoff": "turn lights off",
        "iot_hue_lighton": "turn lights on",
        "iot_hue_lightup": "brighten the lights",
        "iot_wemo_off": "turn a smart plug off",
        "iot_wemo_on": "turn a smart plug on",
        "lists_createoradd": "create a list or add to it",
        "lists_query": "ask what is on a list",
        "lists_remove": "remove a list or an item",
        "music_dislikeness": "say a song or artist is disliked",
        "music_likeness": "say a song or artist is liked",
        "music_query": "ask what music is playing",
        "music_settings": "shuffle, repeat or other music settings",
        "news_query": "ask for news",
        "play_audiobook": "play an audiobook",
        "play_game": "play a game",
        "play_music": "play music",
        "play_podcasts": "play a podcast",
        "play_radio": "play the radio",
        "qa_currency": "currency exchange question",
        "qa_definition": "ask for a definition",
        "qa_factoid": "factual question",
        "qa_maths": "maths question",
        "qa_stock": "stock price question",
        "recommendation_events": "recommend events",
        "recommendation_locations": "recommend places",
        "recommendation_movies": "recommend movies",
        "social_post": "post on social media or complain to a company",
        "social_query": "ask about social-media updates",
        "takeaway_order": "order takeaway food",
        "takeaway_query": "ask about a takeaway order or restaurant",
        "transport_query": "ask about trains or transport",
        "transport_taxi": "book a taxi",
        "transport_ticket": "book a ticket",
        "transport_traffic": "ask about traffic",
        "weather_query": "ask about the weather"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 04:11 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "recommendation",
      "probabilities": {
        "alarm": 0,
        "audio": 0,
        "calendar": 0.04,
        "cooking": 0,
        "datetime": 0,
        "email": 0,
        "general": 0.02,
        "iot": 0,
        "lists": 0,
        "music": 0.01,
        "news": 0,
        "play": 0.01,
        "qa": 0,
        "recommendation": 0.91,
        "social": 0,
        "takeaway": 0,
        "transport": 0.01,
        "weather": 0
      },
      "confidence": 0.9
    },
    "intent": {
      "type": "choice",
      "choice": "recommendation_events",
      "probabilities": {
        "alarm_query": 0,
        "alarm_remove": 0,
        "alarm_set": 0.01,
        "audio_volume_down": 0,
        "audio_volume_mute": 0,
        "audio_volume_other": 0,
        "audio_volume_up": 0,
        "calendar_query": 0,
        "calendar_remove": 0,
        "calendar_set": 0.11,
        "cooking_query": 0,
        "cooking_recipe": 0,
        "datetime_convert": 0,
        "datetime_query": 0,
        "email_addcontact": 0,
        "email_query": 0,
        "email_querycontact": 0,
        "email_sendemail": 0,
        "general_greet": 0,
        "general_joke": 0,
        "general_quirky": 0.18000000000000002,
        "iot_cleaning": 0,
        "iot_coffee": 0,
        "iot_hue_lightchange": 0,
        "iot_hue_lightdim": 0,
        "iot_hue_lightoff": 0,
        "iot_hue_lighton": 0,
        "iot_hue_lightup": 0,
        "iot_wemo_off": 0,
        "iot_wemo_on": 0,
        "lists_createoradd": 0,
        "lists_query": 0,
        "lists_remove": 0,
        "music_dislikeness": 0,
        "music_likeness": 0.01,
        "music_query": 0.01,
        "music_settings": 0,
        "news_query": 0,
        "play_audiobook": 0,
        "play_game": 0,
        "play_music": 0.02,
        "play_podcasts": 0,
        "play_radio": 0,
        "qa_currency": 0,
        "qa_definition": 0,
        "qa_factoid": 0.01,
        "qa_maths": 0,
        "qa_stock": 0,
        "recommendation_events": 0.54,
        "recommendation_locations": 0.08,
        "recommendation_movies": 0,
        "social_post": 0,
        "social_query": 0,
        "takeaway_order": 0,
        "takeaway_query": 0,
        "transport_query": 0,
        "transport_taxi": 0.01,
        "transport_ticket": 0,
        "transport_traffic": 0.01,
        "weather_query": 0
      },
      "confidence": 0.53
    }
  },
  "usage": {
    "input_tokens": 1720,
    "output_tokens": 725,
    "cost": 0.00007224
  },
  "id": "gen-dec-1791519079-6r3e7QU7NSasOGpuT70n",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 04:11 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "recommendation",
      "probabilities": {
        "alarm": 0.0012,
        "audio": 0.0015,
        "calendar": 0.1151,
        "cooking": 0.0007,
        "datetime": 0.0014,
        "email": 0.0091,
        "general": 0.0056,
        "iot": 0.0007,
        "lists": 0.0013,
        "music": 0.0087,
        "news": 0.0012,
        "play": 0.0073,
        "qa": 0.0012,
        "recommendation": 0.8339,
        "social": 0.0031,
        "takeaway": 0.0007,
        "transport": 0.0042,
        "weather": 0.0031
      },
      "confidence": 0.8241
    },
    "intent": {
      "type": "choice",
      "choice": "recommendation_events",
      "probabilities": {
        "alarm_query": 0.0011,
        "alarm_remove": 0.0007,
        "alarm_set": 0.0057,
        "audio_volume_down": 0.0006,
        "audio_volume_mute": 0.001,
        "audio_volume_other": 0.0009,
        "audio_volume_up": 0.0012,
        "calendar_query": 0.0329,
        "calendar_remove": 0.0088,
        "calendar_set": 0.1545,
        "cooking_query": 0.0005,
        "cooking_recipe": 0.0009,
        "datetime_convert": 0.0005,
        "datetime_query": 0.001,
        "email_addcontact": 0.0012,
        "email_query": 0.002,
        "email_querycontact": 0.0027,
        "email_sendemail": 0.0117,
        "general_greet": 0.0012,
        "general_joke": 0.001,
        "general_quirky": 0.0259,
        "iot_cleaning": 0.0005,
        "iot_coffee": 0.0004,
        "iot_hue_lightchange": 0.0004,
        "iot_hue_lightdim": 0.0005,
        "iot_hue_lightoff": 0.0004,
        "iot_hue_lighton": 0.0005,
        "iot_hue_lightup": 0.0006,
        "iot_wemo_off": 0.0005,
        "iot_wemo_on": 0.0006,
        "lists_createoradd": 0.0013,
        "lists_query": 0.001,
        "lists_remove": 0.0007,
        "music_dislikeness": 0.0011,
        "music_likeness": 0.0039,
        "music_query": 0.0012,
        "music_settings": 0.0014,
        "news_query": 0.0013,
        "play_audiobook": 0.0012,
        "play_game": 0.0007,
        "play_music": 0.0196,
        "play_podcasts": 0.0018,
        "play_radio": 0.0053,
        "qa_currency": 0.0004,
        "qa_definition": 0.0007,
        "qa_factoid": 0.0027,
        "qa_maths": 0.0004,
        "qa_stock": 0.0005,
        "recommendation_events": 0.6405,
        "recommendation_locations": 0.0224,
        "recommendation_movies": 0.0055,
        "social_post": 0.002,
        "social_query": 0.0009,
        "takeaway_order": 0.0006,
        "takeaway_query": 0.0008,
        "transport_query": 0.0026,
        "transport_taxi": 0.0021,
        "transport_ticket": 0.0094,
        "transport_traffic": 0.0009,
        "weather_query": 0.0066
      },
      "confidence": 0.6344
    }
  },
  "usage": {
    "input_tokens": 1872,
    "output_tokens": 0,
    "cost": 0.000078624
  },
  "id": "gen-dec-1791519083-IBcAhk7I5mMJewwu4Oqq",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 04:11 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "recommendation",
      "probabilities": {
        "alarm": 0.0016,
        "audio": 0.0016,
        "calendar": 0.4328,
        "cooking": 0.001,
        "datetime": 0.0019,
        "email": 0.0012,
        "general": 0.0158,
        "iot": 0.0026,
        "lists": 0.0025,
        "music": 0.0074,
        "news": 0.0026,
        "play": 0.0071,
        "qa": 0.0032,
        "recommendation": 0.4981,
        "social": 0.0103,
        "takeaway": 0.0019,
        "transport": 0.0039,
        "weather": 0.0044
      },
      "confidence": 0.4686
    },
    "intent": {
      "type": "choice",
      "choice": "calendar_set",
      "probabilities": {
        "alarm_query": 0.0016,
        "alarm_remove": 0.001,
        "alarm_set": 0.0021,
        "audio_volume_down": 0.001,
        "audio_volume_mute": 0.0011,
        "audio_volume_other": 0.001,
        "audio_volume_up": 0.0013,
        "calendar_query": 0.088,
        "calendar_remove": 0.0094,
        "calendar_set": 0.3824,
        "cooking_query": 0.0007,
        "cooking_recipe": 0.0011,
        "datetime_convert": 0.0009,
        "datetime_query": 0.0016,
        "email_addcontact": 0.0009,
        "email_query": 0.0008,
        "email_querycontact": 0.001,
        "email_sendemail": 0.001,
        "general_greet": 0.0027,
        "general_joke": 0.0012,
        "general_quirky": 0.0246,
        "iot_cleaning": 0.0009,
        "iot_coffee": 0.0008,
        "iot_hue_lightchange": 0.0007,
        "iot_hue_lightdim": 0.0008,
        "iot_hue_lightoff": 0.001,
        "iot_hue_lighton": 0.0008,
        "iot_hue_lightup": 0.0007,
        "iot_wemo_off": 0.0009,
        "iot_wemo_on": 0.0013,
        "lists_createoradd": 0.0018,
        "lists_query": 0.001,
        "lists_remove": 0.0012,
        "music_dislikeness": 0.0025,
        "music_likeness": 0.0119,
        "music_query": 0.0046,
        "music_settings": 0.0039,
        "news_query": 0.0025,
        "play_audiobook": 0.0011,
        "play_game": 0.001,
        "play_music": 0.0047,
        "play_podcasts": 0.0016,
        "play_radio": 0.0052,
        "qa_currency": 0.0008,
        "qa_definition": 0.001,
        "qa_factoid": 0.0023,
        "qa_maths": 0.0008,
        "qa_stock": 0.0009,
        "recommendation_events": 0.3592,
        "recommendation_locations": 0.0248,
        "recommendation_movies": 0.0097,
        "social_post": 0.0104,
        "social_query": 0.0013,
        "takeaway_order": 0.0009,
        "takeaway_query": 0.0019,
        "transport_query": 0.0013,
        "transport_taxi": 0.0015,
        "transport_ticket": 0.0021,
        "transport_traffic": 0.001,
        "weather_query": 0.0035
      },
      "confidence": 0.3719
    }
  },
  "usage": {
    "input_tokens": 1872,
    "output_tokens": 0,
    "cost": 0.000039312
  },
  "id": "gen-dec-1791519086-bxJrFmZ8Z5Jejb8KiyKl",
  "provider": "PrimeIntellect"
}
Voice-assistant areaMASSIVE bn-BD · CC-BY-4.0

কি ঘটনা শীঘ্রই ঘটছে

“What events are happening soon?”

Which area does this request to a voice assistant belong to? · 18 options
Dataset answer: recommendation

  • ✓Nirnoy-flashrecommendation89%
  • ✗Nirnoy 12Bcalendar92%
  • ✗Jevnews48%
  • ✗Clefcalendar89%
  • ✗Clef-flashcalendar83%

Events in general, not the user's own calendar: a request for suggestions.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "কি ঘটনা শীঘ্রই ঘটছে",
  "questions": {
    "scenario": {
      "type": "choice",
      "instructions": "Which area does this request to a voice assistant belong to?",
      "criteria": {
        "alarm": "alarms",
        "audio": "device volume",
        "calendar": "calendar events and reminders",
        "cooking": "recipes and cooking",
        "datetime": "date, time and time zones",
        "email": "email and contacts",
        "general": "greetings, jokes and chit-chat",
        "iot": "smart-home devices: lights, plugs, vacuum, coffee machine",
        "lists": "to-do and shopping lists",
        "music": "music preferences and settings",
        "news": "news",
        "play": "play music, radio, podcasts, audiobooks or games",
        "qa": "factual questions, definitions, maths, stocks, currency",
        "recommendation": "recommendations for places, events or movies",
        "social": "social-media posts and complaints",
        "takeaway": "food delivery and takeaway",
        "transport": "taxis, tickets, traffic and trains",
        "weather": "weather"
      }
    },
    "intent": {
      "type": "choice",
      "instructions": "What does the user want the voice assistant to do?",
      "criteria": {
        "alarm_query": "ask about alarms that are set",
        "alarm_remove": "delete an alarm",
        "alarm_set": "set an alarm",
        "audio_volume_down": "lower the volume",
        "audio_volume_mute": "mute or silence the device",
        "audio_volume_other": "other volume request",
        "audio_volume_up": "raise the volume",
        "calendar_query": "ask about calendar events",
        "calendar_remove": "delete a calendar event",
        "calendar_set": "add an event or reminder",
        "cooking_query": "cooking question",
        "cooking_recipe": "ask for a recipe",
        "datetime_convert": "convert time or time zone",
        "datetime_query": "ask the date or time",
        "email_addcontact": "add an email contact",
        "email_query": "check or read emails",
        "email_querycontact": "ask about a contact's details",
        "email_sendemail": "send or reply to an email",
        "general_greet": "greeting",
        "general_joke": "ask for a joke",
        "general_quirky": "chit-chat or an odd request",
        "iot_cleaning": "start the robot vacuum",
        "iot_coffee": "make coffee",
        "iot_hue_lightchange": "change light colour",
        "iot_hue_lightdim": "dim the lights",
        "iot_hue_lightoff": "turn lights off",
        "iot_hue_lighton": "turn lights on",
        "iot_hue_lightup": "brighten the lights",
        "iot_wemo_off": "turn a smart plug off",
        "iot_wemo_on": "turn a smart plug on",
        "lists_createoradd": "create a list or add to it",
        "lists_query": "ask what is on a list",
        "lists_remove": "remove a list or an item",
        "music_dislikeness": "say a song or artist is disliked",
        "music_likeness": "say a song or artist is liked",
        "music_query": "ask what music is playing",
        "music_settings": "shuffle, repeat or other music settings",
        "news_query": "ask for news",
        "play_audiobook": "play an audiobook",
        "play_game": "play a game",
        "play_music": "play music",
        "play_podcasts": "play a podcast",
        "play_radio": "play the radio",
        "qa_currency": "currency exchange question",
        "qa_definition": "ask for a definition",
        "qa_factoid": "factual question",
        "qa_maths": "maths question",
        "qa_stock": "stock price question",
        "recommendation_events": "recommend events",
        "recommendation_locations": "recommend places",
        "recommendation_movies": "recommend movies",
        "social_post": "post on social media or complain to a company",
        "social_query": "ask about social-media updates",
        "takeaway_order": "order takeaway food",
        "takeaway_query": "ask about a takeaway order or restaurant",
        "transport_query": "ask about trains or transport",
        "transport_taxi": "book a taxi",
        "transport_ticket": "book a ticket",
        "transport_traffic": "ask about traffic",
        "weather_query": "ask about the weather"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 04:11 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "news",
      "probabilities": {
        "alarm": 0.01,
        "audio": 0,
        "calendar": 0.22,
        "cooking": 0,
        "datetime": 0.07,
        "email": 0,
        "general": 0.08,
        "iot": 0,
        "lists": 0,
        "music": 0,
        "news": 0.48,
        "play": 0,
        "qa": 0.11,
        "recommendation": 0.02,
        "social": 0,
        "takeaway": 0,
        "transport": 0,
        "weather": 0.01
      },
      "confidence": 0.45
    },
    "intent": {
      "type": "choice",
      "choice": "calendar_query",
      "probabilities": {
        "alarm_query": 0.03,
        "alarm_remove": 0,
        "alarm_set": 0,
        "audio_volume_down": 0,
        "audio_volume_mute": 0,
        "audio_volume_other": 0,
        "audio_volume_up": 0,
        "calendar_query": 0.52,
        "calendar_remove": 0,
        "calendar_set": 0,
        "cooking_query": 0,
        "cooking_recipe": 0,
        "datetime_convert": 0,
        "datetime_query": 0.04,
        "email_addcontact": 0,
        "email_query": 0,
        "email_querycontact": 0,
        "email_sendemail": 0,
        "general_greet": 0,
        "general_joke": 0,
        "general_quirky": 0.11,
        "iot_cleaning": 0,
        "iot_coffee": 0,
        "iot_hue_lightchange": 0,
        "iot_hue_lightdim": 0,
        "iot_hue_lightoff": 0,
        "iot_hue_lighton": 0,
        "iot_hue_lightup": 0,
        "iot_wemo_off": 0,
        "iot_wemo_on": 0,
        "lists_createoradd": 0,
        "lists_query": 0,
        "lists_remove": 0,
        "music_dislikeness": 0,
        "music_likeness": 0,
        "music_query": 0,
        "music_settings": 0,
        "news_query": 0.08,
        "play_audiobook": 0,
        "play_game": 0,
        "play_music": 0,
        "play_podcasts": 0,
        "play_radio": 0,
        "qa_currency": 0,
        "qa_definition": 0,
        "qa_factoid": 0.15000000000000002,
        "qa_maths": 0,
        "qa_stock": 0,
        "recommendation_events": 0.05,
        "recommendation_locations": 0,
        "recommendation_movies": 0,
        "social_post": 0,
        "social_query": 0,
        "takeaway_order": 0,
        "takeaway_query": 0,
        "transport_query": 0,
        "transport_taxi": 0,
        "transport_ticket": 0,
        "transport_traffic": 0,
        "weather_query": 0.01
      },
      "confidence": 0.51
    }
  },
  "usage": {
    "input_tokens": 1684,
    "output_tokens": 719,
    "cost": 0.000070728
  },
  "id": "gen-dec-1791519079-AbW0ZxAerRHqv1Tx0FTv",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 04:11 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "calendar",
      "probabilities": {
        "alarm": 0.0016,
        "audio": 0.0012,
        "calendar": 0.8861,
        "cooking": 0.0013,
        "datetime": 0.0026,
        "email": 0.0018,
        "general": 0.0075,
        "iot": 0.0011,
        "lists": 0.0018,
        "music": 0.0009,
        "news": 0.0632,
        "play": 0.0013,
        "qa": 0.0031,
        "recommendation": 0.021,
        "social": 0.0014,
        "takeaway": 0.0007,
        "transport": 0.0012,
        "weather": 0.0022
      },
      "confidence": 0.8794
    },
    "intent": {
      "type": "choice",
      "choice": "calendar_query",
      "probabilities": {
        "alarm_query": 0.0024,
        "alarm_remove": 0.0005,
        "alarm_set": 0.0007,
        "audio_volume_down": 0.0005,
        "audio_volume_mute": 0.0006,
        "audio_volume_other": 0.0006,
        "audio_volume_up": 0.0006,
        "calendar_query": 0.8257,
        "calendar_remove": 0.0082,
        "calendar_set": 0.007,
        "cooking_query": 0.0007,
        "cooking_recipe": 0.0012,
        "datetime_convert": 0.0006,
        "datetime_query": 0.0018,
        "email_addcontact": 0.0005,
        "email_query": 0.0026,
        "email_querycontact": 0.0009,
        "email_sendemail": 0.0007,
        "general_greet": 0.0019,
        "general_joke": 0.0007,
        "general_quirky": 0.0147,
        "iot_cleaning": 0.0007,
        "iot_coffee": 0.0006,
        "iot_hue_lightchange": 0.0004,
        "iot_hue_lightdim": 0.0005,
        "iot_hue_lightoff": 0.0004,
        "iot_hue_lighton": 0.0004,
        "iot_hue_lightup": 0.0006,
        "iot_wemo_off": 0.0005,
        "iot_wemo_on": 0.0005,
        "lists_createoradd": 0.001,
        "lists_query": 0.0035,
        "lists_remove": 0.0007,
        "music_dislikeness": 0.0005,
        "music_likeness": 0.0005,
        "music_query": 0.0013,
        "music_settings": 0.0006,
        "news_query": 0.0564,
        "play_audiobook": 0.0007,
        "play_game": 0.0006,
        "play_music": 0.0007,
        "play_podcasts": 0.0016,
        "play_radio": 0.0011,
        "qa_currency": 0.0006,
        "qa_definition": 0.0009,
        "qa_factoid": 0.0045,
        "qa_maths": 0.0005,
        "qa_stock": 0.0006,
        "recommendation_events": 0.0341,
        "recommendation_locations": 0.001,
        "recommendation_movies": 0.0011,
        "social_post": 0.0008,
        "social_query": 0.0034,
        "takeaway_order": 0.0004,
        "takeaway_query": 0.0009,
        "transport_query": 0.0009,
        "transport_taxi": 0.0006,
        "transport_ticket": 0.0006,
        "transport_traffic": 0.0007,
        "weather_query": 0.0018
      },
      "confidence": 0.8227
    }
  },
  "usage": {
    "input_tokens": 1842,
    "output_tokens": 0,
    "cost": 0.000077364
  },
  "id": "gen-dec-1791519082-kLk6fV9SPmZafJ6LeRCT",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 04:11 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "calendar",
      "probabilities": {
        "alarm": 0.0028,
        "audio": 0.0019,
        "calendar": 0.8343,
        "cooking": 0.0021,
        "datetime": 0.0098,
        "email": 0.0037,
        "general": 0.0079,
        "iot": 0.0031,
        "lists": 0.0049,
        "music": 0.0022,
        "news": 0.0712,
        "play": 0.0032,
        "qa": 0.0086,
        "recommendation": 0.0242,
        "social": 0.0049,
        "takeaway": 0.0022,
        "transport": 0.0054,
        "weather": 0.0076
      },
      "confidence": 0.8246
    },
    "intent": {
      "type": "choice",
      "choice": "calendar_query",
      "probabilities": {
        "alarm_query": 0.0021,
        "alarm_remove": 0.0012,
        "alarm_set": 0.0012,
        "audio_volume_down": 0.0012,
        "audio_volume_mute": 0.0012,
        "audio_volume_other": 0.0011,
        "audio_volume_up": 0.0014,
        "calendar_query": 0.7723,
        "calendar_remove": 0.0099,
        "calendar_set": 0.0076,
        "cooking_query": 0.0011,
        "cooking_recipe": 0.0012,
        "datetime_convert": 0.0013,
        "datetime_query": 0.0048,
        "email_addcontact": 0.001,
        "email_query": 0.0018,
        "email_querycontact": 0.0015,
        "email_sendemail": 0.0012,
        "general_greet": 0.0017,
        "general_joke": 0.0012,
        "general_quirky": 0.005,
        "iot_cleaning": 0.0013,
        "iot_coffee": 0.001,
        "iot_hue_lightchange": 0.0009,
        "iot_hue_lightdim": 0.0011,
        "iot_hue_lightoff": 0.0009,
        "iot_hue_lighton": 0.0009,
        "iot_hue_lightup": 0.0011,
        "iot_wemo_off": 0.001,
        "iot_wemo_on": 0.001,
        "lists_createoradd": 0.0014,
        "lists_query": 0.0021,
        "lists_remove": 0.0015,
        "music_dislikeness": 0.0011,
        "music_likeness": 0.0012,
        "music_query": 0.0014,
        "music_settings": 0.0013,
        "news_query": 0.0577,
        "play_audiobook": 0.0009,
        "play_game": 0.0009,
        "play_music": 0.001,
        "play_podcasts": 0.0011,
        "play_radio": 0.001,
        "qa_currency": 0.0011,
        "qa_definition": 0.0014,
        "qa_factoid": 0.0032,
        "qa_maths": 0.001,
        "qa_stock": 0.0014,
        "recommendation_events": 0.0675,
        "recommendation_locations": 0.0013,
        "recommendation_movies": 0.0016,
        "social_post": 0.0016,
        "social_query": 0.0039,
        "takeaway_order": 0.0009,
        "takeaway_query": 0.0013,
        "transport_query": 0.0022,
        "transport_taxi": 0.0011,
        "transport_ticket": 0.0012,
        "transport_traffic": 0.0019,
        "weather_query": 0.005
      },
      "confidence": 0.7684
    }
  },
  "usage": {
    "input_tokens": 1842,
    "output_tokens": 0,
    "cost": 0.000038682
  },
  "id": "gen-dec-1791519085-M2ADpDaWWn16fyEw7VJg",
  "provider": "PrimeIntellect"
}
Voice-assistant intentMASSIVE bn-BD · CC-BY-4.0

চ্যানেল আই থেকে বিজ্ঞপ্তি সেট করুন

“Set notifications from Channel i.”

What does the user want the voice assistant to do? · 60 options
Dataset answer: news query

  • ✓Nirnoy-flashnews query95%
  • ✓Nirnoy 12Bnews query86%
  • ✗Jevalarm set47%
  • ✗Clefalarm set41%
  • ✗Clef-flashcalendar set29%

Channel i is a Bangladeshi TV channel: the user wants news updates.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "চ্যানেল আই থেকে বিজ্ঞপ্তি সেট করুন",
  "questions": {
    "scenario": {
      "type": "choice",
      "instructions": "Which area does this request to a voice assistant belong to?",
      "criteria": {
        "alarm": "alarms",
        "audio": "device volume",
        "calendar": "calendar events and reminders",
        "cooking": "recipes and cooking",
        "datetime": "date, time and time zones",
        "email": "email and contacts",
        "general": "greetings, jokes and chit-chat",
        "iot": "smart-home devices: lights, plugs, vacuum, coffee machine",
        "lists": "to-do and shopping lists",
        "music": "music preferences and settings",
        "news": "news",
        "play": "play music, radio, podcasts, audiobooks or games",
        "qa": "factual questions, definitions, maths, stocks, currency",
        "recommendation": "recommendations for places, events or movies",
        "social": "social-media posts and complaints",
        "takeaway": "food delivery and takeaway",
        "transport": "taxis, tickets, traffic and trains",
        "weather": "weather"
      }
    },
    "intent": {
      "type": "choice",
      "instructions": "What does the user want the voice assistant to do?",
      "criteria": {
        "alarm_query": "ask about alarms that are set",
        "alarm_remove": "delete an alarm",
        "alarm_set": "set an alarm",
        "audio_volume_down": "lower the volume",
        "audio_volume_mute": "mute or silence the device",
        "audio_volume_other": "other volume request",
        "audio_volume_up": "raise the volume",
        "calendar_query": "ask about calendar events",
        "calendar_remove": "delete a calendar event",
        "calendar_set": "add an event or reminder",
        "cooking_query": "cooking question",
        "cooking_recipe": "ask for a recipe",
        "datetime_convert": "convert time or time zone",
        "datetime_query": "ask the date or time",
        "email_addcontact": "add an email contact",
        "email_query": "check or read emails",
        "email_querycontact": "ask about a contact's details",
        "email_sendemail": "send or reply to an email",
        "general_greet": "greeting",
        "general_joke": "ask for a joke",
        "general_quirky": "chit-chat or an odd request",
        "iot_cleaning": "start the robot vacuum",
        "iot_coffee": "make coffee",
        "iot_hue_lightchange": "change light colour",
        "iot_hue_lightdim": "dim the lights",
        "iot_hue_lightoff": "turn lights off",
        "iot_hue_lighton": "turn lights on",
        "iot_hue_lightup": "brighten the lights",
        "iot_wemo_off": "turn a smart plug off",
        "iot_wemo_on": "turn a smart plug on",
        "lists_createoradd": "create a list or add to it",
        "lists_query": "ask what is on a list",
        "lists_remove": "remove a list or an item",
        "music_dislikeness": "say a song or artist is disliked",
        "music_likeness": "say a song or artist is liked",
        "music_query": "ask what music is playing",
        "music_settings": "shuffle, repeat or other music settings",
        "news_query": "ask for news",
        "play_audiobook": "play an audiobook",
        "play_game": "play a game",
        "play_music": "play music",
        "play_podcasts": "play a podcast",
        "play_radio": "play the radio",
        "qa_currency": "currency exchange question",
        "qa_definition": "ask for a definition",
        "qa_factoid": "factual question",
        "qa_maths": "maths question",
        "qa_stock": "stock price question",
        "recommendation_events": "recommend events",
        "recommendation_locations": "recommend places",
        "recommendation_movies": "recommend movies",
        "social_post": "post on social media or complain to a company",
        "social_query": "ask about social-media updates",
        "takeaway_order": "order takeaway food",
        "takeaway_query": "ask about a takeaway order or restaurant",
        "transport_query": "ask about trains or transport",
        "transport_taxi": "book a taxi",
        "transport_ticket": "book a ticket",
        "transport_traffic": "ask about traffic",
        "weather_query": "ask about the weather"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "news",
      "probabilities": {
        "alarm": 0.19,
        "audio": 0,
        "calendar": 0.34,
        "cooking": 0,
        "datetime": 0,
        "email": 0,
        "general": 0.01,
        "iot": 0,
        "lists": 0,
        "music": 0,
        "news": 0.37,
        "play": 0.09,
        "qa": 0,
        "recommendation": 0,
        "social": 0,
        "takeaway": 0,
        "transport": 0,
        "weather": 0
      },
      "confidence": 0.33
    },
    "intent": {
      "type": "choice",
      "choice": "alarm_set",
      "probabilities": {
        "alarm_query": 0,
        "alarm_remove": 0,
        "alarm_set": 0.47,
        "audio_volume_down": 0,
        "audio_volume_mute": 0,
        "audio_volume_other": 0,
        "audio_volume_up": 0,
        "calendar_query": 0,
        "calendar_remove": 0,
        "calendar_set": 0.43,
        "cooking_query": 0,
        "cooking_recipe": 0,
        "datetime_convert": 0,
        "datetime_query": 0,
        "email_addcontact": 0,
        "email_query": 0,
        "email_querycontact": 0,
        "email_sendemail": 0,
        "general_greet": 0,
        "general_joke": 0,
        "general_quirky": 0.01,
        "iot_cleaning": 0,
        "iot_coffee": 0,
        "iot_hue_lightchange": 0,
        "iot_hue_lightdim": 0,
        "iot_hue_lightoff": 0,
        "iot_hue_lighton": 0,
        "iot_hue_lightup": 0,
        "iot_wemo_off": 0,
        "iot_wemo_on": 0,
        "lists_createoradd": 0.01,
        "lists_query": 0,
        "lists_remove": 0,
        "music_dislikeness": 0,
        "music_likeness": 0,
        "music_query": 0,
        "music_settings": 0,
        "news_query": 0.06,
        "play_audiobook": 0,
        "play_game": 0,
        "play_music": 0,
        "play_podcasts": 0,
        "play_radio": 0.01,
        "qa_currency": 0,
        "qa_definition": 0,
        "qa_factoid": 0,
        "qa_maths": 0,
        "qa_stock": 0,
        "recommendation_events": 0,
        "recommendation_locations": 0,
        "recommendation_movies": 0,
        "social_post": 0.01,
        "social_query": 0,
        "takeaway_order": 0,
        "takeaway_query": 0,
        "transport_query": 0,
        "transport_taxi": 0,
        "transport_ticket": 0,
        "transport_traffic": 0,
        "weather_query": 0
      },
      "confidence": 0.46
    }
  },
  "usage": {
    "input_tokens": 1693,
    "output_tokens": 719,
    "cost": 0.000071106
  },
  "id": "gen-dec-1791517790-ZREAX0bJHrnuoaO7PVRU",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "play",
      "probabilities": {
        "alarm": 0.2218,
        "audio": 0.0061,
        "calendar": 0.1576,
        "cooking": 0.0012,
        "datetime": 0.0033,
        "email": 0.0265,
        "general": 0.0027,
        "iot": 0.0051,
        "lists": 0.0048,
        "music": 0.0566,
        "news": 0.1312,
        "play": 0.2479,
        "qa": 0.0036,
        "recommendation": 0.003,
        "social": 0.1227,
        "takeaway": 0.0012,
        "transport": 0.0017,
        "weather": 0.0028
      },
      "confidence": 0.2037
    },
    "intent": {
      "type": "choice",
      "choice": "alarm_set",
      "probabilities": {
        "alarm_query": 0.0576,
        "alarm_remove": 0.0165,
        "alarm_set": 0.4063,
        "audio_volume_down": 0.0006,
        "audio_volume_mute": 0.0009,
        "audio_volume_other": 0.0011,
        "audio_volume_up": 0.0012,
        "calendar_query": 0.0064,
        "calendar_remove": 0.0059,
        "calendar_set": 0.0611,
        "cooking_query": 0.0003,
        "cooking_recipe": 0.0007,
        "datetime_convert": 0.0005,
        "datetime_query": 0.0003,
        "email_addcontact": 0.0017,
        "email_query": 0.007,
        "email_querycontact": 0.0016,
        "email_sendemail": 0.0057,
        "general_greet": 0.0004,
        "general_joke": 0.0004,
        "general_quirky": 0.0045,
        "iot_cleaning": 0.0005,
        "iot_coffee": 0.0004,
        "iot_hue_lightchange": 0.0012,
        "iot_hue_lightdim": 0.0003,
        "iot_hue_lightoff": 0.0004,
        "iot_hue_lighton": 0.0005,
        "iot_hue_lightup": 0.0006,
        "iot_wemo_off": 0.001,
        "iot_wemo_on": 0.0015,
        "lists_createoradd": 0.0019,
        "lists_query": 0.001,
        "lists_remove": 0.0014,
        "music_dislikeness": 0.0013,
        "music_likeness": 0.0046,
        "music_query": 0.0011,
        "music_settings": 0.0183,
        "news_query": 0.0427,
        "play_audiobook": 0.0062,
        "play_game": 0.0021,
        "play_music": 0.0405,
        "play_podcasts": 0.0848,
        "play_radio": 0.1667,
        "qa_currency": 0.0002,
        "qa_definition": 0.0008,
        "qa_factoid": 0.0014,
        "qa_maths": 0.0003,
        "qa_stock": 0.0004,
        "recommendation_events": 0.0007,
        "recommendation_locations": 0.0005,
        "recommendation_movies": 0.0006,
        "social_post": 0.0225,
        "social_query": 0.0093,
        "takeaway_order": 0.0003,
        "takeaway_query": 0.0004,
        "transport_query": 0.0007,
        "transport_taxi": 0.0004,
        "transport_ticket": 0.0007,
        "transport_traffic": 0.0004,
        "weather_query": 0.0009
      },
      "confidence": 0.3962
    }
  },
  "usage": {
    "input_tokens": 1848,
    "output_tokens": 0,
    "cost": 0.000077616
  },
  "id": "gen-dec-1791517795-xZ7rMIsSxax42MnBfH5L",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "calendar",
      "probabilities": {
        "alarm": 0.0292,
        "audio": 0.0038,
        "calendar": 0.545,
        "cooking": 0.0016,
        "datetime": 0.0032,
        "email": 0.0234,
        "general": 0.0043,
        "iot": 0.0241,
        "lists": 0.0049,
        "music": 0.03,
        "news": 0.0646,
        "play": 0.2021,
        "qa": 0.0059,
        "recommendation": 0.0037,
        "social": 0.034,
        "takeaway": 0.0026,
        "transport": 0.0041,
        "weather": 0.0134
      },
      "confidence": 0.5182
    },
    "intent": {
      "type": "choice",
      "choice": "calendar_set",
      "probabilities": {
        "alarm_query": 0.0046,
        "alarm_remove": 0.0016,
        "alarm_set": 0.0499,
        "audio_volume_down": 0.001,
        "audio_volume_mute": 0.0015,
        "audio_volume_other": 0.001,
        "audio_volume_up": 0.001,
        "calendar_query": 0.0303,
        "calendar_remove": 0.0104,
        "calendar_set": 0.2939,
        "cooking_query": 0.0007,
        "cooking_recipe": 0.0012,
        "datetime_convert": 0.0009,
        "datetime_query": 0.0011,
        "email_addcontact": 0.004,
        "email_query": 0.0081,
        "email_querycontact": 0.0016,
        "email_sendemail": 0.0038,
        "general_greet": 0.001,
        "general_joke": 0.0008,
        "general_quirky": 0.0076,
        "iot_cleaning": 0.0032,
        "iot_coffee": 0.0011,
        "iot_hue_lightchange": 0.0009,
        "iot_hue_lightdim": 0.0006,
        "iot_hue_lightoff": 0.0007,
        "iot_hue_lighton": 0.0011,
        "iot_hue_lightup": 0.0007,
        "iot_wemo_off": 0.0011,
        "iot_wemo_on": 0.0029,
        "lists_createoradd": 0.0022,
        "lists_query": 0.0011,
        "lists_remove": 0.0011,
        "music_dislikeness": 0.0023,
        "music_likeness": 0.0019,
        "music_query": 0.0016,
        "music_settings": 0.0046,
        "news_query": 0.0577,
        "play_audiobook": 0.013,
        "play_game": 0.0023,
        "play_music": 0.0257,
        "play_podcasts": 0.1147,
        "play_radio": 0.238,
        "qa_currency": 0.0007,
        "qa_definition": 0.0016,
        "qa_factoid": 0.0024,
        "qa_maths": 0.0007,
        "qa_stock": 0.0008,
        "recommendation_events": 0.0018,
        "recommendation_locations": 0.0012,
        "recommendation_movies": 0.0012,
        "social_post": 0.0495,
        "social_query": 0.0226,
        "takeaway_order": 0.0011,
        "takeaway_query": 0.001,
        "transport_query": 0.0012,
        "transport_taxi": 0.0014,
        "transport_ticket": 0.0023,
        "transport_traffic": 0.0009,
        "weather_query": 0.005
      },
      "confidence": 0.2819
    }
  },
  "usage": {
    "input_tokens": 1848,
    "output_tokens": 0,
    "cost": 0.000038808
  },
  "id": "gen-dec-1791517800-0uwqAji3HtkhUYzimO6f",
  "provider": "PrimeIntellect"
}
Voice-assistant intentMASSIVE bn-BD · CC-BY-4.0

ফেনি আজকে কোন মজার ঘটনা আছে

“Are there any fun events in Feni today?”

What does the user want the voice assistant to do? · 60 options
Dataset answer: recommendation events

  • ✓Nirnoy-flashrecommendation events96%
  • ✓Nirnoy 12Brecommendation events96%
  • ✗Jevgeneral joke30%
  • ✗Clefgeneral quirky50%
  • ✗Clef-flashgeneral joke58%

Feni is a town in Bangladesh; the user wants event suggestions, not a joke.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "ফেনি আজকে কোন মজার ঘটনা আছে",
  "questions": {
    "scenario": {
      "type": "choice",
      "instructions": "Which area does this request to a voice assistant belong to?",
      "criteria": {
        "alarm": "alarms",
        "audio": "device volume",
        "calendar": "calendar events and reminders",
        "cooking": "recipes and cooking",
        "datetime": "date, time and time zones",
        "email": "email and contacts",
        "general": "greetings, jokes and chit-chat",
        "iot": "smart-home devices: lights, plugs, vacuum, coffee machine",
        "lists": "to-do and shopping lists",
        "music": "music preferences and settings",
        "news": "news",
        "play": "play music, radio, podcasts, audiobooks or games",
        "qa": "factual questions, definitions, maths, stocks, currency",
        "recommendation": "recommendations for places, events or movies",
        "social": "social-media posts and complaints",
        "takeaway": "food delivery and takeaway",
        "transport": "taxis, tickets, traffic and trains",
        "weather": "weather"
      }
    },
    "intent": {
      "type": "choice",
      "instructions": "What does the user want the voice assistant to do?",
      "criteria": {
        "alarm_query": "ask about alarms that are set",
        "alarm_remove": "delete an alarm",
        "alarm_set": "set an alarm",
        "audio_volume_down": "lower the volume",
        "audio_volume_mute": "mute or silence the device",
        "audio_volume_other": "other volume request",
        "audio_volume_up": "raise the volume",
        "calendar_query": "ask about calendar events",
        "calendar_remove": "delete a calendar event",
        "calendar_set": "add an event or reminder",
        "cooking_query": "cooking question",
        "cooking_recipe": "ask for a recipe",
        "datetime_convert": "convert time or time zone",
        "datetime_query": "ask the date or time",
        "email_addcontact": "add an email contact",
        "email_query": "check or read emails",
        "email_querycontact": "ask about a contact's details",
        "email_sendemail": "send or reply to an email",
        "general_greet": "greeting",
        "general_joke": "ask for a joke",
        "general_quirky": "chit-chat or an odd request",
        "iot_cleaning": "start the robot vacuum",
        "iot_coffee": "make coffee",
        "iot_hue_lightchange": "change light colour",
        "iot_hue_lightdim": "dim the lights",
        "iot_hue_lightoff": "turn lights off",
        "iot_hue_lighton": "turn lights on",
        "iot_hue_lightup": "brighten the lights",
        "iot_wemo_off": "turn a smart plug off",
        "iot_wemo_on": "turn a smart plug on",
        "lists_createoradd": "create a list or add to it",
        "lists_query": "ask what is on a list",
        "lists_remove": "remove a list or an item",
        "music_dislikeness": "say a song or artist is disliked",
        "music_likeness": "say a song or artist is liked",
        "music_query": "ask what music is playing",
        "music_settings": "shuffle, repeat or other music settings",
        "news_query": "ask for news",
        "play_audiobook": "play an audiobook",
        "play_game": "play a game",
        "play_music": "play music",
        "play_podcasts": "play a podcast",
        "play_radio": "play the radio",
        "qa_currency": "currency exchange question",
        "qa_definition": "ask for a definition",
        "qa_factoid": "factual question",
        "qa_maths": "maths question",
        "qa_stock": "stock price question",
        "recommendation_events": "recommend events",
        "recommendation_locations": "recommend places",
        "recommendation_movies": "recommend movies",
        "social_post": "post on social media or complain to a company",
        "social_query": "ask about social-media updates",
        "takeaway_order": "order takeaway food",
        "takeaway_query": "ask about a takeaway order or restaurant",
        "transport_query": "ask about trains or transport",
        "transport_taxi": "book a taxi",
        "transport_ticket": "book a ticket",
        "transport_traffic": "ask about traffic",
        "weather_query": "ask about the weather"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "news",
      "probabilities": {
        "alarm": 0,
        "audio": 0,
        "calendar": 0.03,
        "cooking": 0,
        "datetime": 0,
        "email": 0,
        "general": 0.38,
        "iot": 0,
        "lists": 0,
        "music": 0,
        "news": 0.49,
        "play": 0,
        "qa": 0.03,
        "recommendation": 0.07,
        "social": 0,
        "takeaway": 0,
        "transport": 0,
        "weather": 0
      },
      "confidence": 0.46
    },
    "intent": {
      "type": "choice",
      "choice": "general_joke",
      "probabilities": {
        "alarm_query": 0,
        "alarm_remove": 0,
        "alarm_set": 0,
        "audio_volume_down": 0,
        "audio_volume_mute": 0,
        "audio_volume_other": 0,
        "audio_volume_up": 0,
        "calendar_query": 0.2,
        "calendar_remove": 0,
        "calendar_set": 0,
        "cooking_query": 0,
        "cooking_recipe": 0,
        "datetime_convert": 0,
        "datetime_query": 0,
        "email_addcontact": 0,
        "email_query": 0,
        "email_querycontact": 0,
        "email_sendemail": 0,
        "general_greet": 0,
        "general_joke": 0.3,
        "general_quirky": 0.23,
        "iot_cleaning": 0,
        "iot_coffee": 0,
        "iot_hue_lightchange": 0,
        "iot_hue_lightdim": 0,
        "iot_hue_lightoff": 0,
        "iot_hue_lighton": 0,
        "iot_hue_lightup": 0,
        "iot_wemo_off": 0,
        "iot_wemo_on": 0,
        "lists_createoradd": 0,
        "lists_query": 0,
        "lists_remove": 0,
        "music_dislikeness": 0,
        "music_likeness": 0,
        "music_query": 0,
        "music_settings": 0,
        "news_query": 0.15000000000000002,
        "play_audiobook": 0,
        "play_game": 0,
        "play_music": 0,
        "play_podcasts": 0,
        "play_radio": 0,
        "qa_currency": 0,
        "qa_definition": 0,
        "qa_factoid": 0.03,
        "qa_maths": 0,
        "qa_stock": 0,
        "recommendation_events": 0.09,
        "recommendation_locations": 0,
        "recommendation_movies": 0,
        "social_post": 0,
        "social_query": 0,
        "takeaway_order": 0,
        "takeaway_query": 0,
        "transport_query": 0,
        "transport_taxi": 0,
        "transport_ticket": 0,
        "transport_traffic": 0,
        "weather_query": 0
      },
      "confidence": 0.29
    }
  },
  "usage": {
    "input_tokens": 1686,
    "output_tokens": 720,
    "cost": 0.000070812
  },
  "id": "gen-dec-1791517791-gQ51HuwZ7h900WYg2BE3",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "general",
      "probabilities": {
        "alarm": 0.0007,
        "audio": 0.0007,
        "calendar": 0.0094,
        "cooking": 0.002,
        "datetime": 0.001,
        "email": 0.0025,
        "general": 0.9132,
        "iot": 0.0007,
        "lists": 0.001,
        "music": 0.0007,
        "news": 0.0424,
        "play": 0.0011,
        "qa": 0.006,
        "recommendation": 0.0113,
        "social": 0.0052,
        "takeaway": 0.0005,
        "transport": 0.0007,
        "weather": 0.001
      },
      "confidence": 0.9081
    },
    "intent": {
      "type": "choice",
      "choice": "general_quirky",
      "probabilities": {
        "alarm_query": 0.001,
        "alarm_remove": 0.0004,
        "alarm_set": 0.0005,
        "audio_volume_down": 0.0004,
        "audio_volume_mute": 0.0005,
        "audio_volume_other": 0.0006,
        "audio_volume_up": 0.0004,
        "calendar_query": 0.0293,
        "calendar_remove": 0.0013,
        "calendar_set": 0.0018,
        "cooking_query": 0.0014,
        "cooking_recipe": 0.0024,
        "datetime_convert": 0.0005,
        "datetime_query": 0.0012,
        "email_addcontact": 0.0005,
        "email_query": 0.0045,
        "email_querycontact": 0.0016,
        "email_sendemail": 0.0007,
        "general_greet": 0.0297,
        "general_joke": 0.2863,
        "general_quirky": 0.4986,
        "iot_cleaning": 0.0006,
        "iot_coffee": 0.0005,
        "iot_hue_lightchange": 0.0005,
        "iot_hue_lightdim": 0.0005,
        "iot_hue_lightoff": 0.0004,
        "iot_hue_lighton": 0.0005,
        "iot_hue_lightup": 0.0005,
        "iot_wemo_off": 0.0005,
        "iot_wemo_on": 0.0006,
        "lists_createoradd": 0.0007,
        "lists_query": 0.0028,
        "lists_remove": 0.0007,
        "music_dislikeness": 0.0005,
        "music_likeness": 0.0006,
        "music_query": 0.001,
        "music_settings": 0.0005,
        "news_query": 0.0641,
        "play_audiobook": 0.0007,
        "play_game": 0.0011,
        "play_music": 0.0007,
        "play_podcasts": 0.0017,
        "play_radio": 0.001,
        "qa_currency": 0.0006,
        "qa_definition": 0.0017,
        "qa_factoid": 0.0134,
        "qa_maths": 0.0006,
        "qa_stock": 0.0007,
        "recommendation_events": 0.0097,
        "recommendation_locations": 0.0014,
        "recommendation_movies": 0.0018,
        "social_post": 0.0015,
        "social_query": 0.0184,
        "takeaway_order": 0.0004,
        "takeaway_query": 0.0008,
        "transport_query": 0.0007,
        "transport_taxi": 0.0005,
        "transport_ticket": 0.0005,
        "transport_traffic": 0.0008,
        "weather_query": 0.0018
      },
      "confidence": 0.4901
    }
  },
  "usage": {
    "input_tokens": 1846,
    "output_tokens": 0,
    "cost": 0.000077532
  },
  "id": "gen-dec-1791517797-NBMK2nezirxYw3lLNiWj",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "general",
      "probabilities": {
        "alarm": 0.0004,
        "audio": 0.0003,
        "calendar": 0.0066,
        "cooking": 0.0005,
        "datetime": 0.0006,
        "email": 0.0014,
        "general": 0.961,
        "iot": 0.001,
        "lists": 0.0011,
        "music": 0.0008,
        "news": 0.0072,
        "play": 0.0014,
        "qa": 0.0043,
        "recommendation": 0.0056,
        "social": 0.005,
        "takeaway": 0.0006,
        "transport": 0.0008,
        "weather": 0.0014
      },
      "confidence": 0.9587
    },
    "intent": {
      "type": "choice",
      "choice": "general_joke",
      "probabilities": {
        "alarm_query": 0.0007,
        "alarm_remove": 0.0004,
        "alarm_set": 0.0004,
        "audio_volume_down": 0.0004,
        "audio_volume_mute": 0.0005,
        "audio_volume_other": 0.0004,
        "audio_volume_up": 0.0004,
        "calendar_query": 0.0139,
        "calendar_remove": 0.0004,
        "calendar_set": 0.0013,
        "cooking_query": 0.0008,
        "cooking_recipe": 0.0009,
        "datetime_convert": 0.0004,
        "datetime_query": 0.0006,
        "email_addcontact": 0.0004,
        "email_query": 0.0011,
        "email_querycontact": 0.0013,
        "email_sendemail": 0.0005,
        "general_greet": 0.0251,
        "general_joke": 0.5769,
        "general_quirky": 0.3186,
        "iot_cleaning": 0.0005,
        "iot_coffee": 0.0005,
        "iot_hue_lightchange": 0.0004,
        "iot_hue_lightdim": 0.0004,
        "iot_hue_lightoff": 0.0005,
        "iot_hue_lighton": 0.0005,
        "iot_hue_lightup": 0.0005,
        "iot_wemo_off": 0.0005,
        "iot_wemo_on": 0.0005,
        "lists_createoradd": 0.0007,
        "lists_query": 0.0008,
        "lists_remove": 0.0005,
        "music_dislikeness": 0.0006,
        "music_likeness": 0.0006,
        "music_query": 0.0006,
        "music_settings": 0.0006,
        "news_query": 0.0105,
        "play_audiobook": 0.0005,
        "play_game": 0.0007,
        "play_music": 0.0006,
        "play_podcasts": 0.0007,
        "play_radio": 0.0006,
        "qa_currency": 0.0004,
        "qa_definition": 0.0009,
        "qa_factoid": 0.0033,
        "qa_maths": 0.0004,
        "qa_stock": 0.0004,
        "recommendation_events": 0.0094,
        "recommendation_locations": 0.001,
        "recommendation_movies": 0.0014,
        "social_post": 0.0031,
        "social_query": 0.0072,
        "takeaway_order": 0.0004,
        "takeaway_query": 0.0006,
        "transport_query": 0.0006,
        "transport_taxi": 0.0005,
        "transport_ticket": 0.0004,
        "transport_traffic": 0.0005,
        "weather_query": 0.0017
      },
      "confidence": 0.5697
    }
  },
  "usage": {
    "input_tokens": 1846,
    "output_tokens": 0,
    "cost": 0.000038766
  },
  "id": "gen-dec-1791517802-PpRc1YmW6a7X0xEwAVRf",
  "provider": "PrimeIntellect"
}
Voice-assistant intentMASSIVE bn-BD · CC-BY-4.0

রাঙ্গামাটির সেরা উপায় কি

“What's the best way to get to Rangamati?”

What does the user want the voice assistant to do? · 60 options
Dataset answer: transport query

  • ✓Nirnoy-flashtransport query60%
  • ✓Nirnoy 12Btransport query47%
  • ✗Jevrecommendation locations65%
  • ✗Clefrecommendation locations29%
  • ✗Clef-flashcooking recipe87%

Rangamati is a district in Bangladesh: the user wants a route, not a place to visit.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "রাঙ্গামাটির সেরা উপায় কি",
  "questions": {
    "scenario": {
      "type": "choice",
      "instructions": "Which area does this request to a voice assistant belong to?",
      "criteria": {
        "alarm": "alarms",
        "audio": "device volume",
        "calendar": "calendar events and reminders",
        "cooking": "recipes and cooking",
        "datetime": "date, time and time zones",
        "email": "email and contacts",
        "general": "greetings, jokes and chit-chat",
        "iot": "smart-home devices: lights, plugs, vacuum, coffee machine",
        "lists": "to-do and shopping lists",
        "music": "music preferences and settings",
        "news": "news",
        "play": "play music, radio, podcasts, audiobooks or games",
        "qa": "factual questions, definitions, maths, stocks, currency",
        "recommendation": "recommendations for places, events or movies",
        "social": "social-media posts and complaints",
        "takeaway": "food delivery and takeaway",
        "transport": "taxis, tickets, traffic and trains",
        "weather": "weather"
      }
    },
    "intent": {
      "type": "choice",
      "instructions": "What does the user want the voice assistant to do?",
      "criteria": {
        "alarm_query": "ask about alarms that are set",
        "alarm_remove": "delete an alarm",
        "alarm_set": "set an alarm",
        "audio_volume_down": "lower the volume",
        "audio_volume_mute": "mute or silence the device",
        "audio_volume_other": "other volume request",
        "audio_volume_up": "raise the volume",
        "calendar_query": "ask about calendar events",
        "calendar_remove": "delete a calendar event",
        "calendar_set": "add an event or reminder",
        "cooking_query": "cooking question",
        "cooking_recipe": "ask for a recipe",
        "datetime_convert": "convert time or time zone",
        "datetime_query": "ask the date or time",
        "email_addcontact": "add an email contact",
        "email_query": "check or read emails",
        "email_querycontact": "ask about a contact's details",
        "email_sendemail": "send or reply to an email",
        "general_greet": "greeting",
        "general_joke": "ask for a joke",
        "general_quirky": "chit-chat or an odd request",
        "iot_cleaning": "start the robot vacuum",
        "iot_coffee": "make coffee",
        "iot_hue_lightchange": "change light colour",
        "iot_hue_lightdim": "dim the lights",
        "iot_hue_lightoff": "turn lights off",
        "iot_hue_lighton": "turn lights on",
        "iot_hue_lightup": "brighten the lights",
        "iot_wemo_off": "turn a smart plug off",
        "iot_wemo_on": "turn a smart plug on",
        "lists_createoradd": "create a list or add to it",
        "lists_query": "ask what is on a list",
        "lists_remove": "remove a list or an item",
        "music_dislikeness": "say a song or artist is disliked",
        "music_likeness": "say a song or artist is liked",
        "music_query": "ask what music is playing",
        "music_settings": "shuffle, repeat or other music settings",
        "news_query": "ask for news",
        "play_audiobook": "play an audiobook",
        "play_game": "play a game",
        "play_music": "play music",
        "play_podcasts": "play a podcast",
        "play_radio": "play the radio",
        "qa_currency": "currency exchange question",
        "qa_definition": "ask for a definition",
        "qa_factoid": "factual question",
        "qa_maths": "maths question",
        "qa_stock": "stock price question",
        "recommendation_events": "recommend events",
        "recommendation_locations": "recommend places",
        "recommendation_movies": "recommend movies",
        "social_post": "post on social media or complain to a company",
        "social_query": "ask about social-media updates",
        "takeaway_order": "order takeaway food",
        "takeaway_query": "ask about a takeaway order or restaurant",
        "transport_query": "ask about trains or transport",
        "transport_taxi": "book a taxi",
        "transport_ticket": "book a ticket",
        "transport_traffic": "ask about traffic",
        "weather_query": "ask about the weather"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "recommendation",
      "probabilities": {
        "alarm": 0,
        "audio": 0,
        "calendar": 0,
        "cooking": 0,
        "datetime": 0,
        "email": 0,
        "general": 0.03,
        "iot": 0,
        "lists": 0,
        "music": 0,
        "news": 0,
        "play": 0,
        "qa": 0.06,
        "recommendation": 0.9,
        "social": 0,
        "takeaway": 0,
        "transport": 0.01,
        "weather": 0
      },
      "confidence": 0.89
    },
    "intent": {
      "type": "choice",
      "choice": "recommendation_locations",
      "probabilities": {
        "alarm_query": 0,
        "alarm_remove": 0,
        "alarm_set": 0,
        "audio_volume_down": 0,
        "audio_volume_mute": 0,
        "audio_volume_other": 0,
        "audio_volume_up": 0,
        "calendar_query": 0,
        "calendar_remove": 0,
        "calendar_set": 0,
        "cooking_query": 0,
        "cooking_recipe": 0,
        "datetime_convert": 0,
        "datetime_query": 0,
        "email_addcontact": 0,
        "email_query": 0,
        "email_querycontact": 0,
        "email_sendemail": 0,
        "general_greet": 0,
        "general_joke": 0,
        "general_quirky": 0.12,
        "iot_cleaning": 0,
        "iot_coffee": 0,
        "iot_hue_lightchange": 0,
        "iot_hue_lightdim": 0,
        "iot_hue_lightoff": 0,
        "iot_hue_lighton": 0,
        "iot_hue_lightup": 0,
        "iot_wemo_off": 0,
        "iot_wemo_on": 0,
        "lists_createoradd": 0,
        "lists_query": 0,
        "lists_remove": 0,
        "music_dislikeness": 0,
        "music_likeness": 0,
        "music_query": 0,
        "music_settings": 0,
        "news_query": 0,
        "play_audiobook": 0,
        "play_game": 0,
        "play_music": 0,
        "play_podcasts": 0,
        "play_radio": 0,
        "qa_currency": 0,
        "qa_definition": 0,
        "qa_factoid": 0.21,
        "qa_maths": 0,
        "qa_stock": 0,
        "recommendation_events": 0.02,
        "recommendation_locations": 0.65,
        "recommendation_movies": 0,
        "social_post": 0,
        "social_query": 0,
        "takeaway_order": 0,
        "takeaway_query": 0,
        "transport_query": 0,
        "transport_taxi": 0,
        "transport_ticket": 0,
        "transport_traffic": 0,
        "weather_query": 0
      },
      "confidence": 0.64
    }
  },
  "usage": {
    "input_tokens": 1689,
    "output_tokens": 725,
    "cost": 0.000070938
  },
  "id": "gen-dec-1791517791-H0G2nO4RWfVIKj6XOY0S",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "recommendation",
      "probabilities": {
        "alarm": 0.0016,
        "audio": 0.0024,
        "calendar": 0.0029,
        "cooking": 0.0509,
        "datetime": 0.0017,
        "email": 0.0033,
        "general": 0.0152,
        "iot": 0.0022,
        "lists": 0.0025,
        "music": 0.003,
        "news": 0.0036,
        "play": 0.0059,
        "qa": 0.1069,
        "recommendation": 0.6655,
        "social": 0.0024,
        "takeaway": 0.0211,
        "transport": 0.1037,
        "weather": 0.0051
      },
      "confidence": 0.6458
    },
    "intent": {
      "type": "choice",
      "choice": "recommendation_locations",
      "probabilities": {
        "alarm_query": 0.0013,
        "alarm_remove": 0.0008,
        "alarm_set": 0.0009,
        "audio_volume_down": 0.0008,
        "audio_volume_mute": 0.0008,
        "audio_volume_other": 0.0012,
        "audio_volume_up": 0.0009,
        "calendar_query": 0.0026,
        "calendar_remove": 0.0011,
        "calendar_set": 0.0017,
        "cooking_query": 0.0127,
        "cooking_recipe": 0.0696,
        "datetime_convert": 0.0014,
        "datetime_query": 0.0011,
        "email_addcontact": 0.0009,
        "email_query": 0.0018,
        "email_querycontact": 0.0036,
        "email_sendemail": 0.0019,
        "general_greet": 0.0017,
        "general_joke": 0.0022,
        "general_quirky": 0.0305,
        "iot_cleaning": 0.0013,
        "iot_coffee": 0.0017,
        "iot_hue_lightchange": 0.0019,
        "iot_hue_lightdim": 0.001,
        "iot_hue_lightoff": 0.0007,
        "iot_hue_lighton": 0.001,
        "iot_hue_lightup": 0.0015,
        "iot_wemo_off": 0.0009,
        "iot_wemo_on": 0.0011,
        "lists_createoradd": 0.0016,
        "lists_query": 0.0023,
        "lists_remove": 0.0013,
        "music_dislikeness": 0.001,
        "music_likeness": 0.0012,
        "music_query": 0.0014,
        "music_settings": 0.0012,
        "news_query": 0.0037,
        "play_audiobook": 0.0014,
        "play_game": 0.0019,
        "play_music": 0.0034,
        "play_podcasts": 0.0023,
        "play_radio": 0.0054,
        "qa_currency": 0.0014,
        "qa_definition": 0.0068,
        "qa_factoid": 0.1044,
        "qa_maths": 0.0011,
        "qa_stock": 0.0011,
        "recommendation_events": 0.1072,
        "recommendation_locations": 0.2914,
        "recommendation_movies": 0.0317,
        "social_post": 0.0025,
        "social_query": 0.0014,
        "takeaway_order": 0.0234,
        "takeaway_query": 0.0085,
        "transport_query": 0.1852,
        "transport_taxi": 0.0089,
        "transport_ticket": 0.0157,
        "transport_traffic": 0.0221,
        "weather_query": 0.009
      },
      "confidence": 0.2794
    }
  },
  "usage": {
    "input_tokens": 1847,
    "output_tokens": 0,
    "cost": 0.000077574
  },
  "id": "gen-dec-1791517797-1Odm6WRHrEoADl7FAStM",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "cooking",
      "probabilities": {
        "alarm": 0.0006,
        "audio": 0.0008,
        "calendar": 0.0016,
        "cooking": 0.9278,
        "datetime": 0.0011,
        "email": 0.0014,
        "general": 0.0122,
        "iot": 0.0063,
        "lists": 0.0022,
        "music": 0.005,
        "news": 0.0028,
        "play": 0.0035,
        "qa": 0.0162,
        "recommendation": 0.0084,
        "social": 0.0022,
        "takeaway": 0.0022,
        "transport": 0.0014,
        "weather": 0.0041
      },
      "confidence": 0.9236
    },
    "intent": {
      "type": "choice",
      "choice": "cooking_recipe",
      "probabilities": {
        "alarm_query": 0.0003,
        "alarm_remove": 0.0002,
        "alarm_set": 0.0002,
        "audio_volume_down": 0.0002,
        "audio_volume_mute": 0.0002,
        "audio_volume_other": 0.0004,
        "audio_volume_up": 0.0003,
        "calendar_query": 0.0005,
        "calendar_remove": 0.0002,
        "calendar_set": 0.0004,
        "cooking_query": 0.0571,
        "cooking_recipe": 0.8726,
        "datetime_convert": 0.0003,
        "datetime_query": 0.0004,
        "email_addcontact": 0.0003,
        "email_query": 0.0003,
        "email_querycontact": 0.0006,
        "email_sendemail": 0.0003,
        "general_greet": 0.0008,
        "general_joke": 0.0015,
        "general_quirky": 0.0196,
        "iot_cleaning": 0.0005,
        "iot_coffee": 0.0011,
        "iot_hue_lightchange": 0.0047,
        "iot_hue_lightdim": 0.0005,
        "iot_hue_lightoff": 0.0005,
        "iot_hue_lighton": 0.0008,
        "iot_hue_lightup": 0.0017,
        "iot_wemo_off": 0.0003,
        "iot_wemo_on": 0.0003,
        "lists_createoradd": 0.0005,
        "lists_query": 0.0004,
        "lists_remove": 0.0003,
        "music_dislikeness": 0.0007,
        "music_likeness": 0.001,
        "music_query": 0.0007,
        "music_settings": 0.0013,
        "news_query": 0.0014,
        "play_audiobook": 0.0003,
        "play_game": 0.0005,
        "play_music": 0.0015,
        "play_podcasts": 0.0005,
        "play_radio": 0.0006,
        "qa_currency": 0.0009,
        "qa_definition": 0.0025,
        "qa_factoid": 0.0075,
        "qa_maths": 0.0009,
        "qa_stock": 0.0003,
        "recommendation_events": 0.0014,
        "recommendation_locations": 0.0019,
        "recommendation_movies": 0.002,
        "social_post": 0.0008,
        "social_query": 0.0003,
        "takeaway_order": 0.0005,
        "takeaway_query": 0.0006,
        "transport_query": 0.0004,
        "transport_taxi": 0.0003,
        "transport_ticket": 0.0003,
        "transport_traffic": 0.0004,
        "weather_query": 0.0018
      },
      "confidence": 0.8704
    }
  },
  "usage": {
    "input_tokens": 1847,
    "output_tokens": 0,
    "cost": 0.000038787
  },
  "id": "gen-dec-1791517802-FGZexOSRWD0NOP4HdPsq",
  "provider": "PrimeIntellect"
}
Voice-assistant intentMASSIVE bn-BD · CC-BY-4.0

ওলি হিমু মামা শোনা যাক

“Olly, let's listen to Himu Mama.”

What does the user want the voice assistant to do? · 60 options
Dataset answer: play audiobook

  • ✗Nirnoy-flashplay music79%
  • ✓Nirnoy 12Bplay audiobook80%
  • ✗Jevgeneral greet36%
  • ✗Clefplay music56%
  • ✗Clef-flashplay music52%

Himu is a character from Humayun Ahmed's novels, so this is an audiobook, not music.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "ওলি হিমু মামা শোনা যাক",
  "questions": {
    "scenario": {
      "type": "choice",
      "instructions": "Which area does this request to a voice assistant belong to?",
      "criteria": {
        "alarm": "alarms",
        "audio": "device volume",
        "calendar": "calendar events and reminders",
        "cooking": "recipes and cooking",
        "datetime": "date, time and time zones",
        "email": "email and contacts",
        "general": "greetings, jokes and chit-chat",
        "iot": "smart-home devices: lights, plugs, vacuum, coffee machine",
        "lists": "to-do and shopping lists",
        "music": "music preferences and settings",
        "news": "news",
        "play": "play music, radio, podcasts, audiobooks or games",
        "qa": "factual questions, definitions, maths, stocks, currency",
        "recommendation": "recommendations for places, events or movies",
        "social": "social-media posts and complaints",
        "takeaway": "food delivery and takeaway",
        "transport": "taxis, tickets, traffic and trains",
        "weather": "weather"
      }
    },
    "intent": {
      "type": "choice",
      "instructions": "What does the user want the voice assistant to do?",
      "criteria": {
        "alarm_query": "ask about alarms that are set",
        "alarm_remove": "delete an alarm",
        "alarm_set": "set an alarm",
        "audio_volume_down": "lower the volume",
        "audio_volume_mute": "mute or silence the device",
        "audio_volume_other": "other volume request",
        "audio_volume_up": "raise the volume",
        "calendar_query": "ask about calendar events",
        "calendar_remove": "delete a calendar event",
        "calendar_set": "add an event or reminder",
        "cooking_query": "cooking question",
        "cooking_recipe": "ask for a recipe",
        "datetime_convert": "convert time or time zone",
        "datetime_query": "ask the date or time",
        "email_addcontact": "add an email contact",
        "email_query": "check or read emails",
        "email_querycontact": "ask about a contact's details",
        "email_sendemail": "send or reply to an email",
        "general_greet": "greeting",
        "general_joke": "ask for a joke",
        "general_quirky": "chit-chat or an odd request",
        "iot_cleaning": "start the robot vacuum",
        "iot_coffee": "make coffee",
        "iot_hue_lightchange": "change light colour",
        "iot_hue_lightdim": "dim the lights",
        "iot_hue_lightoff": "turn lights off",
        "iot_hue_lighton": "turn lights on",
        "iot_hue_lightup": "brighten the lights",
        "iot_wemo_off": "turn a smart plug off",
        "iot_wemo_on": "turn a smart plug on",
        "lists_createoradd": "create a list or add to it",
        "lists_query": "ask what is on a list",
        "lists_remove": "remove a list or an item",
        "music_dislikeness": "say a song or artist is disliked",
        "music_likeness": "say a song or artist is liked",
        "music_query": "ask what music is playing",
        "music_settings": "shuffle, repeat or other music settings",
        "news_query": "ask for news",
        "play_audiobook": "play an audiobook",
        "play_game": "play a game",
        "play_music": "play music",
        "play_podcasts": "play a podcast",
        "play_radio": "play the radio",
        "qa_currency": "currency exchange question",
        "qa_definition": "ask for a definition",
        "qa_factoid": "factual question",
        "qa_maths": "maths question",
        "qa_stock": "stock price question",
        "recommendation_events": "recommend events",
        "recommendation_locations": "recommend places",
        "recommendation_movies": "recommend movies",
        "social_post": "post on social media or complain to a company",
        "social_query": "ask about social-media updates",
        "takeaway_order": "order takeaway food",
        "takeaway_query": "ask about a takeaway order or restaurant",
        "transport_query": "ask about trains or transport",
        "transport_taxi": "book a taxi",
        "transport_ticket": "book a ticket",
        "transport_traffic": "ask about traffic",
        "weather_query": "ask about the weather"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "play",
      "probabilities": {
        "alarm": 0,
        "audio": 0.13,
        "calendar": 0.01,
        "cooking": 0,
        "datetime": 0.01,
        "email": 0.04,
        "general": 0.34,
        "iot": 0.01,
        "lists": 0,
        "music": 0.04,
        "news": 0.01,
        "play": 0.35,
        "qa": 0.02,
        "recommendation": 0,
        "social": 0.04,
        "takeaway": 0,
        "transport": 0,
        "weather": 0
      },
      "confidence": 0.31
    },
    "intent": {
      "type": "choice",
      "choice": "general_greet",
      "probabilities": {
        "alarm_query": 0,
        "alarm_remove": 0,
        "alarm_set": 0,
        "audio_volume_down": 0.01,
        "audio_volume_mute": 0.04,
        "audio_volume_other": 0,
        "audio_volume_up": 0.06999999999999999,
        "calendar_query": 0,
        "calendar_remove": 0,
        "calendar_set": 0,
        "cooking_query": 0,
        "cooking_recipe": 0,
        "datetime_convert": 0,
        "datetime_query": 0,
        "email_addcontact": 0,
        "email_query": 0,
        "email_querycontact": 0.04,
        "email_sendemail": 0.01,
        "general_greet": 0.36,
        "general_joke": 0,
        "general_quirky": 0.3,
        "iot_cleaning": 0,
        "iot_coffee": 0,
        "iot_hue_lightchange": 0,
        "iot_hue_lightdim": 0,
        "iot_hue_lightoff": 0,
        "iot_hue_lighton": 0,
        "iot_hue_lightup": 0,
        "iot_wemo_off": 0,
        "iot_wemo_on": 0,
        "lists_createoradd": 0,
        "lists_query": 0,
        "lists_remove": 0,
        "music_dislikeness": 0,
        "music_likeness": 0,
        "music_query": 0.01,
        "music_settings": 0,
        "news_query": 0,
        "play_audiobook": 0,
        "play_game": 0,
        "play_music": 0.12,
        "play_podcasts": 0,
        "play_radio": 0.02,
        "qa_currency": 0,
        "qa_definition": 0,
        "qa_factoid": 0.02,
        "qa_maths": 0,
        "qa_stock": 0,
        "recommendation_events": 0,
        "recommendation_locations": 0,
        "recommendation_movies": 0,
        "social_post": 0,
        "social_query": 0,
        "takeaway_order": 0,
        "takeaway_query": 0,
        "transport_query": 0,
        "transport_taxi": 0,
        "transport_ticket": 0,
        "transport_traffic": 0,
        "weather_query": 0
      },
      "confidence": 0.35
    }
  },
  "usage": {
    "input_tokens": 1686,
    "output_tokens": 720,
    "cost": 0.000070812
  },
  "id": "gen-dec-1791517790-CEkwBKx8enFMqyQXaseY",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "play",
      "probabilities": {
        "alarm": 0.0024,
        "audio": 0.0114,
        "calendar": 0.0029,
        "cooking": 0.003,
        "datetime": 0.0013,
        "email": 0.0026,
        "general": 0.0212,
        "iot": 0.002,
        "lists": 0.0016,
        "music": 0.0045,
        "news": 0.0059,
        "play": 0.9209,
        "qa": 0.009,
        "recommendation": 0.0039,
        "social": 0.0019,
        "takeaway": 0.0013,
        "transport": 0.0022,
        "weather": 0.0021
      },
      "confidence": 0.9162
    },
    "intent": {
      "type": "choice",
      "choice": "play_music",
      "probabilities": {
        "alarm_query": 0.001,
        "alarm_remove": 0.0006,
        "alarm_set": 0.0014,
        "audio_volume_down": 0.0015,
        "audio_volume_mute": 0.0015,
        "audio_volume_other": 0.0018,
        "audio_volume_up": 0.0025,
        "calendar_query": 0.0012,
        "calendar_remove": 0.0006,
        "calendar_set": 0.0015,
        "cooking_query": 0.0007,
        "cooking_recipe": 0.0026,
        "datetime_convert": 0.0006,
        "datetime_query": 0.0008,
        "email_addcontact": 0.0005,
        "email_query": 0.0022,
        "email_querycontact": 0.0012,
        "email_sendemail": 0.0013,
        "general_greet": 0.002,
        "general_joke": 0.0336,
        "general_quirky": 0.0116,
        "iot_cleaning": 0.0008,
        "iot_coffee": 0.0008,
        "iot_hue_lightchange": 0.0005,
        "iot_hue_lightdim": 0.0006,
        "iot_hue_lightoff": 0.0006,
        "iot_hue_lighton": 0.0007,
        "iot_hue_lightup": 0.0006,
        "iot_wemo_off": 0.0007,
        "iot_wemo_on": 0.0009,
        "lists_createoradd": 0.0014,
        "lists_query": 0.0011,
        "lists_remove": 0.0009,
        "music_dislikeness": 0.0014,
        "music_likeness": 0.0015,
        "music_query": 0.0032,
        "music_settings": 0.0016,
        "news_query": 0.0037,
        "play_audiobook": 0.1249,
        "play_game": 0.0435,
        "play_music": 0.56,
        "play_podcasts": 0.0577,
        "play_radio": 0.0966,
        "qa_currency": 0.0006,
        "qa_definition": 0.0017,
        "qa_factoid": 0.0075,
        "qa_maths": 0.0012,
        "qa_stock": 0.0006,
        "recommendation_events": 0.0013,
        "recommendation_locations": 0.0009,
        "recommendation_movies": 0.0021,
        "social_post": 0.0011,
        "social_query": 0.0009,
        "takeaway_order": 0.0009,
        "takeaway_query": 0.0007,
        "transport_query": 0.0018,
        "transport_taxi": 0.0009,
        "transport_ticket": 0.0012,
        "transport_traffic": 0.0006,
        "weather_query": 0.0019
      },
      "confidence": 0.5525
    }
  },
  "usage": {
    "input_tokens": 1843,
    "output_tokens": 0,
    "cost": 0.000077406
  },
  "id": "gen-dec-1791517796-EdzcJfZfYZLxdeEJGwD2",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "play",
      "probabilities": {
        "alarm": 0.0017,
        "audio": 0.0064,
        "calendar": 0.0031,
        "cooking": 0.0024,
        "datetime": 0.0014,
        "email": 0.0038,
        "general": 0.0117,
        "iot": 0.0038,
        "lists": 0.0017,
        "music": 0.0068,
        "news": 0.0065,
        "play": 0.9317,
        "qa": 0.0047,
        "recommendation": 0.0034,
        "social": 0.003,
        "takeaway": 0.0025,
        "transport": 0.0029,
        "weather": 0.0026
      },
      "confidence": 0.9277
    },
    "intent": {
      "type": "choice",
      "choice": "play_music",
      "probabilities": {
        "alarm_query": 0.0014,
        "alarm_remove": 0.0007,
        "alarm_set": 0.0011,
        "audio_volume_down": 0.0028,
        "audio_volume_mute": 0.002,
        "audio_volume_other": 0.0029,
        "audio_volume_up": 0.0059,
        "calendar_query": 0.0021,
        "calendar_remove": 0.0008,
        "calendar_set": 0.0012,
        "cooking_query": 0.001,
        "cooking_recipe": 0.0025,
        "datetime_convert": 0.0009,
        "datetime_query": 0.0009,
        "email_addcontact": 0.001,
        "email_query": 0.0028,
        "email_querycontact": 0.0048,
        "email_sendemail": 0.0013,
        "general_greet": 0.0026,
        "general_joke": 0.0065,
        "general_quirky": 0.0063,
        "iot_cleaning": 0.0017,
        "iot_coffee": 0.0012,
        "iot_hue_lightchange": 0.0007,
        "iot_hue_lightdim": 0.0008,
        "iot_hue_lightoff": 0.0007,
        "iot_hue_lighton": 0.001,
        "iot_hue_lightup": 0.0008,
        "iot_wemo_off": 0.001,
        "iot_wemo_on": 0.0014,
        "lists_createoradd": 0.0015,
        "lists_query": 0.0014,
        "lists_remove": 0.001,
        "music_dislikeness": 0.0021,
        "music_likeness": 0.0046,
        "music_query": 0.0122,
        "music_settings": 0.0044,
        "news_query": 0.006,
        "play_audiobook": 0.0498,
        "play_game": 0.0447,
        "play_music": 0.517,
        "play_podcasts": 0.1282,
        "play_radio": 0.1419,
        "qa_currency": 0.0007,
        "qa_definition": 0.0019,
        "qa_factoid": 0.0029,
        "qa_maths": 0.001,
        "qa_stock": 0.0009,
        "recommendation_events": 0.0013,
        "recommendation_locations": 0.0011,
        "recommendation_movies": 0.0015,
        "social_post": 0.0025,
        "social_query": 0.0017,
        "takeaway_order": 0.0011,
        "takeaway_query": 0.0012,
        "transport_query": 0.0015,
        "transport_taxi": 0.0011,
        "transport_ticket": 0.001,
        "transport_traffic": 0.001,
        "weather_query": 0.0022
      },
      "confidence": 0.5088
    }
  },
  "usage": {
    "input_tokens": 1843,
    "output_tokens": 0,
    "cost": 0.000038703
  },
  "id": "gen-dec-1791517801-7zncYgPCWtWPQgGvNrhM",
  "provider": "PrimeIntellect"
}
Voice-assistant intentMASSIVE bn-BD · CC-BY-4.0

ক্লাসিক হিটগুলি তে টিউন করুন

“Tune in to classic hits.”

What does the user want the voice assistant to do? · 60 options
Dataset answer: play radio

  • ✓Nirnoy-flashplay radio86%
  • ✓Nirnoy 12Bplay radio62%
  • ✗Jevplay music62%
  • ✗Clefplay music74%
  • ✗Clef-flashplay music47%

“Tune in” points to the radio, not to playing music.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "ক্লাসিক হিটগুলি তে টিউন করুন",
  "questions": {
    "scenario": {
      "type": "choice",
      "instructions": "Which area does this request to a voice assistant belong to?",
      "criteria": {
        "alarm": "alarms",
        "audio": "device volume",
        "calendar": "calendar events and reminders",
        "cooking": "recipes and cooking",
        "datetime": "date, time and time zones",
        "email": "email and contacts",
        "general": "greetings, jokes and chit-chat",
        "iot": "smart-home devices: lights, plugs, vacuum, coffee machine",
        "lists": "to-do and shopping lists",
        "music": "music preferences and settings",
        "news": "news",
        "play": "play music, radio, podcasts, audiobooks or games",
        "qa": "factual questions, definitions, maths, stocks, currency",
        "recommendation": "recommendations for places, events or movies",
        "social": "social-media posts and complaints",
        "takeaway": "food delivery and takeaway",
        "transport": "taxis, tickets, traffic and trains",
        "weather": "weather"
      }
    },
    "intent": {
      "type": "choice",
      "instructions": "What does the user want the voice assistant to do?",
      "criteria": {
        "alarm_query": "ask about alarms that are set",
        "alarm_remove": "delete an alarm",
        "alarm_set": "set an alarm",
        "audio_volume_down": "lower the volume",
        "audio_volume_mute": "mute or silence the device",
        "audio_volume_other": "other volume request",
        "audio_volume_up": "raise the volume",
        "calendar_query": "ask about calendar events",
        "calendar_remove": "delete a calendar event",
        "calendar_set": "add an event or reminder",
        "cooking_query": "cooking question",
        "cooking_recipe": "ask for a recipe",
        "datetime_convert": "convert time or time zone",
        "datetime_query": "ask the date or time",
        "email_addcontact": "add an email contact",
        "email_query": "check or read emails",
        "email_querycontact": "ask about a contact's details",
        "email_sendemail": "send or reply to an email",
        "general_greet": "greeting",
        "general_joke": "ask for a joke",
        "general_quirky": "chit-chat or an odd request",
        "iot_cleaning": "start the robot vacuum",
        "iot_coffee": "make coffee",
        "iot_hue_lightchange": "change light colour",
        "iot_hue_lightdim": "dim the lights",
        "iot_hue_lightoff": "turn lights off",
        "iot_hue_lighton": "turn lights on",
        "iot_hue_lightup": "brighten the lights",
        "iot_wemo_off": "turn a smart plug off",
        "iot_wemo_on": "turn a smart plug on",
        "lists_createoradd": "create a list or add to it",
        "lists_query": "ask what is on a list",
        "lists_remove": "remove a list or an item",
        "music_dislikeness": "say a song or artist is disliked",
        "music_likeness": "say a song or artist is liked",
        "music_query": "ask what music is playing",
        "music_settings": "shuffle, repeat or other music settings",
        "news_query": "ask for news",
        "play_audiobook": "play an audiobook",
        "play_game": "play a game",
        "play_music": "play music",
        "play_podcasts": "play a podcast",
        "play_radio": "play the radio",
        "qa_currency": "currency exchange question",
        "qa_definition": "ask for a definition",
        "qa_factoid": "factual question",
        "qa_maths": "maths question",
        "qa_stock": "stock price question",
        "recommendation_events": "recommend events",
        "recommendation_locations": "recommend places",
        "recommendation_movies": "recommend movies",
        "social_post": "post on social media or complain to a company",
        "social_query": "ask about social-media updates",
        "takeaway_order": "order takeaway food",
        "takeaway_query": "ask about a takeaway order or restaurant",
        "transport_query": "ask about trains or transport",
        "transport_taxi": "book a taxi",
        "transport_ticket": "book a ticket",
        "transport_traffic": "ask about traffic",
        "weather_query": "ask about the weather"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "play",
      "probabilities": {
        "alarm": 0,
        "audio": 0,
        "calendar": 0,
        "cooking": 0,
        "datetime": 0,
        "email": 0,
        "general": 0,
        "iot": 0,
        "lists": 0,
        "music": 0.29,
        "news": 0,
        "play": 0.71,
        "qa": 0,
        "recommendation": 0,
        "social": 0,
        "takeaway": 0,
        "transport": 0,
        "weather": 0
      },
      "confidence": 0.69
    },
    "intent": {
      "type": "choice",
      "choice": "play_music",
      "probabilities": {
        "alarm_query": 0,
        "alarm_remove": 0,
        "alarm_set": 0,
        "audio_volume_down": 0,
        "audio_volume_mute": 0,
        "audio_volume_other": 0,
        "audio_volume_up": 0,
        "calendar_query": 0,
        "calendar_remove": 0,
        "calendar_set": 0,
        "cooking_query": 0,
        "cooking_recipe": 0,
        "datetime_convert": 0,
        "datetime_query": 0,
        "email_addcontact": 0,
        "email_query": 0,
        "email_querycontact": 0,
        "email_sendemail": 0,
        "general_greet": 0,
        "general_joke": 0,
        "general_quirky": 0,
        "iot_cleaning": 0,
        "iot_coffee": 0,
        "iot_hue_lightchange": 0,
        "iot_hue_lightdim": 0,
        "iot_hue_lightoff": 0,
        "iot_hue_lighton": 0,
        "iot_hue_lightup": 0,
        "iot_wemo_off": 0,
        "iot_wemo_on": 0,
        "lists_createoradd": 0,
        "lists_query": 0,
        "lists_remove": 0,
        "music_dislikeness": 0,
        "music_likeness": 0,
        "music_query": 0,
        "music_settings": 0.01,
        "news_query": 0,
        "play_audiobook": 0,
        "play_game": 0,
        "play_music": 0.62,
        "play_podcasts": 0,
        "play_radio": 0.37,
        "qa_currency": 0,
        "qa_definition": 0,
        "qa_factoid": 0,
        "qa_maths": 0,
        "qa_stock": 0,
        "recommendation_events": 0,
        "recommendation_locations": 0,
        "recommendation_movies": 0,
        "social_post": 0,
        "social_query": 0,
        "takeaway_order": 0,
        "takeaway_query": 0,
        "transport_query": 0,
        "transport_taxi": 0,
        "transport_ticket": 0,
        "transport_traffic": 0,
        "weather_query": 0
      },
      "confidence": 0.61
    }
  },
  "usage": {
    "input_tokens": 1692,
    "output_tokens": 719,
    "cost": 0.000071064
  },
  "id": "gen-dec-1791517792-cYDkoWH7UkgNTcNEqpFF",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "play",
      "probabilities": {
        "alarm": 0.0008,
        "audio": 0.0014,
        "calendar": 0.0008,
        "cooking": 0.0008,
        "datetime": 0.0009,
        "email": 0.0006,
        "general": 0.0008,
        "iot": 0.0009,
        "lists": 0.0009,
        "music": 0.009,
        "news": 0.0009,
        "play": 0.9769,
        "qa": 0.0008,
        "recommendation": 0.0012,
        "social": 0.0008,
        "takeaway": 0.0008,
        "transport": 0.001,
        "weather": 0.0007
      },
      "confidence": 0.9755
    },
    "intent": {
      "type": "choice",
      "choice": "play_music",
      "probabilities": {
        "alarm_query": 0.0006,
        "alarm_remove": 0.0005,
        "alarm_set": 0.0007,
        "audio_volume_down": 0.0006,
        "audio_volume_mute": 0.0005,
        "audio_volume_other": 0.0007,
        "audio_volume_up": 0.0008,
        "calendar_query": 0.0005,
        "calendar_remove": 0.0004,
        "calendar_set": 0.0006,
        "cooking_query": 0.0005,
        "cooking_recipe": 0.0006,
        "datetime_convert": 0.0005,
        "datetime_query": 0.0006,
        "email_addcontact": 0.0004,
        "email_query": 0.0005,
        "email_querycontact": 0.0004,
        "email_sendemail": 0.0005,
        "general_greet": 0.0005,
        "general_joke": 0.0005,
        "general_quirky": 0.0008,
        "iot_cleaning": 0.0006,
        "iot_coffee": 0.0005,
        "iot_hue_lightchange": 0.0005,
        "iot_hue_lightdim": 0.0005,
        "iot_hue_lightoff": 0.0004,
        "iot_hue_lighton": 0.0005,
        "iot_hue_lightup": 0.0004,
        "iot_wemo_off": 0.0006,
        "iot_wemo_on": 0.0007,
        "lists_createoradd": 0.0007,
        "lists_query": 0.0006,
        "lists_remove": 0.0006,
        "music_dislikeness": 0.0009,
        "music_likeness": 0.0012,
        "music_query": 0.0018,
        "music_settings": 0.0047,
        "news_query": 0.0008,
        "play_audiobook": 0.0239,
        "play_game": 0.0099,
        "play_music": 0.7365,
        "play_podcasts": 0.0316,
        "play_radio": 0.1605,
        "qa_currency": 0.0005,
        "qa_definition": 0.0006,
        "qa_factoid": 0.0008,
        "qa_maths": 0.0004,
        "qa_stock": 0.0005,
        "recommendation_events": 0.0005,
        "recommendation_locations": 0.0005,
        "recommendation_movies": 0.0007,
        "social_post": 0.0007,
        "social_query": 0.0005,
        "takeaway_order": 0.0004,
        "takeaway_query": 0.0005,
        "transport_query": 0.0008,
        "transport_taxi": 0.0006,
        "transport_ticket": 0.0007,
        "transport_traffic": 0.0005,
        "weather_query": 0.0005
      },
      "confidence": 0.732
    }
  },
  "usage": {
    "input_tokens": 1847,
    "output_tokens": 0,
    "cost": 0.000077574
  },
  "id": "gen-dec-1791517798-1DS3tvqw3Y3rHeErkUPe",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "play",
      "probabilities": {
        "alarm": 0.0007,
        "audio": 0.0007,
        "calendar": 0.0007,
        "cooking": 0.0009,
        "datetime": 0.0012,
        "email": 0.0007,
        "general": 0.0009,
        "iot": 0.0012,
        "lists": 0.0008,
        "music": 0.0021,
        "news": 0.001,
        "play": 0.9836,
        "qa": 0.001,
        "recommendation": 0.001,
        "social": 0.0007,
        "takeaway": 0.0009,
        "transport": 0.0012,
        "weather": 0.0007
      },
      "confidence": 0.9826
    },
    "intent": {
      "type": "choice",
      "choice": "play_music",
      "probabilities": {
        "alarm_query": 0.0008,
        "alarm_remove": 0.0009,
        "alarm_set": 0.0007,
        "audio_volume_down": 0.0007,
        "audio_volume_mute": 0.0009,
        "audio_volume_other": 0.0007,
        "audio_volume_up": 0.0008,
        "calendar_query": 0.0007,
        "calendar_remove": 0.0009,
        "calendar_set": 0.0008,
        "cooking_query": 0.0008,
        "cooking_recipe": 0.0008,
        "datetime_convert": 0.0011,
        "datetime_query": 0.0011,
        "email_addcontact": 0.0008,
        "email_query": 0.001,
        "email_querycontact": 0.0008,
        "email_sendemail": 0.0011,
        "general_greet": 0.0006,
        "general_joke": 0.0008,
        "general_quirky": 0.0007,
        "iot_cleaning": 0.0009,
        "iot_coffee": 0.0006,
        "iot_hue_lightchange": 0.0008,
        "iot_hue_lightdim": 0.0009,
        "iot_hue_lightoff": 0.0009,
        "iot_hue_lighton": 0.0009,
        "iot_hue_lightup": 0.0009,
        "iot_wemo_off": 0.0011,
        "iot_wemo_on": 0.0011,
        "lists_createoradd": 0.0013,
        "lists_query": 0.0009,
        "lists_remove": 0.0009,
        "music_dislikeness": 0.0006,
        "music_likeness": 0.0009,
        "music_query": 0.0014,
        "music_settings": 0.0023,
        "news_query": 0.0009,
        "play_audiobook": 0.0051,
        "play_game": 0.0072,
        "play_music": 0.4689,
        "play_podcasts": 0.0134,
        "play_radio": 0.458,
        "qa_currency": 0.0006,
        "qa_definition": 0.0007,
        "qa_factoid": 0.0006,
        "qa_maths": 0.0006,
        "qa_stock": 0.0006,
        "recommendation_events": 0.0006,
        "recommendation_locations": 0.0006,
        "recommendation_movies": 0.0007,
        "social_post": 0.0011,
        "social_query": 0.0007,
        "takeaway_order": 0.0007,
        "takeaway_query": 0.0008,
        "transport_query": 0.0009,
        "transport_taxi": 0.0008,
        "transport_ticket": 0.0008,
        "transport_traffic": 0.0008,
        "weather_query": 0.0008
      },
      "confidence": 0.4599
    }
  },
  "usage": {
    "input_tokens": 1847,
    "output_tokens": 0,
    "cost": 0.000038787
  },
  "id": "gen-dec-1791517803-MeRzduUFkbda9Wxk6Fu9",
  "provider": "PrimeIntellect"
}
Voice-assistant intentMASSIVE bn-BD · CC-BY-4.0

ঠিক আছে গুগল একজন রাষ্ট্রপতি কি

“OK Google, what is a president?”

What does the user want the voice assistant to do? · 60 options
Dataset answer: qa definition

  • ✗Nirnoy-flashqa factoid94%
  • ✓Nirnoy 12Bqa definition95%
  • ✗Jevqa factoid64%
  • ✗Clefqa factoid74%
  • ✗Clef-flashqa factoid87%

Asking what something is: a definition, not a fact lookup.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "ঠিক আছে গুগল একজন রাষ্ট্রপতি কি",
  "questions": {
    "scenario": {
      "type": "choice",
      "instructions": "Which area does this request to a voice assistant belong to?",
      "criteria": {
        "alarm": "alarms",
        "audio": "device volume",
        "calendar": "calendar events and reminders",
        "cooking": "recipes and cooking",
        "datetime": "date, time and time zones",
        "email": "email and contacts",
        "general": "greetings, jokes and chit-chat",
        "iot": "smart-home devices: lights, plugs, vacuum, coffee machine",
        "lists": "to-do and shopping lists",
        "music": "music preferences and settings",
        "news": "news",
        "play": "play music, radio, podcasts, audiobooks or games",
        "qa": "factual questions, definitions, maths, stocks, currency",
        "recommendation": "recommendations for places, events or movies",
        "social": "social-media posts and complaints",
        "takeaway": "food delivery and takeaway",
        "transport": "taxis, tickets, traffic and trains",
        "weather": "weather"
      }
    },
    "intent": {
      "type": "choice",
      "instructions": "What does the user want the voice assistant to do?",
      "criteria": {
        "alarm_query": "ask about alarms that are set",
        "alarm_remove": "delete an alarm",
        "alarm_set": "set an alarm",
        "audio_volume_down": "lower the volume",
        "audio_volume_mute": "mute or silence the device",
        "audio_volume_other": "other volume request",
        "audio_volume_up": "raise the volume",
        "calendar_query": "ask about calendar events",
        "calendar_remove": "delete a calendar event",
        "calendar_set": "add an event or reminder",
        "cooking_query": "cooking question",
        "cooking_recipe": "ask for a recipe",
        "datetime_convert": "convert time or time zone",
        "datetime_query": "ask the date or time",
        "email_addcontact": "add an email contact",
        "email_query": "check or read emails",
        "email_querycontact": "ask about a contact's details",
        "email_sendemail": "send or reply to an email",
        "general_greet": "greeting",
        "general_joke": "ask for a joke",
        "general_quirky": "chit-chat or an odd request",
        "iot_cleaning": "start the robot vacuum",
        "iot_coffee": "make coffee",
        "iot_hue_lightchange": "change light colour",
        "iot_hue_lightdim": "dim the lights",
        "iot_hue_lightoff": "turn lights off",
        "iot_hue_lighton": "turn lights on",
        "iot_hue_lightup": "brighten the lights",
        "iot_wemo_off": "turn a smart plug off",
        "iot_wemo_on": "turn a smart plug on",
        "lists_createoradd": "create a list or add to it",
        "lists_query": "ask what is on a list",
        "lists_remove": "remove a list or an item",
        "music_dislikeness": "say a song or artist is disliked",
        "music_likeness": "say a song or artist is liked",
        "music_query": "ask what music is playing",
        "music_settings": "shuffle, repeat or other music settings",
        "news_query": "ask for news",
        "play_audiobook": "play an audiobook",
        "play_game": "play a game",
        "play_music": "play music",
        "play_podcasts": "play a podcast",
        "play_radio": "play the radio",
        "qa_currency": "currency exchange question",
        "qa_definition": "ask for a definition",
        "qa_factoid": "factual question",
        "qa_maths": "maths question",
        "qa_stock": "stock price question",
        "recommendation_events": "recommend events",
        "recommendation_locations": "recommend places",
        "recommendation_movies": "recommend movies",
        "social_post": "post on social media or complain to a company",
        "social_query": "ask about social-media updates",
        "takeaway_order": "order takeaway food",
        "takeaway_query": "ask about a takeaway order or restaurant",
        "transport_query": "ask about trains or transport",
        "transport_taxi": "book a taxi",
        "transport_ticket": "book a ticket",
        "transport_traffic": "ask about traffic",
        "weather_query": "ask about the weather"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "qa",
      "probabilities": {
        "alarm": 0,
        "audio": 0,
        "calendar": 0,
        "cooking": 0,
        "datetime": 0,
        "email": 0,
        "general": 0.02,
        "iot": 0,
        "lists": 0,
        "music": 0,
        "news": 0,
        "play": 0,
        "qa": 0.98,
        "recommendation": 0,
        "social": 0,
        "takeaway": 0,
        "transport": 0,
        "weather": 0
      },
      "confidence": 0.98
    },
    "intent": {
      "type": "choice",
      "choice": "qa_factoid",
      "probabilities": {
        "alarm_query": 0,
        "alarm_remove": 0,
        "alarm_set": 0,
        "audio_volume_down": 0,
        "audio_volume_mute": 0,
        "audio_volume_other": 0,
        "audio_volume_up": 0,
        "calendar_query": 0,
        "calendar_remove": 0,
        "calendar_set": 0,
        "cooking_query": 0,
        "cooking_recipe": 0,
        "datetime_convert": 0,
        "datetime_query": 0,
        "email_addcontact": 0,
        "email_query": 0,
        "email_querycontact": 0,
        "email_sendemail": 0,
        "general_greet": 0,
        "general_joke": 0,
        "general_quirky": 0.17,
        "iot_cleaning": 0,
        "iot_coffee": 0,
        "iot_hue_lightchange": 0,
        "iot_hue_lightdim": 0,
        "iot_hue_lightoff": 0,
        "iot_hue_lighton": 0,
        "iot_hue_lightup": 0,
        "iot_wemo_off": 0,
        "iot_wemo_on": 0,
        "lists_createoradd": 0,
        "lists_query": 0,
        "lists_remove": 0,
        "music_dislikeness": 0,
        "music_likeness": 0,
        "music_query": 0,
        "music_settings": 0,
        "news_query": 0,
        "play_audiobook": 0,
        "play_game": 0,
        "play_music": 0,
        "play_podcasts": 0,
        "play_radio": 0,
        "qa_currency": 0,
        "qa_definition": 0.19,
        "qa_factoid": 0.64,
        "qa_maths": 0,
        "qa_stock": 0,
        "recommendation_events": 0,
        "recommendation_locations": 0,
        "recommendation_movies": 0,
        "social_post": 0,
        "social_query": 0,
        "takeaway_order": 0,
        "takeaway_query": 0,
        "transport_query": 0,
        "transport_taxi": 0,
        "transport_ticket": 0,
        "transport_traffic": 0,
        "weather_query": 0
      },
      "confidence": 0.63
    }
  },
  "usage": {
    "input_tokens": 1691,
    "output_tokens": 720,
    "cost": 0.000071022
  },
  "id": "gen-dec-1791517791-MO8rdYFfAC8hqaGOTyWp",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "qa",
      "probabilities": {
        "alarm": 0.003,
        "audio": 0.0034,
        "calendar": 0.0038,
        "cooking": 0.0035,
        "datetime": 0.0042,
        "email": 0.0069,
        "general": 0.051,
        "iot": 0.0033,
        "lists": 0.003,
        "music": 0.0031,
        "news": 0.067,
        "play": 0.0043,
        "qa": 0.8227,
        "recommendation": 0.0035,
        "social": 0.0054,
        "takeaway": 0.0028,
        "transport": 0.0035,
        "weather": 0.0059
      },
      "confidence": 0.8123
    },
    "intent": {
      "type": "choice",
      "choice": "qa_factoid",
      "probabilities": {
        "alarm_query": 0.0022,
        "alarm_remove": 0.0011,
        "alarm_set": 0.0012,
        "audio_volume_down": 0.0013,
        "audio_volume_mute": 0.0013,
        "audio_volume_other": 0.0013,
        "audio_volume_up": 0.0012,
        "calendar_query": 0.0026,
        "calendar_remove": 0.0013,
        "calendar_set": 0.0016,
        "cooking_query": 0.0021,
        "cooking_recipe": 0.0025,
        "datetime_convert": 0.0013,
        "datetime_query": 0.0023,
        "email_addcontact": 0.0012,
        "email_query": 0.0035,
        "email_querycontact": 0.0036,
        "email_sendemail": 0.002,
        "general_greet": 0.0133,
        "general_joke": 0.0085,
        "general_quirky": 0.0942,
        "iot_cleaning": 0.0016,
        "iot_coffee": 0.0013,
        "iot_hue_lightchange": 0.0011,
        "iot_hue_lightdim": 0.0013,
        "iot_hue_lightoff": 0.001,
        "iot_hue_lighton": 0.0011,
        "iot_hue_lightup": 0.0012,
        "iot_wemo_off": 0.0013,
        "iot_wemo_on": 0.0014,
        "lists_createoradd": 0.0015,
        "lists_query": 0.0023,
        "lists_remove": 0.0014,
        "music_dislikeness": 0.0016,
        "music_likeness": 0.0016,
        "music_query": 0.0018,
        "music_settings": 0.0012,
        "news_query": 0.0277,
        "play_audiobook": 0.0017,
        "play_game": 0.0015,
        "play_music": 0.0015,
        "play_podcasts": 0.0023,
        "play_radio": 0.0027,
        "qa_currency": 0.0039,
        "qa_definition": 0.0191,
        "qa_factoid": 0.7382,
        "qa_maths": 0.0041,
        "qa_stock": 0.004,
        "recommendation_events": 0.0013,
        "recommendation_locations": 0.0012,
        "recommendation_movies": 0.0013,
        "social_post": 0.0029,
        "social_query": 0.0025,
        "takeaway_order": 0.0011,
        "takeaway_query": 0.0015,
        "transport_query": 0.0018,
        "transport_taxi": 0.0014,
        "transport_ticket": 0.0012,
        "transport_traffic": 0.0016,
        "weather_query": 0.0033
      },
      "confidence": 0.7338
    }
  },
  "usage": {
    "input_tokens": 1850,
    "output_tokens": 0,
    "cost": 0.0000777
  },
  "id": "gen-dec-1791517796-cdnsJSev1K8gLmM8bEfX",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "qa",
      "probabilities": {
        "alarm": 0.0054,
        "audio": 0.0053,
        "calendar": 0.0082,
        "cooking": 0.0062,
        "datetime": 0.006,
        "email": 0.0162,
        "general": 0.0251,
        "iot": 0.0068,
        "lists": 0.0053,
        "music": 0.0047,
        "news": 0.0215,
        "play": 0.0064,
        "qa": 0.8458,
        "recommendation": 0.0077,
        "social": 0.009,
        "takeaway": 0.0062,
        "transport": 0.0064,
        "weather": 0.0082
      },
      "confidence": 0.8367
    },
    "intent": {
      "type": "choice",
      "choice": "qa_factoid",
      "probabilities": {
        "alarm_query": 0.0014,
        "alarm_remove": 0.0013,
        "alarm_set": 0.0013,
        "audio_volume_down": 0.0015,
        "audio_volume_mute": 0.0015,
        "audio_volume_other": 0.0012,
        "audio_volume_up": 0.0015,
        "calendar_query": 0.0017,
        "calendar_remove": 0.0013,
        "calendar_set": 0.0014,
        "cooking_query": 0.0021,
        "cooking_recipe": 0.0018,
        "datetime_convert": 0.0013,
        "datetime_query": 0.0016,
        "email_addcontact": 0.0015,
        "email_query": 0.002,
        "email_querycontact": 0.0051,
        "email_sendemail": 0.0014,
        "general_greet": 0.0022,
        "general_joke": 0.0017,
        "general_quirky": 0.0139,
        "iot_cleaning": 0.0015,
        "iot_coffee": 0.0014,
        "iot_hue_lightchange": 0.0013,
        "iot_hue_lightdim": 0.0016,
        "iot_hue_lightoff": 0.0013,
        "iot_hue_lighton": 0.0014,
        "iot_hue_lightup": 0.0016,
        "iot_wemo_off": 0.0013,
        "iot_wemo_on": 0.0013,
        "lists_createoradd": 0.0016,
        "lists_query": 0.0017,
        "lists_remove": 0.0015,
        "music_dislikeness": 0.0012,
        "music_likeness": 0.0013,
        "music_query": 0.0015,
        "music_settings": 0.0012,
        "news_query": 0.0058,
        "play_audiobook": 0.0013,
        "play_game": 0.0013,
        "play_music": 0.0012,
        "play_podcasts": 0.0015,
        "play_radio": 0.0013,
        "qa_currency": 0.0039,
        "qa_definition": 0.0117,
        "qa_factoid": 0.8717,
        "qa_maths": 0.0059,
        "qa_stock": 0.0056,
        "recommendation_events": 0.0014,
        "recommendation_locations": 0.0015,
        "recommendation_movies": 0.0013,
        "social_post": 0.0024,
        "social_query": 0.0015,
        "takeaway_order": 0.0013,
        "takeaway_query": 0.0016,
        "transport_query": 0.0013,
        "transport_taxi": 0.0015,
        "transport_ticket": 0.0013,
        "transport_traffic": 0.0013,
        "weather_query": 0.0018
      },
      "confidence": 0.8695
    }
  },
  "usage": {
    "input_tokens": 1850,
    "output_tokens": 0,
    "cost": 0.00003885
  },
  "id": "gen-dec-1791517801-JQLhNybv4VrrO0hJh32m",
  "provider": "PrimeIntellect"
}
Voice-assistant intentMASSIVE bn-BD · CC-BY-4.0

লাইব্রেরিতে পরবর্তী ঘটনা কি

“What's the next event at the library?”

What does the user want the voice assistant to do? · 60 options
Dataset answer: recommendation events

  • ✗Nirnoy-flashcalendar query88%
  • ✓Nirnoy 12Brecommendation events87%
  • ✗Jevcalendar query38%
  • ✗Clefcalendar query80%
  • ✗Clef-flashcalendar query79%

The user wants event suggestions, not their own calendar.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "লাইব্রেরিতে পরবর্তী ঘটনা কি",
  "questions": {
    "scenario": {
      "type": "choice",
      "instructions": "Which area does this request to a voice assistant belong to?",
      "criteria": {
        "alarm": "alarms",
        "audio": "device volume",
        "calendar": "calendar events and reminders",
        "cooking": "recipes and cooking",
        "datetime": "date, time and time zones",
        "email": "email and contacts",
        "general": "greetings, jokes and chit-chat",
        "iot": "smart-home devices: lights, plugs, vacuum, coffee machine",
        "lists": "to-do and shopping lists",
        "music": "music preferences and settings",
        "news": "news",
        "play": "play music, radio, podcasts, audiobooks or games",
        "qa": "factual questions, definitions, maths, stocks, currency",
        "recommendation": "recommendations for places, events or movies",
        "social": "social-media posts and complaints",
        "takeaway": "food delivery and takeaway",
        "transport": "taxis, tickets, traffic and trains",
        "weather": "weather"
      }
    },
    "intent": {
      "type": "choice",
      "instructions": "What does the user want the voice assistant to do?",
      "criteria": {
        "alarm_query": "ask about alarms that are set",
        "alarm_remove": "delete an alarm",
        "alarm_set": "set an alarm",
        "audio_volume_down": "lower the volume",
        "audio_volume_mute": "mute or silence the device",
        "audio_volume_other": "other volume request",
        "audio_volume_up": "raise the volume",
        "calendar_query": "ask about calendar events",
        "calendar_remove": "delete a calendar event",
        "calendar_set": "add an event or reminder",
        "cooking_query": "cooking question",
        "cooking_recipe": "ask for a recipe",
        "datetime_convert": "convert time or time zone",
        "datetime_query": "ask the date or time",
        "email_addcontact": "add an email contact",
        "email_query": "check or read emails",
        "email_querycontact": "ask about a contact's details",
        "email_sendemail": "send or reply to an email",
        "general_greet": "greeting",
        "general_joke": "ask for a joke",
        "general_quirky": "chit-chat or an odd request",
        "iot_cleaning": "start the robot vacuum",
        "iot_coffee": "make coffee",
        "iot_hue_lightchange": "change light colour",
        "iot_hue_lightdim": "dim the lights",
        "iot_hue_lightoff": "turn lights off",
        "iot_hue_lighton": "turn lights on",
        "iot_hue_lightup": "brighten the lights",
        "iot_wemo_off": "turn a smart plug off",
        "iot_wemo_on": "turn a smart plug on",
        "lists_createoradd": "create a list or add to it",
        "lists_query": "ask what is on a list",
        "lists_remove": "remove a list or an item",
        "music_dislikeness": "say a song or artist is disliked",
        "music_likeness": "say a song or artist is liked",
        "music_query": "ask what music is playing",
        "music_settings": "shuffle, repeat or other music settings",
        "news_query": "ask for news",
        "play_audiobook": "play an audiobook",
        "play_game": "play a game",
        "play_music": "play music",
        "play_podcasts": "play a podcast",
        "play_radio": "play the radio",
        "qa_currency": "currency exchange question",
        "qa_definition": "ask for a definition",
        "qa_factoid": "factual question",
        "qa_maths": "maths question",
        "qa_stock": "stock price question",
        "recommendation_events": "recommend events",
        "recommendation_locations": "recommend places",
        "recommendation_movies": "recommend movies",
        "social_post": "post on social media or complain to a company",
        "social_query": "ask about social-media updates",
        "takeaway_order": "order takeaway food",
        "takeaway_query": "ask about a takeaway order or restaurant",
        "transport_query": "ask about trains or transport",
        "transport_taxi": "book a taxi",
        "transport_ticket": "book a ticket",
        "transport_traffic": "ask about traffic",
        "weather_query": "ask about the weather"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "qa",
      "probabilities": {
        "alarm": 0,
        "audio": 0,
        "calendar": 0.19,
        "cooking": 0,
        "datetime": 0.02,
        "email": 0,
        "general": 0.11,
        "iot": 0.01,
        "lists": 0.01,
        "music": 0,
        "news": 0.05,
        "play": 0.07,
        "qa": 0.45,
        "recommendation": 0.09,
        "social": 0,
        "takeaway": 0,
        "transport": 0,
        "weather": 0
      },
      "confidence": 0.42
    },
    "intent": {
      "type": "choice",
      "choice": "calendar_query",
      "probabilities": {
        "alarm_query": 0,
        "alarm_remove": 0,
        "alarm_set": 0,
        "audio_volume_down": 0,
        "audio_volume_mute": 0,
        "audio_volume_other": 0,
        "audio_volume_up": 0,
        "calendar_query": 0.38,
        "calendar_remove": 0,
        "calendar_set": 0,
        "cooking_query": 0,
        "cooking_recipe": 0,
        "datetime_convert": 0,
        "datetime_query": 0.01,
        "email_addcontact": 0,
        "email_query": 0,
        "email_querycontact": 0,
        "email_sendemail": 0,
        "general_greet": 0,
        "general_joke": 0,
        "general_quirky": 0.24,
        "iot_cleaning": 0,
        "iot_coffee": 0,
        "iot_hue_lightchange": 0,
        "iot_hue_lightdim": 0,
        "iot_hue_lightoff": 0,
        "iot_hue_lighton": 0,
        "iot_hue_lightup": 0,
        "iot_wemo_off": 0,
        "iot_wemo_on": 0,
        "lists_createoradd": 0,
        "lists_query": 0.04,
        "lists_remove": 0,
        "music_dislikeness": 0,
        "music_likeness": 0,
        "music_query": 0.01,
        "music_settings": 0,
        "news_query": 0.02,
        "play_audiobook": 0.01,
        "play_game": 0,
        "play_music": 0,
        "play_podcasts": 0,
        "play_radio": 0,
        "qa_currency": 0,
        "qa_definition": 0,
        "qa_factoid": 0.21,
        "qa_maths": 0,
        "qa_stock": 0,
        "recommendation_events": 0.060000000000000005,
        "recommendation_locations": 0.01,
        "recommendation_movies": 0,
        "social_post": 0,
        "social_query": 0,
        "takeaway_order": 0,
        "takeaway_query": 0,
        "transport_query": 0,
        "transport_taxi": 0,
        "transport_ticket": 0,
        "transport_traffic": 0,
        "weather_query": 0
      },
      "confidence": 0.37
    }
  },
  "usage": {
    "input_tokens": 1691,
    "output_tokens": 719,
    "cost": 0.000071022
  },
  "id": "gen-dec-1791517791-q9rMp8wMDMu4ETTJO9aR",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "calendar",
      "probabilities": {
        "alarm": 0.0015,
        "audio": 0.0015,
        "calendar": 0.8589,
        "cooking": 0.0017,
        "datetime": 0.0016,
        "email": 0.003,
        "general": 0.0121,
        "iot": 0.0015,
        "lists": 0.0061,
        "music": 0.0019,
        "news": 0.0196,
        "play": 0.007,
        "qa": 0.0112,
        "recommendation": 0.0637,
        "social": 0.0025,
        "takeaway": 0.001,
        "transport": 0.0029,
        "weather": 0.0023
      },
      "confidence": 0.8506
    },
    "intent": {
      "type": "choice",
      "choice": "calendar_query",
      "probabilities": {
        "alarm_query": 0.0014,
        "alarm_remove": 0.0005,
        "alarm_set": 0.0006,
        "audio_volume_down": 0.0005,
        "audio_volume_mute": 0.0006,
        "audio_volume_other": 0.0006,
        "audio_volume_up": 0.0005,
        "calendar_query": 0.7994,
        "calendar_remove": 0.0203,
        "calendar_set": 0.009,
        "cooking_query": 0.0007,
        "cooking_recipe": 0.0014,
        "datetime_convert": 0.0006,
        "datetime_query": 0.0008,
        "email_addcontact": 0.0005,
        "email_query": 0.0025,
        "email_querycontact": 0.0013,
        "email_sendemail": 0.0007,
        "general_greet": 0.0008,
        "general_joke": 0.0011,
        "general_quirky": 0.0169,
        "iot_cleaning": 0.0006,
        "iot_coffee": 0.0006,
        "iot_hue_lightchange": 0.0004,
        "iot_hue_lightdim": 0.0005,
        "iot_hue_lightoff": 0.0004,
        "iot_hue_lighton": 0.0005,
        "iot_hue_lightup": 0.0005,
        "iot_wemo_off": 0.0005,
        "iot_wemo_on": 0.0006,
        "lists_createoradd": 0.0013,
        "lists_query": 0.0059,
        "lists_remove": 0.0008,
        "music_dislikeness": 0.0005,
        "music_likeness": 0.0005,
        "music_query": 0.0022,
        "music_settings": 0.0008,
        "news_query": 0.0156,
        "play_audiobook": 0.0151,
        "play_game": 0.0009,
        "play_music": 0.0013,
        "play_podcasts": 0.0067,
        "play_radio": 0.0023,
        "qa_currency": 0.0005,
        "qa_definition": 0.0031,
        "qa_factoid": 0.0118,
        "qa_maths": 0.0005,
        "qa_stock": 0.0006,
        "recommendation_events": 0.0305,
        "recommendation_locations": 0.0068,
        "recommendation_movies": 0.0151,
        "social_post": 0.0017,
        "social_query": 0.0031,
        "takeaway_order": 0.0004,
        "takeaway_query": 0.0008,
        "transport_query": 0.0015,
        "transport_taxi": 0.0007,
        "transport_ticket": 0.0012,
        "transport_traffic": 0.0008,
        "weather_query": 0.002
      },
      "confidence": 0.796
    }
  },
  "usage": {
    "input_tokens": 1846,
    "output_tokens": 0,
    "cost": 0.000077532
  },
  "id": "gen-dec-1791517797-xY6Veqv5TAnImzMXMQx0",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "calendar",
      "probabilities": {
        "alarm": 0.0044,
        "audio": 0.0031,
        "calendar": 0.7566,
        "cooking": 0.0046,
        "datetime": 0.0079,
        "email": 0.0091,
        "general": 0.0184,
        "iot": 0.0063,
        "lists": 0.0111,
        "music": 0.0063,
        "news": 0.0383,
        "play": 0.0484,
        "qa": 0.0307,
        "recommendation": 0.0239,
        "social": 0.0079,
        "takeaway": 0.0035,
        "transport": 0.0089,
        "weather": 0.0105
      },
      "confidence": 0.7423
    },
    "intent": {
      "type": "choice",
      "choice": "calendar_query",
      "probabilities": {
        "alarm_query": 0.0015,
        "alarm_remove": 0.0008,
        "alarm_set": 0.0009,
        "audio_volume_down": 0.001,
        "audio_volume_mute": 0.0011,
        "audio_volume_other": 0.0009,
        "audio_volume_up": 0.001,
        "calendar_query": 0.7908,
        "calendar_remove": 0.0162,
        "calendar_set": 0.0132,
        "cooking_query": 0.0016,
        "cooking_recipe": 0.002,
        "datetime_convert": 0.0012,
        "datetime_query": 0.0025,
        "email_addcontact": 0.001,
        "email_query": 0.0022,
        "email_querycontact": 0.002,
        "email_sendemail": 0.001,
        "general_greet": 0.0018,
        "general_joke": 0.0016,
        "general_quirky": 0.0113,
        "iot_cleaning": 0.0011,
        "iot_coffee": 0.001,
        "iot_hue_lightchange": 0.0008,
        "iot_hue_lightdim": 0.0008,
        "iot_hue_lightoff": 0.0008,
        "iot_hue_lighton": 0.0009,
        "iot_hue_lightup": 0.0009,
        "iot_wemo_off": 0.0008,
        "iot_wemo_on": 0.0009,
        "lists_createoradd": 0.0018,
        "lists_query": 0.004,
        "lists_remove": 0.0014,
        "music_dislikeness": 0.0013,
        "music_likeness": 0.0012,
        "music_query": 0.0016,
        "music_settings": 0.001,
        "news_query": 0.0157,
        "play_audiobook": 0.0316,
        "play_game": 0.0028,
        "play_music": 0.0026,
        "play_podcasts": 0.0104,
        "play_radio": 0.0027,
        "qa_currency": 0.0012,
        "qa_definition": 0.0041,
        "qa_factoid": 0.0126,
        "qa_maths": 0.0013,
        "qa_stock": 0.0014,
        "recommendation_events": 0.0128,
        "recommendation_locations": 0.0033,
        "recommendation_movies": 0.0031,
        "social_post": 0.0022,
        "social_query": 0.0025,
        "takeaway_order": 0.001,
        "takeaway_query": 0.001,
        "transport_query": 0.0038,
        "transport_taxi": 0.0013,
        "transport_ticket": 0.0021,
        "transport_traffic": 0.0013,
        "weather_query": 0.0031
      },
      "confidence": 0.7873
    }
  },
  "usage": {
    "input_tokens": 1846,
    "output_tokens": 0,
    "cost": 0.000038766
  },
  "id": "gen-dec-1791517802-EGQ5IzyBbDUz5V0ezVV6",
  "provider": "PrimeIntellect"
}
Voice-assistant intentMASSIVE bn-BD · CC-BY-4.0

আমরা কি সূর্যে যেতে পারি

“Can we go to the sun?”

What does the user want the voice assistant to do? · 60 options
Dataset answer: general quirky

  • ✓Nirnoy-flashgeneral quirky70%
  • ✓Nirnoy 12Bgeneral quirky60%
  • ✗Jevqa factoid85%
  • ✗Clefqa factoid67%
  • ✗Clef-flashweather query82%

A playful question, not a fact lookup or the weather.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "আমরা কি সূর্যে যেতে পারি",
  "questions": {
    "scenario": {
      "type": "choice",
      "instructions": "Which area does this request to a voice assistant belong to?",
      "criteria": {
        "alarm": "alarms",
        "audio": "device volume",
        "calendar": "calendar events and reminders",
        "cooking": "recipes and cooking",
        "datetime": "date, time and time zones",
        "email": "email and contacts",
        "general": "greetings, jokes and chit-chat",
        "iot": "smart-home devices: lights, plugs, vacuum, coffee machine",
        "lists": "to-do and shopping lists",
        "music": "music preferences and settings",
        "news": "news",
        "play": "play music, radio, podcasts, audiobooks or games",
        "qa": "factual questions, definitions, maths, stocks, currency",
        "recommendation": "recommendations for places, events or movies",
        "social": "social-media posts and complaints",
        "takeaway": "food delivery and takeaway",
        "transport": "taxis, tickets, traffic and trains",
        "weather": "weather"
      }
    },
    "intent": {
      "type": "choice",
      "instructions": "What does the user want the voice assistant to do?",
      "criteria": {
        "alarm_query": "ask about alarms that are set",
        "alarm_remove": "delete an alarm",
        "alarm_set": "set an alarm",
        "audio_volume_down": "lower the volume",
        "audio_volume_mute": "mute or silence the device",
        "audio_volume_other": "other volume request",
        "audio_volume_up": "raise the volume",
        "calendar_query": "ask about calendar events",
        "calendar_remove": "delete a calendar event",
        "calendar_set": "add an event or reminder",
        "cooking_query": "cooking question",
        "cooking_recipe": "ask for a recipe",
        "datetime_convert": "convert time or time zone",
        "datetime_query": "ask the date or time",
        "email_addcontact": "add an email contact",
        "email_query": "check or read emails",
        "email_querycontact": "ask about a contact's details",
        "email_sendemail": "send or reply to an email",
        "general_greet": "greeting",
        "general_joke": "ask for a joke",
        "general_quirky": "chit-chat or an odd request",
        "iot_cleaning": "start the robot vacuum",
        "iot_coffee": "make coffee",
        "iot_hue_lightchange": "change light colour",
        "iot_hue_lightdim": "dim the lights",
        "iot_hue_lightoff": "turn lights off",
        "iot_hue_lighton": "turn lights on",
        "iot_hue_lightup": "brighten the lights",
        "iot_wemo_off": "turn a smart plug off",
        "iot_wemo_on": "turn a smart plug on",
        "lists_createoradd": "create a list or add to it",
        "lists_query": "ask what is on a list",
        "lists_remove": "remove a list or an item",
        "music_dislikeness": "say a song or artist is disliked",
        "music_likeness": "say a song or artist is liked",
        "music_query": "ask what music is playing",
        "music_settings": "shuffle, repeat or other music settings",
        "news_query": "ask for news",
        "play_audiobook": "play an audiobook",
        "play_game": "play a game",
        "play_music": "play music",
        "play_podcasts": "play a podcast",
        "play_radio": "play the radio",
        "qa_currency": "currency exchange question",
        "qa_definition": "ask for a definition",
        "qa_factoid": "factual question",
        "qa_maths": "maths question",
        "qa_stock": "stock price question",
        "recommendation_events": "recommend events",
        "recommendation_locations": "recommend places",
        "recommendation_movies": "recommend movies",
        "social_post": "post on social media or complain to a company",
        "social_query": "ask about social-media updates",
        "takeaway_order": "order takeaway food",
        "takeaway_query": "ask about a takeaway order or restaurant",
        "transport_query": "ask about trains or transport",
        "transport_taxi": "book a taxi",
        "transport_ticket": "book a ticket",
        "transport_traffic": "ask about traffic",
        "weather_query": "ask about the weather"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 04:06 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "qa",
      "probabilities": {
        "alarm": 0,
        "audio": 0,
        "calendar": 0,
        "cooking": 0,
        "datetime": 0,
        "email": 0,
        "general": 0,
        "iot": 0,
        "lists": 0,
        "music": 0,
        "news": 0,
        "play": 0,
        "qa": 1,
        "recommendation": 0,
        "social": 0,
        "takeaway": 0,
        "transport": 0,
        "weather": 0
      },
      "confidence": 1
    },
    "intent": {
      "type": "choice",
      "choice": "qa_factoid",
      "probabilities": {
        "alarm_query": 0,
        "alarm_remove": 0,
        "alarm_set": 0,
        "audio_volume_down": 0,
        "audio_volume_mute": 0,
        "audio_volume_other": 0,
        "audio_volume_up": 0,
        "calendar_query": 0,
        "calendar_remove": 0,
        "calendar_set": 0,
        "cooking_query": 0,
        "cooking_recipe": 0,
        "datetime_convert": 0,
        "datetime_query": 0,
        "email_addcontact": 0,
        "email_query": 0,
        "email_querycontact": 0,
        "email_sendemail": 0,
        "general_greet": 0,
        "general_joke": 0,
        "general_quirky": 0.15,
        "iot_cleaning": 0,
        "iot_coffee": 0,
        "iot_hue_lightchange": 0,
        "iot_hue_lightdim": 0,
        "iot_hue_lightoff": 0,
        "iot_hue_lighton": 0,
        "iot_hue_lightup": 0,
        "iot_wemo_off": 0,
        "iot_wemo_on": 0,
        "lists_createoradd": 0,
        "lists_query": 0,
        "lists_remove": 0,
        "music_dislikeness": 0,
        "music_likeness": 0,
        "music_query": 0,
        "music_settings": 0,
        "news_query": 0,
        "play_audiobook": 0,
        "play_game": 0,
        "play_music": 0,
        "play_podcasts": 0,
        "play_radio": 0,
        "qa_currency": 0,
        "qa_definition": 0,
        "qa_factoid": 0.85,
        "qa_maths": 0,
        "qa_stock": 0,
        "recommendation_events": 0,
        "recommendation_locations": 0,
        "recommendation_movies": 0,
        "social_post": 0,
        "social_query": 0,
        "takeaway_order": 0,
        "takeaway_query": 0,
        "transport_query": 0,
        "transport_taxi": 0,
        "transport_ticket": 0,
        "transport_traffic": 0,
        "weather_query": 0
      },
      "confidence": 0.85
    }
  },
  "usage": {
    "input_tokens": 1687,
    "output_tokens": 720,
    "cost": 0.000070854
  },
  "id": "gen-dec-1791518778-p1o1KOR6KDT83CNIxoKK",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by Cloudflare · 2026-10-09 04:06 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "qa",
      "probabilities": {
        "alarm": 0.003,
        "audio": 0.0034,
        "calendar": 0.0038,
        "cooking": 0.0045,
        "datetime": 0.0045,
        "email": 0.0028,
        "general": 0.117,
        "iot": 0.0027,
        "lists": 0.0025,
        "music": 0.0024,
        "news": 0.0052,
        "play": 0.004,
        "qa": 0.8159,
        "recommendation": 0.0041,
        "social": 0.003,
        "takeaway": 0.0024,
        "transport": 0.0079,
        "weather": 0.0109
      },
      "confidence": 0.6608
    },
    "intent": {
      "type": "choice",
      "choice": "qa_factoid",
      "probabilities": {
        "alarm_query": 0.0013,
        "alarm_remove": 0.0007,
        "alarm_set": 0.0009,
        "audio_volume_down": 0.0008,
        "audio_volume_mute": 0.0008,
        "audio_volume_other": 0.001,
        "audio_volume_up": 0.0008,
        "calendar_query": 0.0019,
        "calendar_remove": 0.0009,
        "calendar_set": 0.0013,
        "cooking_query": 0.0029,
        "cooking_recipe": 0.0041,
        "datetime_convert": 0.001,
        "datetime_query": 0.0014,
        "email_addcontact": 0.0008,
        "email_query": 0.0013,
        "email_querycontact": 0.0015,
        "email_sendemail": 0.001,
        "general_greet": 0.002,
        "general_joke": 0.0022,
        "general_quirky": 0.2389,
        "iot_cleaning": 0.0011,
        "iot_coffee": 0.001,
        "iot_hue_lightchange": 0.0008,
        "iot_hue_lightdim": 0.001,
        "iot_hue_lightoff": 0.0009,
        "iot_hue_lighton": 0.0009,
        "iot_hue_lightup": 0.001,
        "iot_wemo_off": 0.001,
        "iot_wemo_on": 0.001,
        "lists_createoradd": 0.0011,
        "lists_query": 0.0015,
        "lists_remove": 0.0011,
        "music_dislikeness": 0.001,
        "music_likeness": 0.001,
        "music_query": 0.0011,
        "music_settings": 0.0008,
        "news_query": 0.0024,
        "play_audiobook": 0.0017,
        "play_game": 0.0015,
        "play_music": 0.0017,
        "play_podcasts": 0.0021,
        "play_radio": 0.0027,
        "qa_currency": 0.0024,
        "qa_definition": 0.0048,
        "qa_factoid": 0.6745,
        "qa_maths": 0.0032,
        "qa_stock": 0.0022,
        "recommendation_events": 0.001,
        "recommendation_locations": 0.001,
        "recommendation_movies": 0.0009,
        "social_post": 0.0012,
        "social_query": 0.0011,
        "takeaway_order": 0.0009,
        "takeaway_query": 0.0011,
        "transport_query": 0.0029,
        "transport_taxi": 0.0011,
        "transport_ticket": 0.0016,
        "transport_traffic": 0.0013,
        "weather_query": 0.0029
      },
      "confidence": 0.5039
    }
  },
  "usage": {
    "input_tokens": 1847,
    "output_tokens": 0,
    "cost": 0.00044328
  },
  "id": "gen-dec-1791518789-s2EmCgBljoeWIimAXnKs",
  "provider": "Cloudflare"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 04:06 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "scenario": {
      "type": "choice",
      "choice": "weather",
      "probabilities": {
        "alarm": 0.0036,
        "audio": 0.0038,
        "calendar": 0.0153,
        "cooking": 0.0156,
        "datetime": 0.0203,
        "email": 0.0053,
        "general": 0.0261,
        "iot": 0.0163,
        "lists": 0.0066,
        "music": 0.0058,
        "news": 0.0131,
        "play": 0.0099,
        "qa": 0.0766,
        "recommendation": 0.0417,
        "social": 0.006,
        "takeaway": 0.006,
        "transport": 0.0123,
        "weather": 0.7157
      },
      "confidence": 0.699
    },
    "intent": {
      "type": "choice",
      "choice": "weather_query",
      "probabilities": {
        "alarm_query": 0.0017,
        "alarm_remove": 0.001,
        "alarm_set": 0.0011,
        "audio_volume_down": 0.0012,
        "audio_volume_mute": 0.0014,
        "audio_volume_other": 0.0013,
        "audio_volume_up": 0.0013,
        "calendar_query": 0.005,
        "calendar_remove": 0.0015,
        "calendar_set": 0.0023,
        "cooking_query": 0.0041,
        "cooking_recipe": 0.0045,
        "datetime_convert": 0.0023,
        "datetime_query": 0.0047,
        "email_addcontact": 0.0011,
        "email_query": 0.0015,
        "email_querycontact": 0.002,
        "email_sendemail": 0.0014,
        "general_greet": 0.0023,
        "general_joke": 0.0017,
        "general_quirky": 0.0248,
        "iot_cleaning": 0.0013,
        "iot_coffee": 0.0012,
        "iot_hue_lightchange": 0.0017,
        "iot_hue_lightdim": 0.0011,
        "iot_hue_lightoff": 0.0013,
        "iot_hue_lighton": 0.0019,
        "iot_hue_lightup": 0.0026,
        "iot_wemo_off": 0.0011,
        "iot_wemo_on": 0.0015,
        "lists_createoradd": 0.0013,
        "lists_query": 0.0016,
        "lists_remove": 0.0012,
        "music_dislikeness": 0.0015,
        "music_likeness": 0.0015,
        "music_query": 0.0018,
        "music_settings": 0.0015,
        "news_query": 0.0055,
        "play_audiobook": 0.0015,
        "play_game": 0.0017,
        "play_music": 0.0021,
        "play_podcasts": 0.0017,
        "play_radio": 0.0023,
        "qa_currency": 0.0019,
        "qa_definition": 0.0047,
        "qa_factoid": 0.0256,
        "qa_maths": 0.0013,
        "qa_stock": 0.0017,
        "recommendation_events": 0.0049,
        "recommendation_locations": 0.016,
        "recommendation_movies": 0.0018,
        "social_post": 0.0019,
        "social_query": 0.0015,
        "takeaway_order": 0.0012,
        "takeaway_query": 0.0017,
        "transport_query": 0.0032,
        "transport_taxi": 0.0015,
        "transport_ticket": 0.0019,
        "transport_traffic": 0.0018,
        "weather_query": 0.8202
      },
      "confidence": 0.8172
    }
  },
  "usage": {
    "input_tokens": 1847,
    "output_tokens": 0,
    "cost": 0.000038787
  },
  "id": "gen-dec-1791518795-fA17zz69oiT0BVh9p3ds",
  "provider": "PrimeIntellect"
}
Emotionnew to NirnoyEmoNoBa · CC-BY-4.0

গানটা অস্থির ছিলো

“The song was amazing.”

Which emotion does this comment mainly express? · 6 options
Dataset answer: joy

  • ✓Nirnoy-flashjoy36%
  • ✗Nirnoy 12Blove45%
  • ✗Jevfear35%
  • ✗Clefsadness63%
  • ✗Clef-flashsadness47%
  • ✓Gemma 12Bjoy100%
  • ✓Gemma E4Bjoy100%

“অস্থির” literally means restless; as slang it means amazing. Emotion was never part of Nirnoy's training.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "গানটা অস্থির ছিলো",
  "questions": {
    "emotion": {
      "type": "choice",
      "instructions": "Which emotion does this comment mainly express?",
      "criteria": {
        "love": null,
        "joy": null,
        "surprise": null,
        "anger": null,
        "sadness": null,
        "fear": null
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "emotion": {
      "type": "choice",
      "choice": "fear",
      "probabilities": {
        "love": 0.01,
        "joy": 0.04,
        "surprise": 0.04,
        "anger": 0.25,
        "sadness": 0.31,
        "fear": 0.35
      },
      "confidence": 0.22
    }
  },
  "usage": {
    "input_tokens": 331,
    "output_tokens": 61,
    "cost": 0.000013902
  },
  "id": "gen-dec-1791517792-FI518GmLEKg1jrbnEP8N",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "emotion": {
      "type": "choice",
      "choice": "sadness",
      "probabilities": {
        "love": 0.0084,
        "joy": 0.0212,
        "surprise": 0.0378,
        "anger": 0.2445,
        "sadness": 0.6268,
        "fear": 0.0613
      },
      "confidence": 0.5522
    }
  },
  "usage": {
    "input_tokens": 181,
    "output_tokens": 0,
    "cost": 0.000007602
  },
  "id": "gen-dec-1791517798-0FOLeJJbPqAlafWO9vkd",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "emotion": {
      "type": "choice",
      "choice": "sadness",
      "probabilities": {
        "love": 0.0283,
        "joy": 0.131,
        "surprise": 0.0432,
        "anger": 0.3088,
        "sadness": 0.4682,
        "fear": 0.0204
      },
      "confidence": 0.3618
    }
  },
  "usage": {
    "input_tokens": 181,
    "output_tokens": 0,
    "cost": 0.000003801
  },
  "id": "gen-dec-1791517803-4b1Sv1diuvZVnRtREVpe",
  "provider": "PrimeIntellect"
}
Emotionnew to NirnoyEmoNoBa · CC-BY-4.0

অসাধারন বলার কোন ভাষা নেই

“Amazing, there are no words to describe it.”

Which emotion does this comment mainly express? · 6 options
Dataset answer: joy

  • ✓Nirnoy-flashjoy79%
  • ✓Nirnoy 12Bjoy60%
  • ✗Jevsurprise41%
  • ✗Clefanger59%
  • ✗Clef-flashsurprise90%
  • ✗Gemma 12Bsurprise97%
  • ✓Gemma E4Bjoy97%

Delight, not surprise or anger.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "অসাধারন বলার কোন ভাষা নেই",
  "questions": {
    "emotion": {
      "type": "choice",
      "instructions": "Which emotion does this comment mainly express?",
      "criteria": {
        "love": null,
        "joy": null,
        "surprise": null,
        "anger": null,
        "sadness": null,
        "fear": null
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "emotion": {
      "type": "choice",
      "choice": "surprise",
      "probabilities": {
        "love": 0.16,
        "joy": 0.29,
        "surprise": 0.41,
        "anger": 0.05,
        "sadness": 0.08,
        "fear": 0.01
      },
      "confidence": 0.29
    }
  },
  "usage": {
    "input_tokens": 335,
    "output_tokens": 62,
    "cost": 0.00001407
  },
  "id": "gen-dec-1791517789-IFeFEd6QIB5fRqih6SHj",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "emotion": {
      "type": "choice",
      "choice": "anger",
      "probabilities": {
        "love": 0.0157,
        "joy": 0.0222,
        "surprise": 0.1539,
        "anger": 0.5853,
        "sadness": 0.1827,
        "fear": 0.0401
      },
      "confidence": 0.5024
    }
  },
  "usage": {
    "input_tokens": 187,
    "output_tokens": 0,
    "cost": 0.000007854
  },
  "id": "gen-dec-1791517795-Cme57VfLhJvOtHNXQvw3",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "emotion": {
      "type": "choice",
      "choice": "surprise",
      "probabilities": {
        "love": 0.0059,
        "joy": 0.0094,
        "surprise": 0.8949,
        "anger": 0.0367,
        "sadness": 0.0294,
        "fear": 0.0237
      },
      "confidence": 0.8739
    }
  },
  "usage": {
    "input_tokens": 187,
    "output_tokens": 0,
    "cost": 0.000003927
  },
  "id": "gen-dec-1791517800-FUKiAKD6YFUHWcpNTNFZ",
  "provider": "PrimeIntellect"
}
Emotionnew to NirnoyEmoNoBa · CC-BY-4.0

শেষের সিন টা দারুন ছিল,,, স্যাড সিকুয়েন্স টা কে কিভাবে রোমান্টিক এ পরিনত করলো,,,,😍👌👌

“The last scene was great... the way they turned the sad sequence into a romantic one 😍👌👌”

Which emotion does this comment mainly express? · 6 options
Dataset answer: joy

  • ✓Nirnoy-flashjoy76%
  • ✓Nirnoy 12Bjoy64%
  • ✗Jevlove59%
  • ✗Clefsadness42%
  • ✗Clef-flashlove90%
  • ✓Gemma 12Bjoy100%
  • ✓Gemma E4Bjoy100%

“Sad” describes the scene, not the writer's feeling.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "শেষের সিন টা দারুন ছিল,,, স্যাড সিকুয়েন্স টা কে কিভাবে রোমান্টিক এ পরিনত করলো,,,,😍👌👌",
  "questions": {
    "emotion": {
      "type": "choice",
      "instructions": "Which emotion does this comment mainly express?",
      "criteria": {
        "love": null,
        "joy": null,
        "surprise": null,
        "anger": null,
        "sadness": null,
        "fear": null
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "emotion": {
      "type": "choice",
      "choice": "love",
      "probabilities": {
        "love": 0.59,
        "joy": 0.37,
        "surprise": 0.03,
        "anger": 0,
        "sadness": 0.01,
        "fear": 0
      },
      "confidence": 0.51
    }
  },
  "usage": {
    "input_tokens": 383,
    "output_tokens": 61,
    "cost": 0.000016086
  },
  "id": "gen-dec-1791517790-S0VVg5GLQwteoh8Kboql",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "emotion": {
      "type": "choice",
      "choice": "sadness",
      "probabilities": {
        "love": 0.3801,
        "joy": 0.17,
        "surprise": 0.0134,
        "anger": 0.0087,
        "sadness": 0.4208,
        "fear": 0.007
      },
      "confidence": 0.305
    }
  },
  "usage": {
    "input_tokens": 226,
    "output_tokens": 0,
    "cost": 0.000009492
  },
  "id": "gen-dec-1791517795-N5WFnRbt9JZ95EVnqGIJ",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "emotion": {
      "type": "choice",
      "choice": "love",
      "probabilities": {
        "love": 0.902,
        "joy": 0.0453,
        "surprise": 0.0082,
        "anger": 0.0055,
        "sadness": 0.0342,
        "fear": 0.0048
      },
      "confidence": 0.8824
    }
  },
  "usage": {
    "input_tokens": 226,
    "output_tokens": 0,
    "cost": 0.000004746
  },
  "id": "gen-dec-1791517801-RiXVH6OnMzhUXqT8qVNj",
  "provider": "PrimeIntellect"
}
Emotionnew to NirnoyEmoNoBa · CC-BY-4.0

ভাইয়া ভালো লাগছে খুব 😍😍😍😍

“Bro, I'm really enjoying it 😍😍😍😍”

Which emotion does this comment mainly express? · 6 options
Dataset answer: joy

  • ✓Nirnoy-flashjoy93%
  • ✗Nirnoy 12Blove78%
  • ✗Jevlove80%
  • ✗Cleflove69%
  • ✗Clef-flashlove95%
  • ✗Gemma 12Blove100%
  • ✓Gemma E4Bjoy100%

Enjoyment, not affection: heart-eyes alone don't make it love.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "ভাইয়া ভালো লাগছে খুব 😍😍😍😍",
  "questions": {
    "emotion": {
      "type": "choice",
      "instructions": "Which emotion does this comment mainly express?",
      "criteria": {
        "love": null,
        "joy": null,
        "surprise": null,
        "anger": null,
        "sadness": null,
        "fear": null
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 04:06 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "emotion": {
      "type": "choice",
      "choice": "love",
      "probabilities": {
        "love": 0.8,
        "joy": 0.2,
        "surprise": 0,
        "anger": 0,
        "sadness": 0,
        "fear": 0
      },
      "confidence": 0.76
    }
  },
  "usage": {
    "input_tokens": 343,
    "output_tokens": 61,
    "cost": 0.000014406
  },
  "id": "gen-dec-1791518777-k7CP0j2FV6XputHR8x7q",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 04:06 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "emotion": {
      "type": "choice",
      "choice": "love",
      "probabilities": {
        "love": 0.6943,
        "joy": 0.2861,
        "surprise": 0.0063,
        "anger": 0.0055,
        "sadness": 0.0039,
        "fear": 0.0039
      },
      "confidence": 0.6332
    }
  },
  "usage": {
    "input_tokens": 192,
    "output_tokens": 0,
    "cost": 0.000008064
  },
  "id": "gen-dec-1791518789-NJiK4gQH8g9QvuNbSbUO",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 04:06 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "emotion": {
      "type": "choice",
      "choice": "love",
      "probabilities": {
        "love": 0.951,
        "joy": 0.0291,
        "surprise": 0.0048,
        "anger": 0.0053,
        "sadness": 0.0053,
        "fear": 0.0045
      },
      "confidence": 0.9412
    }
  },
  "usage": {
    "input_tokens": 192,
    "output_tokens": 0,
    "cost": 0.000004032
  },
  "id": "gen-dec-1791518795-IjXSuTlSUrdv54mpLOb5",
  "provider": "PrimeIntellect"
}
Hate speechBanTH · MIT

Amader Bangladesh er manush eto innocent j ekhono kichui bujhte parche na.

“The people of our Bangladesh are so innocent that they still don't understand anything.”

Is this comment hate speech? · yes / no
Dataset answer: no

  • ✓Nirnoy-flashno83%
  • ✓Nirnoy 12Bno75%
  • ✗Jevyes74%
  • ✗Clefyes77%
  • ✗Clef-flashyes80%
  • ✓Gemma 12Bno100%
  • ✓Gemma E4Bno100%

Critical of people, but not hate speech.

See the request and the responses

Sent to https://openrouter.ai/api/alpha/decisions, once per model, with "model" set to typesafe/jev-1.13, cloudflare/clef or cloudflare/clef-flash, plus our data setting in "provider" (no storing or training on it).

{
  "model": "cloudflare/clef",
  "state": "Amader Bangladesh er manush eto innocent j ekhono kichui bujhte parche na.",
  "questions": {
    "hate": {
      "type": "noul",
      "instructions": "Is this comment hate speech?",
      "criteria": {
        "true": "it attacks, insults, threatens or dehumanises a person or group",
        "false": "it does not attack anyone, or only criticises ideas or actions without abuse"
      }
    }
  }
}

Jev's response, exactly as returned · served by TypeSafe · 2026-10-09 03:49 UTC

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "hate": {
      "type": "noul",
      "noul": 0.74
    }
  },
  "usage": {
    "input_tokens": 340,
    "output_tokens": 20,
    "cost": 0.00001428
  },
  "id": "gen-dec-1791517792-526eAtbCF7W28aDymSu9",
  "provider": "TypeSafe"
}

Clef's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:49 UTC

{
  "model": "cloudflare/clef",
  "answers": {
    "hate": {
      "type": "noul",
      "noul": 0.7677
    }
  },
  "usage": {
    "input_tokens": 176,
    "output_tokens": 0,
    "cost": 0.000007392
  },
  "id": "gen-dec-1791517797-ytefsJyJtl0jnVOp6Feg",
  "provider": "PrimeIntellect"
}

Clef-flash's response, exactly as returned · served by PrimeIntellect · 2026-10-09 03:50 UTC

{
  "model": "cloudflare/clef-flash",
  "answers": {
    "hate": {
      "type": "noul",
      "noul": 0.795
    }
  },
  "usage": {
    "input_tokens": 176,
    "output_tokens": 0,
    "cost": 0.000003696
  },
  "id": "gen-dec-1791517803-Nh2cgi4C30Dr8vGUlQVL",
  "provider": "PrimeIntellect"
}

How we picked

Every answer here is the model's real output. Jev, Clef and Clef-flash answered through their public APIs (OpenRouter, 9 October 2026), given the same text, question and answer options as Nirnoy; Nirnoy and the untrained Gemma 4 models answered in our benchmark runs, the Gemma models with the same prompts as Nirnoy. Nirnoy, Jev, Clef and Clef-flash saw the question with every answer option. The Gemma models were only run with up to 10 options, always including the dataset's answer, so they are left out on the intent and scenario questions, which have more. We looked for questions where a Nirnoy model matched the dataset and Jev, Clef and Clef-flash did not, picked ones whose dataset answer we agree with, and left out messages with personal details or offensive language. The percentage is each model's probability for its own answer.

Nirnoy trained on other examples from these datasets, except emotion, which is new to it. None of these questions was in its training data.

The texts come from BnSentMix (MIT), MASSIVE bn-BD (CC BY 4.0), EmoNoBa (CC BY 4.0) and BanTH (MIT), used under those licences; the texts are unchanged. The English glosses and notes are ours.