Skip to main content
Benchmarks are published to explain where Mibyan is strong and where it still needs review. A benchmark score is not a guarantee for every prompt, and scores from different harnesses are not comparable unless the dataset, prompt, sampling settings, and scoring rules are the same.

​
What is being evaluated

Mibyan is an Arabic-first model family focused on Arabic language quality and practical business use: Modern Standard Arabic, regional terminology, Arabic-English code switching, structured writing, and instruction adherence. Use mibyan-4.1 for production integrations. We evaluate five dimensions:

​
Scorecard contract

Every published scorecard must answer the questions a researcher or buyer needs before trusting the comparison. A number without this context is a marketing artifact, not an evaluation result.

​
Comparison with frontier models

The right comparison depends on the job. Frontier models may lead on breadth, novel reasoning, multimodal capability, or tool ecosystems. Mibyan is optimized for Arabic-first professional workflows and predictable integration through a stable API. A fair evaluation compares the same prompts, context, output limits, temperature, tools, and judge rubric across every model.
Numerical results are intentionally not filled with estimates. We will publish a score only with a versioned dataset, test date, sample count, prompt template, decoding settings, and reproducible scoring method.

​
What a fair comparison does not say

  • It does not claim a general winner from one language, one prompt family, or one judge.
  • It does not compare Mibyan’s task-specific Arabic result against a frontier model’s unrelated public leaderboard score.
  • It does not hide a different context window, output cap, tool policy, retry policy, or region behind a single latency figure.
  • It does not turn an internal routing choice into a claim about model weights or base-model capability.
This is how we make a useful statement such as: Mibyan delivered a stronger acceptance rate for a named Arabic professional workload under a defined serving profile. It is deliberately more precise—and more valuable—than claiming universal superiority.
  1. Freeze the model IDs and test date.
  2. Build a held-out set that reflects real Arabic work; do not train on test prompts.
  3. Include MSA, Gulf terminology, Arabic-English mixed prompts, and adversarial or ambiguous requests.
  4. Run at least three seeds when sampling is enabled and report mean plus variation.
  5. Use exact-match metrics for deterministic tasks and blinded human or judge scoring for writing quality.
  6. Report failures separately: hallucination, missed constraint, unsafe answer, bad Arabic, formatting error, timeout, and tool error.
  7. Publish the harness version and raw aggregate counts so results can be audited.

​
What a published score must include

Every result should state: model ID, evaluation version, dataset provenance, language mix, number of examples, prompt and system instructions, temperature and token limits, judge model or human rubric, confidence interval where applicable, and whether tools or retrieval were enabled. For production decisions, run a private pilot on your own prompts as well. Measure acceptance rate, edit distance, time saved, cost per accepted output, and failure recovery—not just a public leaderboard score.