> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mibyanai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Mibyan benchmarks

> How we evaluate Arabic-first quality, instruction following, safety, and production usefulness

Benchmarks are published to explain where Mibyan is strong and where it still needs review. A benchmark score is not a guarantee for every prompt, and scores from different harnesses are not comparable unless the dataset, prompt, sampling settings, and scoring rules are the same.

## What is being evaluated

Mibyan is an Arabic-first model family focused on Arabic language quality and practical business use: Modern Standard Arabic, regional terminology, Arabic-English code switching, structured writing, and instruction adherence. Use `mibyan-4.1` for production integrations.

We evaluate five dimensions:

| Dimension             | Examples of tasks                                                  | Primary measures                        |
| --------------------- | ------------------------------------------------------------------ | --------------------------------------- |
| Arabic                | summarisation, rewriting, terminology, dialect-aware understanding | expert rating, rubric score, factuality |
| Reasoning             | multi-step analysis, extraction, classification, calculations      | exact match, pass rate, judge score     |
| Instruction following | constraints, formats, tone, refusal boundaries                     | constraint satisfaction                 |
| Multilingual work     | Arabic/English translation and mixed prompts                       | adequacy, terminology, human preference |
| Production quality    | latency, errors, streaming, cost, consistency                      | p50/p95 latency, success rate, tokens   |

## Scorecard contract

Every published scorecard must answer the questions a researcher or buyer needs before trusting the comparison. A number without this context is a marketing artifact, not an evaluation result.

| Required field    | Publication rule                                                                                      |
| ----------------- | ----------------------------------------------------------------------------------------------------- |
| Release identity  | State the public model ID, evaluation date, and harness version                                       |
| Data provenance   | Describe the held-out dataset, language mix, task families, and exclusions                            |
| Run configuration | Publish prompt template, decoding settings, context/output limits, tools, and retry policy            |
| Scoring           | Name the metric, rubric, judge or human-review method, and adjudication policy                        |
| Uncertainty       | Give sample count and variance, confidence interval, or an explicit reason it is unavailable          |
| Failure analysis  | Break out missed constraint, factual error, language error, formatting error, timeout, and tool error |
| Serving context   | For latency/cost claims, state region, concurrency, and whether streaming and retries were enabled    |

## Comparison with frontier models

The right comparison depends on the job. Frontier models may lead on breadth, novel reasoning, multimodal capability, or tool ecosystems. Mibyan is optimized for Arabic-first professional workflows and predictable integration through a stable API. A fair evaluation compares the same prompts, context, output limits, temperature, tools, and judge rubric across every model.

| Track                        | Mibyan                                  | Frontier reference models    | Interpretation                                             |
| ---------------------------- | --------------------------------------- | ---------------------------- | ---------------------------------------------------------- |
| Arabic professional writing  | Publish per-dataset score               | Run the same harness         | Quality and terminology matter more than a general average |
| Arabic instruction following | Publish pass rate                       | Run the same harness         | Count every required constraint, not only fluency          |
| General reasoning            | Publish exact/rubric score              | Run the same harness         | Use as a capability signal, not a product promise          |
| Production serving           | Publish measured p50/p95 and error rate | Measure from the same region | Include retries, streaming, and rate limits                |

<Note>Numerical results are intentionally not filled with estimates. We will publish a score only with a versioned dataset, test date, sample count, prompt template, decoding settings, and reproducible scoring method.</Note>

## What a fair comparison does not say

* It does not claim a general winner from one language, one prompt family, or one judge.
* It does not compare Mibyan's task-specific Arabic result against a frontier model's unrelated public leaderboard score.
* It does not hide a different context window, output cap, tool policy, retry policy, or region behind a single latency figure.
* It does not turn an internal routing choice into a claim about model weights or base-model capability.

This is how we make a useful statement such as: *Mibyan delivered a stronger acceptance rate for a named Arabic professional workload under a defined serving profile.* It is deliberately more precise—and more valuable—than claiming universal superiority.

## Recommended evaluation protocol

1. Freeze the model IDs and test date.
2. Build a held-out set that reflects real Arabic work; do not train on test prompts.
3. Include MSA, Gulf terminology, Arabic-English mixed prompts, and adversarial or ambiguous requests.
4. Run at least three seeds when sampling is enabled and report mean plus variation.
5. Use exact-match metrics for deterministic tasks and blinded human or judge scoring for writing quality.
6. Report failures separately: hallucination, missed constraint, unsafe answer, bad Arabic, formatting error, timeout, and tool error.
7. Publish the harness version and raw aggregate counts so results can be audited.

## What a published score must include

Every result should state: model ID, evaluation version, dataset provenance, language mix, number of examples, prompt and system instructions, temperature and token limits, judge model or human rubric, confidence interval where applicable, and whether tools or retrieval were enabled.

For production decisions, run a private pilot on your own prompts as well. Measure acceptance rate, edit distance, time saved, cost per accepted output, and failure recovery—not just a public leaderboard score.