> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mibyanai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Middleware

> Behavior-changing plugin middleware for LLM and tool calls: contract, execution order, examples

Mibyan middleware is the behavior-changing companion to observer hooks.
Observer hooks report what happened. Middleware can change what happens by
rewriting a request before execution or by wrapping the execution callback
itself.

This contract is intentionally backend-neutral. A plugin can use it for local
policy, request shaping, tracing, adaptive routing, cache control, sandbox
selection, or handoff to runtimes such as NeMo Relay without changing Mibyan'
planner, model provider adapters, tool registry, memory, or CLI UX.

With middleware enabled, plugins can:

* Rewrite LLM provider request kwargs before Mibyan calls the provider.
* Rewrite tool arguments before guardrails, approval checks, hooks, and tool
  execution see them.
* Wrap the actual LLM execution callback while preserving Mibyan retry,
  streaming, interrupt, and hook behavior.
* Wrap the actual tool execution callback while preserving Mibyan guardrails,
  approval, post-tool hooks, and tool-result transformation.

## Contract

Plugins register middleware from `register(ctx)`:

```python theme={null}
def register(ctx):
    ctx.register_middleware("llm_request", on_llm_request)
    ctx.register_middleware("llm_execution", on_llm_execution)
    ctx.register_middleware("tool_request", on_tool_request)
    ctx.register_middleware("tool_execution", on_tool_execution)
```

Every middleware callback receives:

* `telemetry_schema_version`: currently `mibyan.observer.v1`
* `middleware_schema_version`: currently `mibyan.middleware.v1`
* Runtime context such as `session_id`, `task_id`, `turn_id`,
  `api_request_id`, `provider`, `model`, `api_mode`, `tool_name`, and
  `tool_call_id` when applicable.

Supported middleware kinds:

| Kind | Payload | Return shape | Purpose |
| - | - | - | - |
| `llm_request` | `request`, `original_request` | `{"request": {...}}` | Replace effective provider kwargs before provider execution. |
| `tool_request` | `tool_name`, `args`, `original_args` | `{"args": {...}}` | Replace effective tool args before hooks, guardrails, approvals, and execution. |
| `llm_execution` | `request`, `original_request`, `next_call` | Any provider response | Wrap or replace the actual provider call. |
| `tool_execution` | `tool_name`, `args`, `original_args`, `next_call` | Any tool result | Wrap or replace the actual tool call. |

Request middleware can return optional trace fields:

```python theme={null}
return {
    "request": updated_request,
    "source": "my-plugin",
    "reason": "selected fallback model",
}
```

Mibyan stores those trace entries in later observer hook payloads as
`middleware_trace`.

Execution middleware receives a `next_call` callback. Call it to continue the
chain:

```python theme={null}
def on_tool_execution(**kwargs):
    result = kwargs["next_call"](kwargs["args"])
    return result
```

If multiple plugins register the same execution middleware kind, Mibyan runs
them as a nested chain in registration order. Middleware failures are fail-open:
Mibyan logs a warning and continues with the next middleware or the base
runtime path. A callback that fails the same way on every call (typically a
signature naming a field the middleware does not send) is reported **once** at
WARNING — the message lists the fields it does provide — and identical repeats go
to DEBUG, so a mis-declared middleware cannot flood the log; a plugin reload
resets the report.

## Execution Order

### LLM Calls

For each provider request, Mibyan applies middleware in this order:

1. Build provider kwargs from the current conversation.
2. Apply `llm_request` middleware.
3. Emit `pre_api_request` observer hooks with the effective request.
4. Run provider execution through `llm_execution` middleware.
5. Emit `post_api_request` or `api_request_error` observer hooks.

Request middleware sees the full provider kwargs, including `messages` or
Responses API `input`, model settings, tool definitions, stream options, and
provider-specific options. Execution middleware receives the same effective
request plus `next_call`.

### Tool Calls

For each tool call, Mibyan applies middleware in this order:

1. Parse and coerce model-provided tool arguments.
2. Apply `tool_request` middleware.
3. Run the normal Mibyan pre-execution path against the effective arguments:
   tool availability checks, observer block directives, guardrails, and
   approval checks.
4. Run tool execution through `tool_execution` middleware.
5. Emit `post_tool_call` observer hooks.
6. Apply `transform_tool_result` hooks before the result is appended back into
   conversation context.

Tool request middleware runs before approval checks. Use it carefully: a
rewritten path, command, or URL is the value downstream policy will evaluate.

## Enablement

Middleware only runs for enabled plugins. For a bundled plugin:

```bash theme={null}
mibyan plugins enable <plugin-name>
```

For isolated local testing, use one `mibyan_HOME` for plugin enablement and the
agent run:

```bash theme={null}
export mibyan_HOME=$HOME/.mibyan/cache/scratch/mibyan-middleware-test
mkdir -p "$mibyan_HOME"
mibyan plugins enable <plugin-name>
mibyan chat --query 'Reply exactly ok'
```

For source checkouts, use the [PM developer workflow](/desktop/reference/package-management#developer-workflow)
and a separate development home so the runtime sees plugins and middleware from
the working tree:

```bash theme={null}
export mibyan_HOME="$HOME/mibyan-middleware-test"
export mibyan_RUNTIME_DIR="$mibyan_HOME/tools"
source ./activate
python mibyan plugins enable <plugin-name>
python mibyan chat --query 'Reply exactly ok'
```

## Generic Plugin Examples

The examples below are intentionally small. They show the middleware contract
shape without depending on NeMo Relay.

### LLM Request Middleware

This plugin tags provider requests and records a middleware trace entry:

```python theme={null}
def register(ctx):
    ctx.register_middleware("llm_request", tag_llm_request)

def tag_llm_request(**kwargs):
    request = dict(kwargs["request"])
    extra_body = dict(request.get("extra_body") or {})
    extra_body.setdefault("metadata", {})["mibyan_middleware_demo"] = True
    request["extra_body"] = extra_body
    return {
        "request": request,
        "source": "middleware-demo",
        "reason": "tagged provider request",
    }
```

The effective request is passed to `pre_api_request`, provider execution, and
`post_api_request`.

### Tool Request Middleware

This plugin constrains `terminal` calls to a known working directory:

```python theme={null}
from pathlib import Path

def register(ctx):
    ctx.register_middleware("tool_request", normalize_terminal_workdir)

def normalize_terminal_workdir(**kwargs):
    if kwargs.get("tool_name") != "terminal":
        return None
    args = dict(kwargs["args"])
    args.setdefault("workdir", str(Path.home() / ".mibyan" / "cache" / "scratch" / "mibyan-middleware-demo"))
    return {
        "args": args,
        "source": "middleware-demo",
        "reason": "defaulted terminal workdir",
    }
```

Because this runs before hooks and approvals, downstream telemetry and policy
observe the rewritten `workdir`.

### LLM Execution Middleware

This plugin wraps the provider call and preserves the raw provider response:

```python theme={null}
import time

def register(ctx):
    ctx.register_middleware("llm_execution", time_llm_execution)

def time_llm_execution(**kwargs):
    started = time.monotonic()
    response = kwargs["next_call"](kwargs["request"])
    elapsed_ms = int((time.monotonic() - started) * 1000)
    print(f"llm_execution elapsed_ms={elapsed_ms}")
    return response
```

Return the same response shape Mibyan expects from the provider adapter. Do not
wrap the response in a plugin-specific envelope unless the rest of the runtime
expects that envelope.

### Tool Execution Middleware

This plugin wraps tool execution while preserving the tool result:

```python theme={null}
def register(ctx):
    ctx.register_middleware("tool_execution", annotate_tool_execution)

def annotate_tool_execution(**kwargs):
    result = kwargs["next_call"](kwargs["args"])
    # Metrics, logging, or external routing can happen here.
    return result
```

Execution middleware may call `next_call(modified_args)` to pass a changed
payload to later middleware and the base tool dispatcher.

Plugin-specific examples should live with the plugin that owns the behavior.
NeMo Relay execution middleware is installed through Relay's discovered user
and system configuration, or through an explicit `plugins.toml` selected with
`mibyan_NEMO_RELAY_PLUGINS_TOML`; see
[Relay shared metrics](/desktop/developer-guide/relay-shared-metrics).

## Safety Notes

* Middleware should be deterministic for the same input unless it is explicitly
  routing to a dynamic external system.
* Request middleware should return complete replacement payloads, not partial
  patches.
* Execution middleware should call `next_call(...)` exactly once unless it is
  intentionally short-circuiting execution.
* If execution middleware raises before calling `next_call(...)`, Mibyan treats
  that as middleware failure and continues with the remaining middleware chain
  and base execution.
* If execution middleware calls `next_call(...)` successfully and then raises
  during post-processing, Mibyan preserves the downstream result and does not
  run the provider or tool a second time.
* If downstream provider or tool execution fails, middleware may let that error
  propagate or translate it deliberately. Mibyan does not convert downstream
  failure into a successful `None` result.
* Tool request middleware runs before approvals. If it mutates file paths,
  commands, URLs, or arguments, the mutated values are what guardrails and
  approvals evaluate.
* Observer hooks remain the right place for read-only telemetry. Use middleware
  only when a plugin needs to alter or wrap behavior.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.