Web Search Provider Plugins

Web-search provider plugins register a backend that services web_search, web_extract, and (optionally) deep-crawl tool calls. Built-in providers — Firecrawl, SearXNG, Tavily, Perplexity, Exa, Parallel, Keenable, Brave Search (free tier), xAI, and DDGS — all ship as plugins under plugins/web/<name>/. You can add a new one, or override a bundled one, by dropping a directory next to them.

Web search is one of several backend plugins Mibyan supports. The others (with their own ABCs) are Image Generation Provider Plugins, Video Generation Provider Plugins, Memory Provider Plugins, Context Engine Plugins, and Model Provider Plugins. General tool/hook/CLI plugins live in Build a Mibyan Plugin.

How discovery works

Mibyan scans for web-search backends in three places:

Bundled — <repo>/plugins/web/<name>/ (auto-loaded with kind: backend, always available)
User — ~/.mibyan/plugins/web/<name>/ (opt-in via plugins.enabled or mibyan plugins enable <name>)
Pip — packages declaring a mibyan_agent.plugins entry point

Each plugin’s register(ctx) function calls ctx.register_web_search_provider(...) — that puts the instance into the registry in agent/web_search_registry.py. The active provider for each capability is picked by config:

Capability	Config key	Falls back to
`web_search`	`web.search_backend`	`web.backend`
`web_extract`	`web.extract_backend`	`web.backend`
Deep crawl modes inside `web_extract`	`web.extract_backend`	`web.backend`

When neither key is set, Mibyan auto-detects the backend from whichever API key/URL is present in the environment. mibyan tools walks users through selection.

Directory structure

plugins/web/my-backend/
├── __init__.py     # register() entry point
├── provider.py     # WebSearchProvider subclass
└── plugin.yaml     # Manifest with kind: backend and provides_web_providers

brave_free/ and ddgs/ are the smallest in-tree references — brave_free for an API-key-gated search-only provider, ddgs for a no-key provider that lazy-installs its SDK.

The WebSearchProvider ABC

Subclass agent.web_search_provider.WebSearchProvider. The only required members are name, is_available(), and whichever of search() / extract() you implement. (Deep crawling is not a separate method — it’s a mode of extract().)

# plugins/web/my-backend/provider.py
from __future__ import annotations

import os
from typing import Any, Dict, List

from agent.web_search_provider import WebSearchProvider

class MyBackendWebSearchProvider(WebSearchProvider):
    """Minimal search-only provider against the My Backend HTTP API."""

    @property
    def name(self) -> str:
        # Stable id used in web.search_backend / web.extract_backend / web.backend
        # config keys. Lowercase, no spaces; hyphens permitted.
        return "my-backend"

    @property
    def display_name(self) -> str:
        # Human label shown in `mibyan tools`. Defaults to `name`.
        return "My Backend"

    def is_available(self) -> bool:
        # Cheap check — env var present, optional dep importable, etc.
        # MUST NOT make network calls (runs on every `mibyan tools` paint).
        return bool(os.getenv("MY_BACKEND_API_KEY", "").strip())

    def supports_search(self) -> bool:
        return True

    def supports_extract(self) -> bool:
        return False

    def search(self, query: str, limit: int = 5) -> Dict[str, Any]:
        import httpx

        api_key = os.environ["MY_BACKEND_API_KEY"]
        try:
            resp = httpx.get(
                "https://api.example.com/search",
                params={"q": query, "count": max(1, min(int(limit), 20))},
                headers={"Authorization": f"Bearer {api_key}"},
                timeout=15,
            )
            resp.raise_for_status()
            data = resp.json()
        except httpx.HTTPError as exc:
            return {"success": False, "error": str(exc)}

        # Response shape is fixed — see "Response shape" below.
        return {
            "success": True,
            "data": {
                "web": [
                    {
                        "title": item.get("title", ""),
                        "url": item.get("url", ""),
                        "description": item.get("snippet", ""),
                        "position": idx + 1,
                    }
                    for idx, item in enumerate(data.get("results", []))
                ],
            },
        }

# plugins/web/my-backend/__init__.py
from plugins.web.my_backend.provider import MyBackendWebSearchProvider

def register(ctx) -> None:
    """Plugin entry point — called once at load time."""
    ctx.register_web_search_provider(MyBackendWebSearchProvider())

plugin.yaml

name: web-my-backend
version: 1.0.0
description: "My Backend web search — Bearer-auth REST API"
author: Your Name
kind: backend
provides_web_providers:
  - my-backend
requires_env:
  - MY_BACKEND_API_KEY

Key	Purpose
`kind: backend`	Routes the plugin through the backend-loading path
`provides_web_providers`	List of provider `name`s this plugin registers — used by the loader to advertise the plugin in `mibyan tools` even before `register()` runs
`requires_env`	Interactive credential prompt during `mibyan plugins install` (see Build a Mibyan Plugin for the rich format)

ABC reference

Full contract in agent/web_search_provider.py. Methods you may override:

Member	Required	Default	Purpose
`name`	✅	—	Stable id used in `web.*_backend` config
`display_name`	—	`name`	Label shown in `mibyan tools`
`is_available()`	✅	—	Cheap availability gate — env vars, optional deps
`supports_search()`	—	`True`	Capability flag for `web_search` routing
`supports_extract()`	—	`False`	Capability flag for `web_extract` routing
`search(query, limit)`	conditional	raises	Required when `supports_search()` returns `True`
`extract(urls, **kwargs)`	conditional	raises	Required when `supports_extract()` returns `True`

Providers can advertise multiple capabilities from a single class — Firecrawl, Tavily, Perplexity, Keenable, Exa, and Parallel all implement both search and extract. Brave Search and DDGS are search-only; SearXNG is search-only with a documented “pair me with an extract provider” workflow.

Response shape

The tool wrapper expects a fixed envelope so it doesn’t have to translate between backends. Search success:

{
    "success": True,
    "data": {
        "web": [
            {"title": str, "url": str, "description": str, "position": int},
            ...
        ],
    },
}

Extract success:

{
    "success": True,
    "data": [
        {
            "url": str,
            "title": str,
            "content": str,
            "raw_content": str,
            "metadata": dict,    # optional
            "error": str,        # optional, only on per-URL failure
        },
        ...
    ],
}

Either capability, on failure:

{"success": False, "error": "human-readable message"}

Both search() and extract() may be async def — the dispatcher detects coroutine functions via inspect.iscoroutinefunction and awaits accordingly. Sync implementations that do blocking I/O (HTTP, SDK calls) are fine for small backends; the dispatcher handles threading.

Capability flags

Mibyan routes calls to the right provider based on the supports_* flags. A common multi-provider setup:

# ~/.mibyan/config.yaml
web:
  search_backend: "brave-free"     # search-only, fast, free 2k/mo
  extract_backend: "firecrawl"     # extract + crawl, paid quota

When web.search_backend or web.extract_backend aren’t set, both fall through to web.backend. When that’s also unset, Mibyan picks the first available provider that supports the requested capability based on env-var presence. If your provider only supports one capability, leave the other flags at their default (False) and the registry will skip it for that tool — users won’t see misleading “provider X failed” errors when they’re using X only for search and asking the agent to extract.

How Mibyan wires it into the tools

The web_search and web_extract tools live in tools/web_tools.py. At call time they:

Read the relevant config key (web.search_backend for web_search, web.extract_backend for web_extract)
Ask the registry for the provider with that name
Check is_available() and the matching supports_*() flag
Dispatch to search() / extract() (deep crawl runs as a mode inside extract()), awaiting if the method is a coroutine
JSON-serialize the response envelope and hand it back to the LLM

Errors surface as the tool result; the LLM decides how to explain them. If no provider is registered (or every available one fails the capability gate), the tool returns a helpful error pointing at mibyan tools.

Lazy-installing optional dependencies

Keep availability checks read-only. For an SDK covered by a Mibyan extra, use pm.available("extra-name") in is_available(). Request pm.ensure_import("extra-name") from the operation that needs it. Report InstallError, including a required restart, to the caller. Declare a third-party plugin’s own dependencies in its manifest or pyproject.toml rather than inventing a Mibyan extra. See Build a Mibyan Plugin → Lazy-install.

Reference implementations

plugins/web/brave_free/ — small, API-key-gated, search-only HTTP provider. Good starting template.
plugins/web/ddgs/ — no-key provider that lazy-installs its SDK. Useful pattern for backends that wrap a Python package.
plugins/web/firecrawl/ — full multi-capability provider (search + extract + crawl) with multiple format modes.
plugins/web/searxng/ — self-hosted, URL-configured backend with no auth.
plugins/web/xai/ — LLM-backed search via Grok’s server-side web_search tool. Shows how to reuse an existing OAuth/env-var credential surface (tools/xai_http.py) without adding new env vars, and how to write a cheap is_available() that honors the no-network contract.

Distribute via pip

# pyproject.toml
[project.entry-points."mibyan_agent.plugins"]
my-backend-web = "my_backend_web_package"

my_backend_web_package must expose a top-level register function. See Distribute via pip in the general plugin guide for the full setup.

Web Search — user-facing feature documentation and per-backend configuration
Plugins overview — all plugin types at a glance
Build a Mibyan Plugin — general tools/hooks/slash commands guide

تطبيق ديسكتوب

البداية

استخدام مبيان

الميزات

منصات المراسلة

التكاملات

الأدلة والدروس

دليل المطوّر

المرجع

Web Search Provider Plugins

How discovery works

Directory structure

The WebSearchProvider ABC

plugin.yaml

ABC reference

Response shape

Capability flags

How Mibyan wires it into the tools

Lazy-installing optional dependencies

Reference implementations

Distribute via pip

​How discovery works

​Directory structure

​The WebSearchProvider ABC

​plugin.yaml

​ABC reference

​Response shape

​Capability flags

​How Mibyan wires it into the tools

​Lazy-installing optional dependencies

​Reference implementations

​Distribute via pip

​Related pages

How discovery works

Directory structure

The WebSearchProvider ABC

plugin.yaml

ABC reference

Response shape

Capability flags

How Mibyan wires it into the tools

Lazy-installing optional dependencies

Reference implementations

Distribute via pip

Related pages