Run the installed chat evaluation suite

Use the installed module CLI when you want to evaluate a provider against the datasets shipped in the GeneralManager wheel. The command runs inside a Django project, so it must load that project's settings before it imports providers or registers fixtures.

Configure the provider first by following Add LLM chat to a GeneralManager project. The eval runner uses the same provider adapter and model configuration as the runtime chat API unless command options override them.

1. Choose the Django settings module

Pass the module explicitly:

python -m general_manager.chat.evals \
  --settings myproject.settings \
  --help

For repeated runs, set the standard Django environment variable instead:

export DJANGO_SETTINGS_MODULE=myproject.settings
python -m general_manager.chat.evals --help

--help is the only normal invocation that does not require settings. A run without --settings or a nonempty DJANGO_SETTINGS_MODULE exits with status 2 before django.setup() runs. GeneralManager 0.62.3 removed the old fallback to the repository's test settings, so installed commands cannot silently evaluate against a development-only configuration.

2. Run a shipped dataset

The wheel includes basic_queries, demo_readiness, edge_cases, follow_ups, large_schema, multi_hop, and planned_orchestration. The legacy CLI runs the six provider-backed datasets by default. The planned_orchestration dataset is packaged separately for deterministic, role-pinned tests and is not part of the legacy CLI suite. The basic_queries dataset can use the built-in toy schema and data:

python -m general_manager.chat.evals \
  --settings myproject.settings \
  --provider general_manager.chat.providers.OllamaProvider \
  --dataset basic_queries \
  --fixture toy \
  --tier 0 \
  --verbose

Omit --provider to use the provider configured in GENERAL_MANAGER["CHAT"]. Omit --dataset to run every legacy-compatible shipped dataset whose managers and expectations fit the configured project. Selecting --dataset planned_orchestration in this CLI is rejected; use the deterministic test command in section 6 instead. Use --fixture large with large_schema; omit fixtures when the selected dataset is intended to run against your project's own managers and data.

3. Narrow or compare a run

Repeat --tag to require several tags, and use --model to override the model inside the selected provider configuration for this invocation:

python -m general_manager.chat.evals \
  --settings myproject.settings \
  --dataset demo_readiness \
  --tier 1 \
  --tag grounding \
  --tag discovery \
  --model llama3 \
  --trace-jsonl /tmp/chat-eval.jsonl

Compare providers by passing their import paths as one comma-separated value:

python -m general_manager.chat.evals \
  --settings myproject.settings \
  --dataset basic_queries \
  --fixture toy \
  --compare general_manager.chat.providers.OllamaProvider,myproject.chat.TestProvider

The command exits with status 0 when every selected case passes and status 1 when any case fails. Provider imports, provider construction, Django setup, and dataset-loading errors propagate and produce a nonzero process exit.

4. Interpret the report

Treat hard product-contract failures as regressions. Strategy diagnostics can identify a skipped discovery path even when the final answer still satisfies the product contract. Use --verbose for failure details and --trace-jsonl when you need the full per-case trace.

See the chat prompt and eval model, the copy-ready command recipes, and the complete chat eval CLI reference.

5. Roll out planned chat safely

Keep planned chat disabled by default while you validate the application's catalog, profile construction, role mappings, and single trust group. The smallest rollout uses the legacy provider as the implicit default profile:

GENERAL_MANAGER = {
    "CHAT": {
        "enabled": True,
        "provider": "general_manager.chat.providers.OllamaProvider",
        "provider_config": {"model": "gemma4:e4b"},
        "planned": {
            "enabled": True,
            "catalog": "myproject.chat.catalog.catalog",
        },
    }
}

For role-specific models, configure explicit profiles. Every role must be mapped, every profile must provide a trust_group, and all mapped profiles must use the same trust group:

GENERAL_MANAGER["CHAT"].update(
    {
        "provider_profiles": {
            "fast_local": {
                "provider": "general_manager.chat.providers.OllamaProvider",
                "provider_config": {"model": "gemma4:e4b"},
                "trust_group": "local",
            },
            "strong_local": {
                "provider": "general_manager.chat.providers.OllamaProvider",
                "provider_config": {"model": "qwen3.5:9b"},
                "trust_group": "local",
            },
        },
        "planned": {
            "enabled": True,
            "catalog": "myproject.chat.catalog.catalog",
            "roles": {
                "planner": "strong_local",
                "simple_executor": "fast_local",
                "complex_executor": "strong_local",
                "synthesizer": "strong_local",
                "fallback_executor": "strong_local",
            },
            "max_concurrent_tasks": 3,
            "evidence_timeout_seconds": 90,
            "synthesis_timeout_seconds": 30,
        },
    }
)

The catalog callable must return metadata for already chat-exposed managers. Each entry requires domain, aliases, use_when, and distinguish_from. Catalog metadata improves local ranking only; it does not expose managers or change permissions:

def catalog():
    return {
        "PartManager": {
            "domain": "manufacturing",
            "aliases": ["part", "component"],
            "use_when": "The question concerns designed components.",
            "distinguish_from": ["MaterialManager"],
        },
    }

Run python manage.py check before enabling traffic. Use one non-production environment first; public requests must never select profiles or trust groups.

Add deterministic fake-provider cases for graph validation, manager resolution, round exhaustion, 90-second evidence and 30-second synthesis deadlines, calculation evidence, partial coverage, and every stable terminal reason. Run those cases together with the existing legacy WebSocket, SSE, and HTTP tests. Then enable GENERAL_MANAGER["CHAT"]["planned"]["enabled"] for the non-production environment and inspect the allowlisted audit events: role, match-source category, hashed canonical call identity, progress, budgets, latency, usage/cost, evidence counts, coverage, and terminal reason. Do not add raw results, profile names, trust groups, plans, hidden manager metadata, credentials, or provider exceptions to an audit sink.

Production rollout is an application-owned settings change and requires no migration. If an evaluation or operational check regresses, disable planned mode; the next request uses the compatible legacy strategy. Mutation requests already use that legacy safety path, including its authentication, mutation allow-listing, confirmation, persistence, and transport behavior. See the planned-chat cookbook for a client-visible event sequence and the Chat API reference for exact payload and error contracts.

6. Run deterministic planned orchestration evaluations

The shipped planned_orchestration dataset is intentionally unavailable to the legacy provider CLI. It is exercised by deterministic role-pinned fake providers in the test suite, not by a network provider. It covers alias resolution, a one-edge dependency, dynamic children, calculation, partial answers, budget and deadline exhaustion, duplicate calls, no-progress termination, and mutation fallback. Run it with the legacy eval regressions and sanitized-diagnostic checks:

python -m pytest \
  tests/unit/test_chat_planned_evals.py \
  tests/unit/test_chat_evals.py \
  tests/unit/test_chat_eval_diagnostics.py -q

These tests require a role override for every planned role and reject mixed trust groups. They assert deterministic fingerprints, aggregate provider usage, public coverage, and traces with profile and trust-group values removed. The legacy strategy remains a separate adapter, so existing _run_turn behavior and event shape remain covered independently.