agent_framework_evaluator.initializer_catalog

Module API

Discover and resolve agent-eval initializer modules (setup + callbacks, prompt defaults).

agent_framework_evaluator.initializer_catalog.resolve_env_path(env_path)

Resolve a user-supplied .env path the same way on server and client expectations.

Relative paths are resolved against the current working directory (the process that runs Uvicorn). Use an absolute env_path in the UI if the server cwd is not your project root.

Parameters:

env_path (str | Path)

Return type:

Path

agent_framework_evaluator.initializer_catalog.evaluator_initializer_root(env_file)

Return AGENT_EVAL_INITIALIZER_DIR from env_file, or None.

Parameters:

env_file (Path)

Return type:

Path | None

agent_framework_evaluator.initializer_catalog.list_initializer_scripts(env_file)

Relative paths (posix) of *.py files under the configured initializer directory.

Parameters:

env_file (Path)

Return type:

list[str]

agent_framework_evaluator.initializer_catalog.resolve_initializer_path(env_file, initializer_ref)

Resolve initializer_ref to a readable .py under the initializer root.

Parameters:

• env_file (Path)

• initializer_ref (str)

Return type:

Path | None

agent_framework_evaluator.initializer_catalog.resolve_setup_path_for_run(env_file, ref)

Resolve UI/CLI initializer field to a setup_path for SessionRunner.

Order: path under AGENT_EVAL_INITIALIZER_DIR, then absolute .py, then .py relative to cwd, then a unique basename match under the initializer tree (so deck-review.py finds …/scripts/eval/deck-review.py when unambiguous).

Parameters:

• env_file (Path)

• ref (str | None)

Return type:

Path | None

agent_framework_evaluator.initializer_catalog.load_initializer_default_prompt(env_file, initializer_ref)

Load initializer/setup module and return its default prompt text.

Parameters:

• env_file (Path)

• initializer_ref (str)

Return type:

str

agent_framework_evaluator.initializer_catalog.load_initializer_default_evaluator_criteria(env_file, initializer_ref)

Load initializer/setup module and return default evaluator criteria text, if any.

Parameters:

• env_file (Path)

• initializer_ref (str)

Return type:

str

agent_framework_evaluator.initializer_catalog.load_initializer_default_agent(env_file, initializer_ref)

Load initializer/setup module and return default agent id, if any.

Parameters:

• env_file (Path)

• initializer_ref (str)

Return type:

str

agent_framework_evaluator.initializer_catalog.load_initializer_default_eval_model(env_file, initializer_ref)

Return preferred evaluator model(s) from DEFAULT_EVAL_MODEL / get_default_eval_model().

Parameters:

• env_file (Path)

• initializer_ref (str)

Return type:

str

agent_framework_evaluator.initializer_catalog.load_initializer_default_agent_model_override(env_file, initializer_ref)

Return preferred agent-under-test model override from initializer defaults.

Parameters:

• env_file (Path)

• initializer_ref (str)

Return type:

str

agent_framework_evaluator.initializer_catalog.load_initializer_default_agent_model_override_scope(env_file, initializer_ref)

Return override scope for the agent-under-test model, if any.

Parameters:

• env_file (Path)

• initializer_ref (str)

Return type:

str

agent_framework_evaluator.initializer_catalog.load_raw_test_cases(env_file, initializer_ref)

Load test case dicts from initializer (includes code_evaluator callables when present).

Parameters:

• env_file (Path)

• initializer_ref (str)

Return type:

list[dict[str, Any]]

agent_framework_evaluator.initializer_catalog.serialize_test_cases(cases)

API-safe rows (no callables).

Parameters:

cases (list[dict[str, Any]])

Return type:

list[dict[str, Any]]

agent_framework_evaluator.initializer_catalog.load_test_cases(env_file, initializer_ref)

Serializable test cases for GET /api/initializer-cases.

Parameters:

• env_file (Path)

• initializer_ref (str)

Return type:

list[dict[str, Any]]