agent_framework_evaluator.case_markdown#

Classes#

Module API#

Load evaluator test cases from markdown files (----separated frontmatter, prompt, criteria).

Use MarkdownCaseLoader from an initializer’s get_test_cases() so cases live in *.md files next to the initializer module, e.g. eval/deck-review-01.md with glob eval/*.md relative to deck-review.py.

agent_framework_evaluator.case_markdown.parse_simple_frontmatter(text)[source]#

Parse key: value lines (no nesting). For nested YAML use yaml.safe_load on the block.

Parameters:

text (str)

Return type:

dict[str, str]

agent_framework_evaluator.case_markdown.parse_case_markdown_file(*, path, evaluator_registry, resolver=None)[source]#

Parse one case file; return case metadata, prompt, criteria, and evaluator hooks.

Parameters:

  • path (Path)

  • evaluator_registry (Mapping[str, Callable[[...], Any]])

  • resolver (FileReferenceResolver | None)

Return type:

dict[str, Any] | None