Memdoor workflow matches Fable 5 on a small LiveCodeBench Hard test

The Memdoor repository publishes a team-solve workflow that recreates the authors’ multi-agent coding setup with five Qwen3.8-27B agents. On the first 10 problems in the authors’ LiveCodeBench Hard list, run five times, it reports an average of 9.8 correct solutions out of 10, compared with 9.6 for the authors’ harness and 8.0 for Fable 5 alone. The 50 workflow runs solved 49 problems and cost $35.79, with requests served through OpenRouter rather than a local model host. Agents begin with fresh contexts and exchange work through files, while executable tests and hidden tests determine whether solutions pass. The result is limited by the small sample, the fact that Fable 5 was not rerun in the same harness, and possible training-data contamination because the contests were held in March and April 2025.

Why it matters

The example is useful because it exposes a reproducible pattern for coding-agent coordination: separate roles, file-based state, and executable verification. However, the reported advantage should not be generalized from ten problems, and the comparison with Fable 5 is not fully controlled because that result came from the authors’ files rather than an identical Memdoor run.

Key points

  • The workflow averaged 9.8/10 over five passes on the first 10 problems in the authors’ LiveCodeBench Hard list.
  • It used five Qwen3.8-27B agents with planning, ideation, solving, management and repair roles, a 128K output cap, and up to 10 rounds.
  • Forty-nine of 50 runs solved their problems; the published cost was $35.79, excluding additional interrupted or cut-off attempts.
  • Agents received fresh contexts and communicated through files; executable grading, including hidden tests, determined success rather than model self-report.
  • The workflow is published for Memdoor and can use hosted connections such as OpenRouter or local runtimes; the repository describes Memdoor itself as a single-binary tool, while the example’s scripts

Based on reporting by The original announcement.