Update evaluate.py

[Keerthykaushik-Dasoju]

In the method initialization, you are expecting model_pair but while using this method you are sending only strong model which is throwing a runtime error while running the code. After making this change in my local it was working fine. Please check this from your end as well

Thank you
Keerthy Kaushik

[exit0-run]

Confirming this fix, with a reproduction, because the bug is worse than a one-line typo suggests: it means the documented evaluation command has not printed its metrics table for anyone since this was filed.

At HEAD (0b64fda), evals/evaluate.py:266-272 passes controller.model_pair.strong into generate_results, whose fourth parameter is named routed_pair and is immediately used as benchmark.get_model_accuracy(routed_pair.weak). ModelPair is a dataclass, so .strong is a str:

AttributeError: 'str' object has no attribute 'weak'

The run does all the router inference first, then dies before printing the 20/50/80% qual, AUC and APGR lines and before writing the plot. So the CPT and APGR figures in Tables 1 to 3 of the paper cannot be produced by the shipped command without this PR applied. With --routers random there is no numeric output at all.

While confirming it I recomputed the README's headline offline, which needs no router and no key, from three files already in the repo:

MT Bench, 72 questions after decontamination
  gpt-4-1106-preview             9.211806
  mistralai/Mixtral-8x7B-...     8.281250
  CPT(50%) = 8.746528  =  94.95% of GPT-4

That 94.95% is where "maintaining 95% GPT-4 performance" comes from. Same recipe gives MMLU 80.587 / 68.091 and GSM8K 85.769 / 63.734, which round to the "(81)" and "(86)" in the Table 2 and Table 3 captions. It is pandas over evals/mt_bench/{judgements,question,contaminated_prompts}.jsonl following benchmarks.py::get_model_accuracy, no network.

One thing I could not find and would rather ask than assume: the cost reduction percentages (85% on MT Bench, 45% MMLU, 35% GSM8K) have no code path in this repository. There is no pricing table and no per-token constant anywhere in the package, and the harness emits CPT, AUC and APGR only. If those came from a separate calculation it would be useful to have it written down, since it is the half of the headline that cannot be checked from here.

Not asking for anything beyond the merge. This is a widely cited repository and the one-line fix has been sitting here since March 2025.