论文实验结果复现

[HH446]

请问训练集构建的记忆数据是经过多少轮迭代形成的呢？直接利用run_appworld.py可以构建出相关记忆数据吗？

[chenrutao]

你好，请问论文结果复现出来了吗，为啥我跑benchmark/appworld/run_appworld.py，用run_exp_statistic.py执行结果是这样的，有reme比无reme效果还差，task_ids只截取了16个去执行，我的配置是（ max_workers = 4， batch_size = 2，有reme则use_memory=True，无reme则use_memory=False， 其他配置未修改）：

(aw-env) root@ecs-7651:/opt/ReMe/benchmark/appworld# python run_exp_statistic.py 
2026-04-30 17:43:52.784 | INFO     | __main__:run_exp_statistic:104 - File exp_result/qwen3-8b/test_normal_no-reme-memory-202604301454.jsonl: 16 tasks, 2 runs per task
2026-04-30 17:43:52.785 | INFO     | __main__:run_exp_statistic:108 - Calculable best@k values: [2, 1]
2026-04-30 17:43:52.785 | INFO     | __main__:run_exp_statistic:129 - file=test_normal_no-reme-memory-202604301454.jsonl best@2=0.6415
2026-04-30 17:43:52.785 | INFO     | __main__:run_exp_statistic:134 - file=test_normal_no-reme-memory-202604301454.jsonl pass@2=0.1250
2026-04-30 17:43:52.785 | INFO     | __main__:run_exp_statistic:129 - file=test_normal_no-reme-memory-202604301454.jsonl best@1=0.5192
2026-04-30 17:43:52.785 | INFO     | __main__:run_exp_statistic:134 - file=test_normal_no-reme-memory-202604301454.jsonl pass@1=0.1250
2026-04-30 17:43:52.795 | INFO     | __main__:run_exp_statistic:104 - File exp_result/qwen3-8b/test_normal_with-fixed-memory-202604301635.jsonl: 16 tasks, 4 runs per task
2026-04-30 17:43:52.795 | INFO     | __main__:run_exp_statistic:108 - Calculable best@k values: [4, 2, 1]
2026-04-30 17:43:52.795 | INFO     | __main__:run_exp_statistic:129 - file=test_normal_with-fixed-memory-202604301635.jsonl best@4=0.5768
2026-04-30 17:43:52.795 | INFO     | __main__:run_exp_statistic:134 - file=test_normal_with-fixed-memory-202604301635.jsonl pass@4=0.0625
2026-04-30 17:43:52.795 | INFO     | __main__:run_exp_statistic:129 - file=test_normal_with-fixed-memory-202604301635.jsonl best@2=0.4993
2026-04-30 17:43:52.795 | INFO     | __main__:run_exp_statistic:134 - file=test_normal_with-fixed-memory-202604301635.jsonl pass@2=0.0625
2026-04-30 17:43:52.795 | INFO     | __main__:run_exp_statistic:129 - file=test_normal_with-fixed-memory-202604301635.jsonl best@1=0.4083
2026-04-30 17:43:52.795 | INFO     | __main__:run_exp_statistic:134 - file=test_normal_with-fixed-memory-202604301635.jsonl pass@1=0.0469
2026-04-30 17:43:52.805 | INFO     | __main__:run_exp_statistic:104 - File exp_result/qwen3-8b/test_normal_no-reme-memory-202604301745.jsonl: 16 tasks, 4 runs per task
2026-04-30 17:43:52.805 | INFO     | __main__:run_exp_statistic:108 - Calculable best@k values: [4, 2, 1]
2026-04-30 17:43:52.805 | INFO     | __main__:run_exp_statistic:129 - file=test_normal_no-reme-memory-202604301745.jsonl best@4=0.6960
2026-04-30 17:43:52.805 | INFO     | __main__:run_exp_statistic:134 - file=test_normal_no-reme-memory-202604301745.jsonl pass@4=0.2500
2026-04-30 17:43:52.805 | INFO     | __main__:run_exp_statistic:129 - file=test_normal_no-reme-memory-202604301745.jsonl best@2=0.5917
2026-04-30 17:43:52.805 | INFO     | __main__:run_exp_statistic:134 - file=test_normal_no-reme-memory-202604301745.jsonl pass@2=0.1875
2026-04-30 17:43:52.805 | INFO     | __main__:run_exp_statistic:129 - file=test_normal_no-reme-memory-202604301745.jsonl best@1=0.4935
2026-04-30 17:43:52.805 | INFO     | __main__:run_exp_statistic:134 - file=test_normal_no-reme-memory-202604301745.jsonl pass@1=0.1406
2026-04-30 17:43:52.810 | INFO     | __main__:run_exp_statistic:104 - File exp_result/qwen3-8b/test_normal_with-fixed-memory-202604301231.jsonl: 16 tasks, 2 runs per task
2026-04-30 17:43:52.810 | INFO     | __main__:run_exp_statistic:108 - Calculable best@k values: [2, 1]
2026-04-30 17:43:52.810 | INFO     | __main__:run_exp_statistic:129 - file=test_normal_with-fixed-memory-202604301231.jsonl best@2=0.6045
2026-04-30 17:43:52.810 | INFO     | __main__:run_exp_statistic:134 - file=test_normal_with-fixed-memory-202604301231.jsonl pass@2=0.1250
2026-04-30 17:43:52.811 | INFO     | __main__:run_exp_statistic:129 - file=test_normal_with-fixed-memory-202604301231.jsonl best@1=0.5022
2026-04-30 17:43:52.811 | INFO     | __main__:run_exp_statistic:134 - file=test_normal_with-fixed-memory-202604301231.jsonl pass@1=0.0625

================================================================================
Experiment Results Summary Table
================================================================================
                                                  pass@1  pass@2  pass@4
file                                                                    
test_normal_no-reme-memory-202604301454.jsonl     0.1250  0.1250     NaN
test_normal_with-fixed-memory-202604301635.jsonl  0.0469  0.0625  0.0625
test_normal_no-reme-memory-202604301745.jsonl     0.1406  0.1875  0.2500
test_normal_with-fixed-memory-202604301231.jsonl  0.0625  0.1250     NaN
================================================================================
2026-04-30 17:43:52.819 | INFO     | __main__:run_exp_statistic:158 - Results table saved to: exp_result/qwen3-8b/experiment_summary.csv
(aw-env) root@ecs-7651:/opt/ReMe/benchmark/appworld#

[chenrutao]

@jinliyl 您好，请问可以帮忙解答一下吗，为啥我跑的结果是这样的，有reme比无reme效果还差一些

[linw-z]

@chenrutao 你好，请问你目前复现出来论文中数据集（appworld）标注的结果了吗？

[jinliyl]

@chenrutao @HH446 Thank you very much for your feedback. Regarding the reproduction of this experiment, please reach out to Yingcao Zou at https://github.com/zouyingcao.
She has already left our team and only responds to questions occasionally. If you're in urgent need of assistance, feel free to contact her directly through her GitHub.

[zouyingcao]

请问训练集构建的记忆数据是经过多少轮迭代形成的呢？直接利用run_appworld.py可以构建出相关记忆数据吗？

@HH446 关于memory pool的初始化，我们针对训练集的每一条query都各run了8条轨迹，可以参考bfcl-quickstart.md-init the task memory pool from scratch中的步骤同理构建appworld的初始记忆数据

[zouyingcao]

@chenrutao 您好，方便提供完整的实验配置吗，以及是否有跑完完整的test_normal结果数据，可以对照一些开源工作[1,2]中no_memory_baseline的结果，首先可能需要确保环境无误能够成功复现baseline，有后续可及时交流～
[1] https://arxiv.org/pdf/2512.01311v2, Table 1: Qwen3 8B avg@8=17.76%
[2] https://arxiv.org/pdf/2604.04804, Table 1: Qwen3-32B avg@4=27.68%