Releases: open-compass/opencompass
Release list
0.5.3
OpenCompass v0.5.3 Release Notes
🌟 Highlights
✨ 🧪 Extensive New Benchmarks and Dataset Support: Added support for a wide range of new benchmarks and datasets, including AIME2026, HMMT Feb 2026, SimpleQA-Verified, AdvancedIF, HLE-Verified, MRCR-V2, Molecular-IQ, SciReasoner 1.5, MP20, PerspectiveGap, ZebraLogic, ArxivRollBench, CL-Bench, and more.
✨ 🧩 RawPromptTemplate Support: Introduced RawPromptTemplate and expanded its support across dataset configs, OpenAISDKStreaming, ChatML datasets, retrievers, and multiple benchmark configurations.
✨ 🤖 New Model & API Support: Added support for the OpenAI Responses API, Gemini SDK API and Claude SDK thinking content.
✨ 🛠️ Infrastructure & Evaluation Enhancements: Improved concurrent inference, repeat analysis, summarizer configuration, judge post-processing, CI coverage, and evaluation robustness.
🚀 New Features
🔧 Added support for new datasets and benchmarks, including AIME2026 and HMMT Feb 2026 (#2404), Molecular-IQ (#2431), SimpleQA-Verified (#2436), AdvancedIF (#2461), HLE-Verified (#2454), MRCR-V2 (#2467), S2-ToM-G Bench (#2476), SciReasoner 1.5 (#2477, #2479), MP20 (#2482), PerspectiveGap (#2484), ZebraLogic (#2464), and ArxivRollBench (#2458).
🔧 Introduced RawPromptTemplate and new dataset configs (#2407).
🔧 Added RawPromptTemplate support for OpenAISDKStreaming and ChatMLDatasets (#2414).
🔧 Added SciReasoner config and support for retrievers using RawPromptTemplate (#2422).
🔧 Added repeat configs for HMMT2025 and UGD_hard (#2425).
🔧 Added new subjective dataset configs and fixed AdvancedIF related configs (#2466).
🔧 Added RawPromptTemplate configs for AlignBench v1.1 (#2472), RULER, NeedleBench, LongBench v2, and BABILong (#2474).
🔧 Added support for repeat analysis (#2455).
🔧 Added OpenAI Responses API model support (#2481).
🔧 Added Gemini SDK API model support (#2494).
🔧 Added support for Claude SDK thinking content (#2487).
🔧 Added CL Bench support (#2483).
🐛 Bug Fixes
🔧 Fixed IFEval path issue (#2406).
🔧 Fixed tag matching in generic_llmjudge_postprocess (#2417).
🔧 Fixed issues in generic.py (#2419).
🔧 Fixed ChatML and dataset utility issues (#2421).
🔧 Fixed OlympiadBench compatibility with RawPromptTemplate (#2423).
🔧 Fixed ellipsis extraction in bio_data (#2429).
🔧 Fixed Molecular-IQ PR test issues (#2432).
🔧 Added multi-round support from api_meta_template (#2435).
🔧 Fixed result metrics in SimpleQA (#2439).
🔧 Fixed no-prediction cases in SciReasoner biology instruction evaluation (#2456).
🔧 Fixed hub version issue (#2457).
🔧 Fixed SimpleQA prompt (#2463).
🔧 Added unsafe pattern handling for MathEvaluator (#2468).
🔧 Fixed MTBench101, WildBench, and AdvancedIF issues (#2470).
🔧 Added subprocess timeout to MATHVerifyEvaluator to prevent parse and verify hangs (#2478).
🔧 Resolved dependency issues for Python 3.12 (#2480).
🔧 Restored LongBench v2 answer prompt punctuation (#2452).
🔧 Fixed MP20 metrics (#2488).
🔧 Prioritized Responses reasoning content (#2490).
⚙ Enhancements and Refactors
⚙ Evaluation and Runtime Improvements:
- Supported concurrent inference across tasks (#2403).
- Reduced RJOB sleep time (#2411).
- Added new template for CompassAcademic (#2447).
- Expanded config options of the summarizer group (#2448).
- Added
--no-progressparameter for repeat analysis to suppress detailed logs (#2473). - Added prediction postprocessor support for judge models in
GenericLLMEvaluator(#2491).
⚙ RawPromptTemplate Documentation and Compatibility:
- Added guide for RawPromptTemplate (#2420).
- Improved RawPromptTemplate coverage across benchmark and dataset configs (#2407, #2414, #2422, #2472, #2474).
⚙ Metadata and Release Updates:
⚙ CI/CD Improvements:
- Refactored ETE test cases and added more unit tests (#2408).
- Added inference test cases using mock API (#2428).
- Added API test cases into PR tests (#2460).
- Removed host network from CI settings (#2462).
⚙ Documentation:
- Fixed minor documentation bugs (#2469).
🎉 Welcome New Contributors
A warm welcome and special thanks to our newest contributors who made this release possible:
- @JifeiShan made their first contribution in (#2473).
- @ssiq made their first contribution in (#2480).
- @yhzhu99 made their first contribution in (#2452).
- @HuangZixian made their first contribution in (#2482).
- @WhymustIhaveaname made their first contribution in (#2484).
- @amanyara made their first contribution in (#2464).
- @liangzid made their first contribution in (#2458).
Full Changelog: 0.5.2...0.5.3
Thank you for using OpenCompass! These updates bring broader benchmark coverage, stronger API support, and more reliable evaluation workflows. Keep exploring, and stay tuned for future innovations! 🌟
Contributors
Assets 2
0.5.2
OpenCompass v0.5.2 Release Notes
🌟 Highlights
✨ 🧪 Extensive New Benchmarks Support: We have introduced comprehensive support for Scientific and General Benchmarks, including SciReasoner, Biology Instructions, Mol Instructions, CMPhysBench, IFBench, LCB-pro, etc.
✨ 🤖 New Model & API Support: Added support for Intern-S1-Pro and TeleChat API evaluation examples.
✨ 🛠️ Infrastructure & Enhancements: Fixed bugs, improved evaluation pipelines and updated CI.
🚀 New Features
🔧 Introduced support for HMMT2025 (#2305), AMO-Bench (#2305), IMO-Bench (#2305), ATLAS (#2297), OpenSWI (#2312), CMPhysBench (#2313), Biology Instructions (#2314), Mol Instructions (#2326), ARC_AGI_2 (#2330), IFBench (#2354), SciReasoner (#2360), PI-LLM (#2283), ProcessBench (#2274), and LCB_pro (#2361).
🔧 Supported monitoring of multi-dimensional evaluation metrics, including output length, logprobs, and finish reasons (#2351).
🔧 Added support for Intern-S1-Pro evaluation examples (#2394).
🔧 Added support for TeleChat API inference (#2371).
🔧 Added LLM-judge-based config for C-Eval (#2398).
🐛 Bug Fixes
🔧 Fixed OpenAISDKStreaming regarding output completeness and related issues (#2367, #2389, #2399).
🔧 Removed Pyext in runtime requirement (#2306).
🔧 Fixed pattern match in Smolinstruct (#2384).
🔧 Fixed buffer-related error in the LiveCodeBench evaluation (#2393).
⚙ Enhancements and Refactors
⚙ Infrastructure Refactors:
- Updated LCBench (#2166).
- Added headers as input param in BigCodeBench (#2302).
- Updated rjob with metadata name (#2316).
- Parametrized timeout in OpenAISDK (#2352).
- Added meta logger in OpenICLInferTask (#2383).
⚙ CI/CD Improvements:
- Refactored dailytest (#2308).
- Added CI for new datasets (#2358, #2369).
- Changed github runner (#2373).
- Added uni-test (#2390).
🎉 Welcome New Contributors
A warm welcome and special thanks to our newest contributors who made this release possible:
- @zhuangziGiantfish made their first contribution in (#2283).
- @xgao922 made their first contribution in (#2307).
- @Jensen246 made their first contribution in (#2310).
- @ccx06 made their first contribution in (#2371).
Full Changelog: 0.5.1.post1...0.5.2
Thank you for using OpenCompass! These updates empower deeper insights and more reliable evaluations. Keep exploring, and stay tuned for future innovations! 🌟
Contributors
Assets 2
0.5.1.post1
OpenCompass v0.5.1 Release Notes
🌟 Highlights
✨ A New Method to quickly Integrate and Evaluate Your Datasets: Added a fast dataset integration and evaluation method based on ChatML format, simplifying the previously complex dataset integration process.
✨ New Datasets: Integrated new benchmarks including SeedBench and BeyondAIME.
✨ Infrastructure & Enhancements: Fixed several bugs and updated CI.
🚀 New Features
🔧 Introduced a new approach for dataset integration and evaluation based on ChatML Template with evaluation examples (#2277).
🔧 Added SeedBench dataset (#2020).
🔧 Added BeyondAIME dataset (#2192).
🐛 Bug Fixes
🔧 Fixed Module Registers(#2262, #2266)
🔧 Fixed duplicate engine config update in TurboMindModelwithChatTemplate (#2276)
🔧 Fixed torchrun to avoid unexpected PATH environment (#2269)
🔧 Fixed the None return value case in cascade evaluator (#2211)
⚙ Enhancements and Refactors
⚙ Infrastructure Refactors:
- Update rjob.py and subjective_eval.py (#2263)
⚙ CI/CD Improvements:
🎉 Welcome New Contributors
A warm welcome and special thanks to our newest contributors who made this release possible:
- @ChenZiHong-Gavin for their first contribution in (#2020)
- @jibuzixin for their first contribution in (#2211)
- @JYLiAo32 for their first contribution in (#2276)
- @ShoupingShan for their first contribution in (#2262)
Full Changelog: 0.5.0...0.5.1.post1
Thank you for using OpenCompass! These updates empower deeper insights and more reliable evaluations. Keep exploring, and stay tuned for future innovations! 🌟
Contributors
Assets 2
0.5.0
OpenCompass v0.5.0 Release Notes
🌟 Highlights
✨ Comprehensive Scientific Benchmarks: Integrated 10+ specialized datasets (MedXpertQA, ClimaQA, SmolInstruct, etc.), covering multiple scientific fields such as chemistry, physics, biology, and earth sciences
✨ Cascade Evaluator: Supported cascading eval methods from rules to LLM judgments.
✨ New Runner: Supported for Rjob Runner has now been completed.
✨ OpenAISDK Streaming: Provided a more stable OpenAI API method.
✨ New Evaluation Examples: Published the real-time evaluation config of CompassAcademic Leaderboard and the Intern-S1 related benchmark evaluation config.
🚀 New Features
🔧 Cascade Evaluator (#1992)
🔧 Rjob Runner (#2144)
🔧 OpenAISDK Streaming (#2208)
🔧 Evaluation Example for CompassAcademic Leaderboard. (#2202)
🔧 Evaluation Example for Intern-S1 and Scientific Benchmarks. (#2220)
🔧 So Many New Scientific Datasets!
- MedXpertQA for expert-level medical knowledge evaluation (#2002)
- ClimaQA for climate question evaluation (#2017)
- HealthBench for better measuring capabilities of AI systems for health (#2099)
- ProteinLMBench for protein related tasks (#2064)
...
📖 Documentation
📝 Fixed 404 links between Chinese/English docs (#2001)
📝 Added CompassAcademic Leaderboard task tutorial (#2202)
📝 Added Intern-S1 evaluation task tutorial (#2220)
📝 Fixed format problems of the dataset statistics page (#2170)
📝 Align NIAH CLI command guide to the actual CLI argument parser (#2194)
📝 Set correct paths for the examples (#2198)
🐛Bug Fixes
🔧 Fixed compare error base_evaluator (#2010)
🔧 Fixed OpenICL Math Evaluator Config (#2007)
🔧 Added Error Case for content filter (#2167)
🔧 Fix the OpenAI SDK to adapt to gpt-5 (#2236)
🔧 Fixed dataset repeat by concatenating (#2039)
🔧 Concat OpenaiSDK reasoning content (#2041)
⚙ Enhancements and Refactors
⚙ Infrastructure Refactors:
- Set dump-eval-details as default behavior (#1999)
- Refactorized openicl eval task (#1990)
- Added openai_extra_kwargs for API customization (#2210)
⚙ CI/CD Improvements:
- Fixed baseline score (#2000)
- Updated baseline for kernal change of vllm and lmdeploy (#2011)
- Updated baseline and fix lmdeploy version (#2098)
- Added check rule (#2101)
- Updated testcases' baseline (#2184)
...
🎉 Welcome New Contributors
A warm welcome to our newest contributors:
- @Yejin0111 for MedXpertQA clinical dataset (#2002)
- @smgjch for matbench development (#2021)
- @taolinzhang for rewardbench dataset (#2029)
- @xiexinch for fixing lawbench evaluation (#2037)
- @xuxuxuxuxuxjh for ClinicBench, PubMedQA and ScienceQA datasets (#2061)
- @mar-cry for NEJM AI benchmark (#2063)
- @bio-mlhui for CARDBiomedBench dataset (#2071)
- @tchenglv520 for Lifescience subset support for MMLU & SciEval (#2059)
- @Flaick for MMLU Pro Biomedical version support (#2081)
- @yuehua-s for o4-mini model (#2083)
- @kkscilife for adding CI check rule (#2101)
- @yusun-nlp for SmolInstruct dataset (#2127)
- @soki123 for SRbench dataset (#2105)
- @suencgo for PHYBench dataset (#2125)
- @Zhouzone for updating Earth Silver benchmark (#2140)
- @uyzhang for R-Bench dataset (ICML 2025) (#2091)
- @f14-bertolotti for stabilizing MBPP evaluation (#2111)
- @fly2tomato for debugging Rjob runner (#2171)
- @debuggingworld for fixing Qwen3 model config field error (#2152)
- @blueternalness for aligning NIAH CLI command guide (#2194)
- @KADCA21 for BlueLM-2.5 API (#2193)
- @dbinthesky for KCLE feature (#2224)
- @FarongWen for EESE dataset and configs (#2223)
- @mazihan880 for CodeCompass dataset and configs (#2214)
Full Changelog: 0.4.2...0.5.0
Thank you for using OpenCompass! These updates empower deeper insights and more reliable evaluations. Keep exploring, and stay tuned for future innovations! 🌟
Contributors
Assets 2
0.4.2
OpenCompass v0.4.2
The OpenCompass team is thrilled to announce the release of OpenCompass v0.4.2! This version introduces powerful new datasets, robust evaluation enhancements, and critical refinements to elevate your benchmarking workflows. Let’s explore the key updates!
🌟 Highlights
✨ SuperGPQA Benchmark: Added support for the SuperGPQA dataset, enabling advanced reasoning evaluation with refined subset metrics. (#1924, #1966)
✨ MultiPL-E & Code Evaluator: Introduced MultiPL-E dataset integration and a dedicated code evaluator for comprehensive programming task assessments. (#1963)
✨ OlympiadMath Dataset: Expanded mathematical reasoning benchmarks with the OlympiadMath dataset. (#1982)
🚀 New Features
🔧 MMLU Pro Evaluation: Implemented generic LLM evaluation for MMLU Pro, broadening assessment flexibility. (#1923)
🔧 LLM Judge Configuration: Enhanced LLM judge capabilities with updated dataset configurations. (#1940)
🔧 Model Configurations:
- Added Intervl-8B and Intervl-38B model configurations. (#1978)
- Introduced Qwen-72B model support. (#1959)
📖 Documentation
📝 Results Persistence Guide: Documented evaluation results persistence workflows for better reproducibility. (#1908)
📝 Dataset Statistics: Added TBD token clarifications in dataset statistics documentation. (#1986)
📝 Typo Fixes: Corrected typos in DeepSeed-R1 documentation. (#1916)
🐛 Bug Fixes
🔧 Math Verification: Fixed math-verify evaluator logic to ensure accurate mathematical reasoning checks. (#1917)
🔧 Summarizer Logic: Resolved summarizer inconsistencies for reliable result aggregation. (#1953)
🔧 Model Compatibility:
- Fixed
model_kwargshandling for vLLM accelerator compatibility. (#1958) - Patched AIME-2024 configuration errors. (#1974)
- Addressed OpenAI model tokenization constraints. (#1960)
⚙ Enhancements and Refactors
⚙ Dataset Optimization:
- Refined dataset configurations for KorBench, LiveMathBench, and others. (#1937, #1967)
- Added no
max_out_lenconfigurations for flexible evaluations. (#1968)
⚙ Infrastructure Upgrades:
- Increased memory allocation for VOLC Runner CPU jobs. (#1962)
- Updated OlympiadBench and LLM Judge pipelines. (#1954)
⚙ CI/CD Improvements:
- Fixed baseline scoring in daily tests for consistent benchmarking. (#1996)
🎉 Welcome New Contributors
🎊 A warm welcome to our newest contributors:
- @kangreen0210 for adding SuperGPQA support in #1924
- @Jiajun0425 for resolving vLLM accelerator issues in #1958
Full Changelog: 0.4.1...0.4.2
Thank you for using OpenCompass! These updates empower deeper insights and more reliable evaluations. Keep exploring, and stay tuned for future innovations! 🌟
Contributors
Assets 2
0.4.1
OpenCompass v0.4.1 Release Notes
The OpenCompass team is thrilled to announce the release of OpenCompass v0.4.1! This version brings exciting new features, enhancements, and bug fixes to improve your experience. Let’s dive into the highlights and changes!
🌟 Highlights
✨ Omni-Math Support: OpenCompass now supports the Omni-Math dataset, expanding its capabilities in mathematical reasoning. (#1837)
✨ OlympiadBench Benchmark: Added support for the OlympiadBench benchmark, enabling evaluation on advanced problem-solving tasks. (#1841)
✨ HLE Dataset: Introduced the Humanity's Last Exam (HLE) dataset, providing a challenging new benchmark for AI models. (#1902)
🚀 New Features
🔧 Math Verification: Added model post-processor for math verification, enhancing accuracy in mathematical evaluations. (#1881)
🔧 Dataset Repeat & G-Pass Compute: Support for dataset repetition and G-Pass computation for each evaluator, improving flexibility in evaluations. (#1886)
🔧 General Math & LLM Judge Evaluator: Added a general math evaluator and LLM judge evaluator for broader evaluation scenarios. (#1892)
🔧 AIME-24 Evaluation: Supported AIME-24 evaluation for DeepSeek-R1 series, enhancing competition-level benchmarking. (#1888)
📖 Documentation
📝 Broken Links Fix: Fixed broken links in README.md to ensure a smoother onboarding experience. (#1852)
📝 Supported Datasets List: Added a list of supported datasets on the HTML page for better accessibility. (#1850)
🐛 Bug Fixes
🔧 Compatibility Issue: Fixed a compatibility issue to ensure smoother integration with existing workflows. (#1904)
🔧 Hard Configs Fix: Updated hard configs with General GPassK for improved evaluation consistency. (#1906)
⚙ Enhancements and Refactors
⚙ Daily Test Updates: Updated daily test scores and scheduler for better CI/CD pipeline reliability. (#1854, #1898)
⚙ Benchmark Updates: Updated BigCodeBench, LCBench, and LiveMathBench configurations for enhanced evaluation accuracy. (#1857, #1862, #1826)
⚙ OpenAI Model Update: Updated OpenAI models and BigCodeBench configurations for better performance. (#1879)
🎉 Welcome New Contributors
🎊 A warm welcome to our new contributors:
- @sudanl for their first contribution in #1841
- @Pablohn26 for their first contribution in #1852
- @Zhudongsheng75 for their first contribution in #1857
Full Changelog: 0.4.0...0.4.1
Thank you for using OpenCompass! We’re excited to see how you leverage these new features and improvements. Stay tuned for more updates! 🚀
Contributors
Assets 2
0.4.0
The OpenCompass team is thrilled to announce the release of OpenCompass v0.4.0!
🌟 Highlights
✨ This version introduces several new features and improvements that enhance the user experience and expand the capabilities of OpenCompass. Notable changes include support for Longbenchv2, compatibility with InternLM3 models and also refactorization of Openai model class. We are also consolidating all AMOTIC configuration files (previously located in ./configs/datasets, ./configs/models, and ./configs/summarizers) into the opencompass package.
🚀 New Features
-🆕 Support for Longbenchv2 has been added to provide more comprehensive model evaluation. (#1801)
-🆕 The Bradley-Terry Subjective Evaluation method has been extended to the Arena Hard dataset. (#1802)
-🆕 Predicted Win Probabilities have been added to CompassArenaBradleyTerrySummarizer for better insights into model performance. (#1815)
-🆕 Support for InternLM3 models is now available, expanding our model library. (#1829)
-🆕 MMLU-CF Benchmark support has been introduced to further enrich our suite of benchmarks. (#1775)
📖 Documentation
-📚 Documentation on adding new datasets (new_dataset.md) has been updated for clarity. (#1827)
-📚 The installation guide (installation.md) has been revised to reflect the latest setup process. (#1830)
🐛 Bug Fixes
-🔧 A path conflict within the CI pipeline has been resolved. (#1814)
-🔧 The logic for max_out_len in OpenAI models has been corrected. (#1839)
⚙ Enhancements and Refactors
-💪 Code refactoring has been performed to improve project structure and maintainability. (#1831)
-💪 The threshold for CI checks has been updated for more stringent quality control. (#1812)
-💪 LiveMathBench has been updated to ensure it remains a cutting-edge benchmarking tool. (#1809)
🎉 Welcome New Contributors
@fistyee contributed support for the MMLU-CF Benchmark. (#1775)
@Myhs-phz improved documentation on adding new datasets. (#1827)
@thejishnunair updated the installation guide. (#1830)
Thank you for being part of the OpenCompass community! Your support and contributions make each release possible. We look forward to your feedback on this release.
Contributors
Assets 2
0.3.9
The OpenCompass team is thrilled to announce the release of OpenCompass v0.3.9!
🌟 Highlights
✨ This version introduces a number of new features and improvements that enhance the user experience and expand the capabilities of OpenCompass. Notable changes include support for G-Pass@k and LiveMathBench, as well as the introduction of the Bradley-Terry subjective evaluation method.
🚀 New Features
-🆕 Support for G-Pass@k and LiveMathBench metrics to better evaluate model performance. (#1772)
-🆕 Theorem QA 0shot CoT configuration has been added for more comprehensive evaluation scenarios. (#1783)
-🆕 A customizable tokenizer for RULER offers greater flexibility in processing inputs. (#1731)
-🆕 Added LiveStemBench Dataset to enrich our collection of datasets. (#1794)
-🆕 Integration of JudgeLLM into o1 evaluation for improved assessment accuracy. (#1795)
-🆕 Implementation of the Bradley-Terry subjective evaluation method on wildbench, alpacaeval, and compassarena datasets. (#1791)
📖 Documentation
-📚 Updated OC academic content to the most recent information as of December 2024. (#1771)
🐛 Bug Fixes
-🔧 Fixed Order error which was causing issues with sequence handling. (#1767)
-🔧 Resolved an issue where the lark report was returning None. (#1769)
-🔧 Corrected the path for saving Local Runner parameters. (#1768)
-🔧 Amended the summarizer abbreviation for models to ensure proper identification. (#1789)
-🔧 Fixed output_path errors to improve file handling reliability. (#1798)
⚙ Enhancements and Refactors
-💪 Fullbench testcase has been integrated into the CI pipeline. (#1766)
Volc status exception handling has been updated for more robust responses. (#1780)
-💪 Removed daily step retry mechanism and updated PR score calculation for efficiency. (#1782)
-💪Deploy Python version has been updated to the latest stable release. (#1784)
-💪Pypi deploy workflow has been refined for smoother deployments. (#1786)
Thank you for being part of the OpenCompass community! Your support and contributions make each release possible.
0.3.8
The OpenCompass team is thrilled to announce the release of OpenCompass v0.3.8!
🎉 Added Features:
-🆕 DLC runner Lark report functionality has been introduced to improve reporting mechanisms. ( #1735 )
-🆕 Chinese SimpleQA configuration has been added to extend language support. ( #1697 )
-🆕 Addition of OC academic 2412 evaluation example ( #1750 )
📖 Documentation
-📚 Supplemented KOR-BENCH readme documentation for better user guidance. ( #1734 )
-📚 Updated README files for various datasets including Korbench to ensure up-to-date information. ( #1737 )
🐛 Bug Fixes
-🔧 Corrected an error in the subjective default summarizer for more accurate results. ( #1740 )
-🔧 Adjusted the max_out_len parameter for ChineseSimpleQA to fix related issues. ( #1757 )
-🔧 Resolved a problem with the transfer of the vllm max_seq_len parameter for consistent behavior. ( #1745 )
⚙ Enhancements and Refactors
-💪 Updated Manifest file to reflect the latest changes in the project. ( #1738 )
-💪 Modified Compassarena metric for enhanced performance measurement. ( #1749 )
-💪 Improved dataset configurations by removing max_out_len where not applicable. ( #1754 )
-💪 Upgraded requirement and deepseek configurations for better compatibility. ( #1764 )
🎉 Welcome New Contributors
We are pleased to welcome @OpenStellarTeam, who contributed to the addition of Chinese SimpleQA config. ( #1697 )
For a comprehensive overview of all changes, please refer to the Full Changelog.
Thank you for being part of the OpenCompass community! Your support and contributions make each release possible.
Contributors
Assets 2
0.3.7
The OpenCompass team is thrilled to announce the release of OpenCompass v0.3.7!
🚀 New Features
- 🆕 Added support for new error code handling, improving system resilience. (#1702)
- 🆕 Introduced the P-MMEval feature for advanced model evaluations. (#1714)
- 🆕 Added LiveMathBench support for dynamic mathematical benchmarking. (#1727)
- 🆕 Included the Openai Simpleqa dataset to expand our question answering capabilities. (#1720)
📖 Documentation
- 📚 Updated configurations and documentation to reflect the latest changes, ensuring a smooth user experience. (#1704, #1717)
🐛 Bug Fixes
- 🔧 Resolved issues in output sequence generation under Turbomind model. (#1707)
- 🔧 Corrected configuration errors in pmmeval_gen to ensure proper functionality. (#1719)
⚙ Enhancements and Refactors
- 💪 Enhanced support for Arc Prize Public Evaluation. (#1690)
- 💪 Increased max_out_len parameters for various datasets to accommodate longer sequences. (#1726)
- 💪 Incorporated Korbench and updated Fullbench to provide more comprehensive benchmarking options. (#1713, #1712)
- 💪 Streamlined CI pipeline by updating the torch version and adding more datasets into daily test cases. (#1701)
🎉 Welcome New Contributors
- 👏 A warm welcome to @epsilondylan and @wanyu2018umac, who have made their first contributions by adding the Korbench dataset and introducing the P-MMEval feature respectively! (#1713, #1714)
For a complete overview of all changes, please refer to the full changelog: 0.3.6...0.3.7
