feat(benchmark): add hle-text-only #81
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds support for the HLE-text-only benchmark dataset by implementing data preparation, configuration, and documentation for evaluating text-only reasoning tasks. The implementation follows the existing pattern of other benchmark integrations in the codebase.
- Added HLE-text-only dataset preparation generator that loads and processes the text-only subset of HLE data
- Created benchmark configuration and agent configuration files for running evaluations with Claude 3.7 Sonnet
- Added comprehensive documentation with setup and usage instructions
Reviewed Changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| utils/prepare_benchmark/main.py | Added import and case handling for hle-text-only dataset preparation |
| utils/prepare_benchmark/gen_hle_text_only.py | New generator module that processes HLE text-only dataset from HuggingFace |
| scripts/run_prepare_benchmark.sh | Added command to prepare hle-text-only dataset |
| docs/mkdocs/docs/hle-text-only.md | Complete documentation for HLE-text-only benchmark usage |
| config/benchmark/hle-text-only.yaml | Benchmark configuration for HLE-text-only dataset |
| config/agent_hle-text-only_claude37sonnet.yaml | Agent configuration for running HLE-text-only with Claude 3.7 Sonnet |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Comment on lines
+29
to
+30
|
|
||
| return |
There was a problem hiding this comment.
The explicit return statement at the end of a generator function is unnecessary. Generator functions automatically return when they reach the end.
Suggested change
| return |
BinWang28
approved these changes
Oct 14, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Describe this PR
What changed?
Why?
Related issues
Checklist for PR
Write a descriptive PR title following the Angular commit message format:
<type>(<scope>): <subject>feat(agent): add pdf tool via mcp,perf: make llm client async,fix(utils): load custom config via importlibfeat,fix,docs,style,refactor,perf,test,build,ci,revertcheck-pr-titleCI job will validate your title formatUpdate README❌ Missing type and colonfeat add new feature❌ Missing colon after typeFeature: add new tool❌ Invalid type (should befeat)feat(Agent): add tool❌ Scope should be lowercasefeat(): add tool❌ Empty scope not allowedfeat(my_scope): add tool❌ Underscores not allowed in scopefeat(my space): add tool❌ Space not allowed in scopefeat(scope):add tool❌ Missing space after colonfeat(scope):❌ Empty subjectRun lint and format locally:
uv tool run [email protected] check --fix .uv tool run [email protected] format .lintenforces ruff default format/lint rules on all new codes.