Spaces:
Sleeping
Sleeping
Commit History
-
Remove unused AstaBench category files and update UI to OpenHands categories
openhandsCleanup actions: - Deleted 4 unused AstaBench category files: c_and_e.py, data_analysis.py, e2e.py, literature_understanding.py - Updated ui_components.py tooltips to show OpenHands categories (Bug Fixing, Frontend Development, etc.) instead of AstaBench categories - Changed tooltip from 'four categories' to 'five categories' - Updated leaderboard_transformer.py comment to reflect actual OpenHands field mappings - Category list now: Bug Fixing, Frontend Development, App Creation, Test Generation, Information Gathering This removes ~1KB of dead code and fixes user-facing tooltips to match actual categories.6a0d1cb -
Add Acknowledgements section crediting AstaBench
openhands- Added new Acknowledgements section to About page - Credits AstaBench leaderboard (Allen Institute for AI) for UI design and methodology - Lists key aspects adapted: macro-averaging, visualization, UI structure - Positioned before Citation section737a3f2 -
Fix score calculation to match AstaBench methodology and update categories
openhandsCritical fixes: - Changed overall score calculation from simple average of all benchmarks to macro-average of category averages - This ensures each category contributes equally regardless of benchmark count - Updated Categories Attempted from 4 to 5 categories (Bug Fixing, Frontend Dev, App Creation, Test Gen, Info Gathering) - Updated category list in leaderboard_transformer.py line 281 - Updated tooltip text to reflect 5 categories This matches the AstaBench calculation methodology.e734bf6 -
Fix agent_version display and make Overall Score bold
openhands- Fixed case mismatch in simple_data_loader.py: 'openhands version' -> 'Openhands version' - Fixed case for 'Language model' -> 'Language model' to match transformer expectations - Modified format_score_column to make Overall Score values bold - Overall Cost remains non-bold as requested0e14c25 -
Swap column order and fix duplicate column warnings
openhands +1: openhands- Swapped Language Model and OpenHands Version column order - Fixed duplicate column warning by renaming columns before getting headers - Updated mock data with proper agent_version values (1.0.1, 1.0.2) - Removed duplicate agent_version keys from metadata.json files Co-authored-by: openhands <openhands@all-hands.dev>3781804 -
Rename columns: Agent→OpenHands Version, Models Used→Language Model, remove Submitter
openhands +1: openhands- Updated simple_data_loader.py to use new column names - Updated leaderboard_transformer.py with fixed_mappings and scatter plot references - Updated ui_components.py tooltips and column handling - Changed metadata.json: agent_name→agent_version across all mock data - Added 'Bug Fixing' tag to swe-bench benchmarks in scores.json - Removed 'Submitter' column from base_cols and column references Co-authored-by: openhands <openhands@all-hands.dev>376500e -
Cleanup codebase: remove unused code, simplify data loading, and add pre-release notice
openhands +1: openhands- Remove unused github_data_loader.py (71 lines of dead code) - Clean up debug logging in app.py (replaced ~40 print statements with proper logger) - Remove scheduler code and APScheduler dependency from requirements.txt - Simplify simple_data_loader.py by removing old JSONL format fallback - Update setup_data.py to fetch data from results/ directory only - Update mock data to use new agent-centric directory structure - Add pre-release notice to main page intro paragraph Co-authored-by: openhands <openhands@all-hands.dev>855423e -
Force rebuild: Update comment
openhands64c8899 -
Fix missing get_combined_icon_html reference in main page
openhands- Fixed remaining reference to removed get_combined_icon_html function - Now using get_openness_icon_html inline function instead - Tested locally and app runs successfully without errorsae0bf64 -
Simplify leaderboard to open vs closed models only
openhandsRemove agent tooling distinction from the leaderboard: - Simplified openness from 4 categories to 2: 'open' and 'closed' - Removed all 'tool_usage' references from data structures - Updated UI to remove Agent Tooling column and icons - Simplified scatter plots to use only 2 colors (open/closed) - Updated legends to show Model Openness instead of Agent Openness/Tooling - Regenerated mock data without tool_usage field - Added openness normalization to handle legacy values Files modified: - aliases.py: Simplified openness constants, removed tool usage - simple_data_loader.py: Added openness normalization, removed tool_usage loading - leaderboard_transformer.py: Removed Agent Tooling column, simplified color mapping - ui_components.py: Removed tooling icons/tooltips, simplified to openness only - generate_mock_jsonl.py: Removed tool_usage from mock data generationca754bb -
Remove submission page and add submission instructions to About page
openhands +1: openhands- Removed submission page import and route from app.py - Removed redirect_submission_on_close_script (no longer needed) - Added new section in About page: 'Submitting Results & Accessing Raw Data' - Instructions now point users to submit via GitHub PR to openhands-index-results - Added information about accessing raw results from GitHub repository - Clarified data format requirements (metadata.json and scores.json) Submissions are now handled via GitHub PR workflow instead of web form Co-authored-by: openhands <openhands@all-hands.dev>701c496
-
Add category workflow diagrams for 5 OpenHands Index categories
openhands +1: openhands- Bug Fixing: GitHub issue to validated fix workflow - App Creation: API spec to library implementation workflow - Frontend Development: UI requirements to web app workflow - Test Generation: Issue to test case generation workflow - Information Gathering: Question to synthesized answer workflow Each diagram shows 4-step process with benchmark names Co-authored-by: openhands <openhands@all-hands.dev>6324e5d -
Update data loader to support agent-centric directory structure
openhands +1: openhands- Add support for new results/YYYYMMDD_model/ directory structure - Each agent directory contains metadata.json and scores.json - Maintain backward compatibility with old JSONL format - Update mock data to use new structure Co-authored-by: openhands <openhands@all-hands.dev>e003f7b -
Update UI: All-Hands-AI color scheme, agent version column names, and OpenHands logo
openhands +1: openhands- Apply All-Hands-AI color scheme (yellow accent, indigo links, beige background) - Change 'Agent' column to 'Agent Version' - Remove 'Submitter' column - Change 'Models Used' to 'Model' - Update mock data with version numbers (1.0.2, 1.0.1, etc.) - Replace logo with OpenHands branding Co-authored-by: openhands <openhands@all-hands.dev>0ee2099 -
Fix category cost calculation - add category-level aggregation
openhands +1: openhands- Updated simple_data_loader.py to build proper category-to-benchmarks mapping from agenteval.json - Added logic to calculate category-level aggregate scores and costs from individual benchmark results - Updated agenteval.json to include category tags (Bug Fixing, App Creation, etc.) - Fixes issue where Bug Fixing cost was showing as 0.0 instead of the average of swe-bench and swe-bench-multimodal costs Co-authored-by: openhands <openhands@all-hands.dev>5e9c3b9 -
Update category structure to OpenHands Index with 5 categories
openhands +1: openhands- Replace ASTA Bench scientific categories with OpenHands software engineering categories: - Bug Fixing (SWE-Bench, SWE-Bench-Multimodal) - App Creation (Commit0) - Frontend Development (Multi-SWE-Bench) - Test Generation (SWT-Bench) - Information Gathering (GAIA) - Update content.py category descriptions for coding agents - Update app.py routes to use new category pages - Add 5 new category page files - Update category_page_builder.py diagram mappings - Change 'four' to 'five' categories in intro - Change 'scientific agents' to 'coding agents' Co-authored-by: openhands <openhands@all-hands.dev>c56f232 -
Update categories to 5 software engineering domains
openhands +1: openhandsCategories: - Bug Fixing: SWE-Bench, SWE-Bench-Multimodal - App Creation: Commit0 - Frontend Development: Multi-SWE-Bench - Test Generation: SWT-Bench - Information Gathering: GAIA Changes: - Updated agenteval.json with new category tags - Rewrote category descriptions in content.py for coding agents - Updated intro paragraph (4 → 5 categories, scientific → coding agents) - Added URLs for each benchmark Co-authored-by: openhands <openhands@all-hands.dev>0db3899 -
Replace all AstaBench references with OpenHands Index
openhands- Updated branding across all files (about.py, content.py, etc.) - Rewrote about.py to focus on software engineering benchmarks - Updated descriptions to reflect OpenHands' focus on coding agents - Maintained original design and layout structure7904c2d -
Fix null columns by mapping source data to expected column names
openhands- Modified simple_data_loader.py to use lowercase column names that match transformer expectations - Changed agent_name → agent, llm_base → models used, tool_usage → agent tooling, submission_time → date - Fixed leaderboard_transformer.py tag map capitalization to match column names - All 24 columns now populated correctly (no null values) - Resolves JavaScript table rendering erroraa07520 -
Add verbose logging and include JSONL data files
openhands +1: openhands- Add comprehensive logging throughout app.py to track initialization progress - Include JSONL data files that were missing from data directory - These files contain the actual leaderboard data needed by SimpleLeaderboardViewer Co-authored-by: openhands <openhands@all-hands.dev>b5df580 -
Remove signal-based timeout (incompatible with HF Spaces)
openhands- signal.SIGALRM not available/working in HF Spaces environment - Rely on subprocess timeouts which work correctly - Add try-except blocks around data fetch/copy for better error handling - Simplify setup_mock_data() implementation9da8453 -
Add comprehensive error handling and timeout protection
openhands- Wrap setup_mock_data() in try-except in app.py to prevent crashes - Add 60-second timeout to data setup using signal.alarm() - Add verbose logging to stderr for debugging HF Space issues - Change logging level to INFO for better visibility - Print clear status messages at each step - Ensures app continues even if data setup fails87e9f6b -
Fix git clone hanging issue - prevent credential prompts
openhands- Add GIT_TERMINAL_PROMPT=0 to prevent git from prompting for credentials - Add stdin=subprocess.DEVNULL to block any input requests - Reduce timeout from 60s to 30s for faster failover - This fixes the app hanging on 'Loading...' when GitHub repo is empty671ebc9 -
Add enhanced debugging output for data setup
openhands- Add verbose logging to track data loading process - Try both relative and absolute paths for mock data - Display current working directory and file listings - Show which data source is being used (GitHub or mock) - Verify copied files after setup2176460 -
Trigger rebuild: data loading fix
openhandse18eb21 -
Fix data loading: fetch from GitHub with fallback to mock data
openhands- Add setup_data.py to fetch data from openhands-index-results GitHub repo - Falls back to mock data when GitHub repo is empty/unavailable - Copy data to expected extraction directory on app startup - Fixes JavaScript error: Cannot read properties of null - Data now loads correctly and tables render properlya63b4cf -
Replace AstaBench logo with OpenHands logo
openhands7d37950 -
Add validation.jsonl data and validation split to agenteval.json
openhandsa04df0a -
Revert homepage simplification - keep original design
openhands742f074 -
Optimize app loading: simplify home page to avoid expensive data loading at startup
openhandsbecc391 -
Fix runtime errors and data loading
openhands- Created combined test.jsonl with all 6 benchmark results - Updated agenteval.json to include test split with all tasks - Added missing imports to submission.py (DatasetDict, Dataset, etc.) - Fixed LeaderboardViewer reference in ui_components.py - Commented out submission page due to Modal component issue - Disabled scheduler that was causing indentation issues66fcacd -
Convert to JSONL data format and remove agent-eval dependency
openhands +1: openhands- Replace parquet files with JSONL for better readability and simplicity - Remove agent-eval library dependency to avoid secret requirements - Create simple_data_loader.py as lightweight replacement for LeaderboardViewer - Create submission_utils.py for submission handling functions - Simplify Dockerfile (no SSH keys or secrets needed) - Reduce requirements.txt to essential packages only - Generate mock data for all 6 benchmarks (swe-bench, multi-swe-bench, swe-bench-multimodal, swt-bench, commit0, gaia) - Update aliases.py to define constants locally - Fix imports in submission.py and test files Co-authored-by: openhands <openhands@all-hands.dev>1027cfb -
Initial OpenHands Index leaderboard based on ASTA Bench
openhands +1: openhands- Updated branding (title, README, logo) - Changed dataset references to OpenHands Index datasets - Modified agenteval.json to include 6 OpenHands datasets: * swe-bench * multi-swe-bench * swe-bench-multimodal * swt-bench * commit0 * gaia - Added mock data for initial testing Co-authored-by: openhands <openhands@all-hands.dev>085a012