Anote - Model Leaderboard
The Anote Model Leaderboard is a public API-driven benchmarking platform for evaluating large language models (LLMs) across diverse task types.
Core trick
Exposes a REST API that registers datasets and model submissions, then scores them with standardized benchmark metrics.
Description
The Anote Model Leaderboard is a public API-driven benchmarking platform for evaluating large language models (LLMs) across diverse task types. It solves the problem of fragmented, slow, and manual model evaluation by providing a scalable, transparent, and community-driven system where new datasets and model submissions can be added programmatically.
The platform supports multiple task categories including:
Text Classification
Named Entity Recognition (NER)
Document-level Q&A (chatbot)
Line-level Q&A (prompting)
Researchers and enterprises can benchmark zero-shot, fine-tuned, and supervised models. This ensures fair comparison of models, and encourages reproducibility through standardized evaluation metrics.
README excerpt
# Anote - Model Leaderboard
Anote's Model Leaderboard provides a way to benchmark and compare model performance. We have an API to:
1. **Adding new datasets** to the leaderboard across **all task types**.
2. **Adding new model submissions** to existing datasets.
The API is the backbone for a transparent, scalable, and community-driven benchmarking platform for AI models, supporting **text classification, named entity recognition, document-level Q&A (chatbot), and line-level Q&A (prompting)**.
### API Endpoints
#### `POST /api/leaderboard/add_dataset`
**Purpose**: Add a new benchmark dataset to the leaderboard.
**Input (JSON)**:
```json
{
"name": "Financial Phrasebank - Classification Accuracy",
"url": "https://huggingface.co/datasets/takala/financial_phrasebank",
"task_type": "text_classification",
"description": "A dataset for financial sentiment classification.",
"models": [
{
"rank": 1,
"model": "Gemini",
"score": 0.95,
"ci": "0.93 - 0.97",
"updated": "Sep 2024"
}
]
}
```
**Response**:
```json
{
"status": "success",
"message": "Dataset added to leaderboard.",
"dataset_id": "uuid"
}
```
---
#### `GET /public/benchmark_csvs`
Purpose: List available CSV benchmark datasets in `frontend/public/benchmark_csvs` with inferred task types and columns.
Response:
```
{
"success": true,
"datasets": [ { "filename": "Commonsense.csv", "task_type": "multiple_choice", "columns": [ ... ] }, ... ]
}
```
#### `POST /public/ru