CRAG-MM 2025 Challenge
🚀 Important Links: 🏆 Official Challenge Page | 📖 CRAG-MM Starter Kit
Welcome to the official Hugging Face space for datasets and indices related to the CRAG-MM 2025 Challenge (KDD Cup 2025).
Below are the core datasets and search indices needed for the CRAG-MM benchmark, organized in tables for clarity.
📌 Single-Turn Resources
| Type | Resource | Description | Latest Release | Previous Release |
|---|---|---|---|---|
| 📚 Dataset | 🤗 crag-mm-single-turn-public | Single-turn conversations, images, and metadata | v0.1.2 | v0.1.1 |
| 🌐 Web Search Index | 🤗 web-search-index-validation | Web index for validation (via CRAG Search API) | Dataset version: v0.1.2, Index: v0.6, API: v0.5.0 |
Dataset version: v0.1.1, Index: v0.4, API: v0.4.0 |
| 🌐 Web Search Index | 🤗 web-search-index-public-test | Web index for public test (via CRAG Search API) | Dataset version: v0.1.2, Index: v0.6, API: v0.5.0 |
|
| 🖼️ Image Search Index | 🤗 image-search-index-validation | Image index for validation (via CRAG Search API) | Dataset version: v0.1.2, Index: v0.5, API: v0.5.0 |
Dataset version: v0.1.1, Index: v0.4, API: v0.4.0 |
| 🖼️ Image Search Index | 🤗 image-search-index-public-test | Image index for public test (via CRAG Search API) | Dataset version: v0.1.2, Index: v0.5, API: v0.5.0 |
Note: For the final evaluation, we will use a different (and larger) index. It will cover everything in the validation and "public_test" index, so you can assume that the behaviors are similar.
📌 Multi-Turn Resources
| Type | Resource | Description | Latest Release | Previous Release |
|---|---|---|---|---|
| 📚 Dataset | 🤗 crag-mm-multi-turn-public | Multi-turn conversations with images and URLs | v0.1.2 | v0.1.1 |
| 🌐 Web Search Index | 🤗 web-search-index-validation | Web index for validation (via CRAG Search API) | Dataset version: v0.1.2, Index: v0.6, API: v0.5.0 |
Dataset version: v0.1.1, Index: v0.4, API: v0.4.0 |
| 🌐 Web Search Index | 🤗 web-search-index-public-test | Web index for public test (via CRAG Search API) | Dataset version: v0.1.2, Index: v0.6, API: v0.5.0 |
|
| 🖼️ Image Search Index | 🤗 image-search-index-validation | Image index for validation (via CRAG Search API) | Dataset version: v0.1.2, Index: v0.5, API: v0.5.0 |
Dataset version: v0.1.1, Index: v0.4, API: v0.4.0 |
| 🖼️ Image Search Index | 🤗 image-search-index-public-test | Image index for public test (via CRAG Search API) | Dataset version: v0.1.2, Index: v0.5, API: v0.5.0 |
👉 Check out the Starter Kit for detailed code examples and baseline models.
⌚️ Release History
[June 2nd] We released new web search index for validation and public_test. Current tag version is v0.6.
- fixed corrupted validation index, resulting in unexpected retrieval results.
- added additional documents (10-20% more web chunks) to validation and public-test search corpus respectively to improve recall.
- no change to image search index
- no change to the seach API
[May 25th] We released v0.1.2 validation and public_test data alongside new validation and public_test indices (v0.5). Major changes include:
- Cleaner data: Removed low-quality or ambiguous image–QA pairs.
- Expanded volume: More samples added across all three tasks.
- Bug fixes: Minor corrections (e.g. consistent domain values).
- Improved web search: Upgraded to BGE text encoder.
[April 24th] We released a new API version (v0.4.0) alongside new validation indices (v0.4). Major changes include:
- fixed the corrupted chromadb index for both web and image search.
- upgraded web search API: enabled web chunk level similarity search, while previous version is at page level. page urls are deduped during retrieval.
- enable downloading the index from Huggingface repo by tags. previous index was tagged with v0.3. current index is v0.4 / main. we recommend always loading from main.
- reset num_threads to os.cpu_count() when loading the web/image search index.