Logo

SIGIR 2025 LiveRAG Challenge

Organized by the Technology Innovation Institute (TII)

With support from

Support

Challenge
Overview

Retrieval Augmented Generation (RAG) has emerged as a key technology to mitigate the issues that Large Language Models (LLMs) face when they lack adequate knowledge. Given a user’s request, a RAG system searches auxiliary sources to augment the prompt associated with the request with relevant content. RAG is attracting a great deal of attention from the AI community, yet it is still hard to assess the quality of RAG systems in a systematic manner.

The goal of the LiveRAG Challenge is to allow research teams across academia and industry to advance their RAG research and compare the performance of their solutions with other teams, on a fixed corpus (derived from the publicly available FineWeb) and a fixed open-source LLM, Falcon3-10B-Instruct.

Image

The LiveRAG challenge requires an application process, after which selected teams will be

  • Awarded up to 1500 USD in AWS compute credits to train their RAG solution, as well as up to 750 USD in Pinecone compute credits to use/generate their RAG indices, and
  • Given early access to TII’s DataMorgana tool (see Challenge Details below) to help them generate synthetic benchmarks for training and testing.

During the Live Challenge Day, the teams will be provided with a stream of unseen questions and will have to return their answers under strict response-time constraints.

Info

Finalists will be requested to present their results at the LiveRAG workshop day to be held at the SIGIR'2025 conference, during which winners will be announced and prizes will be awarded.

Image

Challenge Calendar

Date (2025) Details
Feb 24 Application submission deadline SIGIR 2025 Easychair site
(Select: SIGIR 2025 LiveRAG Challenge track)
Mar 12
  • Application acceptance notification
  • Opening of easychair site for short paper submission
  • AWS and Pinecone resources and credits made available to accepted participants together with detailed operational instructions
Mar 15 Training and testing tool (DataMorgana) made available to participants
May 8 “Dry” test for participants of live service on a small question set
May 12 Live Challenge Day – test questions shared and live service for answers submission opens
May 19 Short paper submission deadline
May 29 Short paper notification and announcement of finalists
July 17
  • LiveRAG Workshop at SIGIR'2025 in Padua, Italy
  • Presentation of research by selected participants
  • Announcement of winner and runner(s)-up

Application Process
and Eligibility

  • To be selected for the Challenge and be eligible for free credits on AWS/Pinecone, teams of no more than five members must submit an application of one page plus references and bios of members in the SIGIR 2025 Easychair site (Select: SIGIR2025 LiveRAG Challenge track).
  • Application abstracts will be assessed by the Challenge Program Committee that will consider various factors such as quality of the research proposal, team qualifications, diversity, etc.
  • Following the Live Challenge Day, all selected teams will be requested to submit their report in the SIGIR 2025 Easychair site with a short paper (up to 4 pages) describing their work and results.
  • Paper submission is mandatory to be considered for prizes. At least one author of each accepted paper must commit to register and attend the LiveRAG Workshop in person at SIGIR'2025 on July 17, 2025.
  • Note that there will be no official proceedings for the Challenge. The authors are welcome to publish their submissions on arXiv and resubmit in other venues at a later stage.
  • Applications and short paper submission guidelines are detailed in Application and Paper Submission Guidelines.

Application Process
and Eligibility

Icon Note

Each team must agree to the Challenge privacy policy and terms of entry as specified in the Challenge’s Terms and Conditions. They should in addition strictly follow the Challenges Guidelines.

Challenge
Key Elements

Selected teams are expected to build a RAG solution over FineWeb-10BT (a 15M documents subset of FineWeb) and integrate it with the Challenge LLM (Falcon3-10B-Instruct) for answer generation. Selected participants have the choice between building their own search indices over the Challenge dataset or taking advantage (leveraging their allocated credits) of two prebuilt indices, a Pinecone Dense index and an Opensearch Sparse index.

Image

DataMorgana: Synthetic Benchmark Generator

Participants will get early access to TII’s DataMorgana, a synthetic and configurable benchmark generator for training and testing their system prior to the live event. DataMorgana is a new tool (See arXiv paper for more details) that allows RAG developers to generate synthetic questions and answers from a given corpus via configuration instructions. They will be able to specify the type of questions they are expecting, as well as the type of users who would express them, so as to ensure benchmark diversity. The same DataMorgana tool will be used for generating an original test set at the live event and for automatic evaluation afterwards.

Image

Evaluation
Process

The Evaluation process will consider two metrics, Relevance and Faithfulness (see Challenge Details for more information), and be conducted in two stages:

  • Automated Evaluation: Using DataMorgana to assess those metrics.
  • Human Evaluation: For the top-performing systems to ensure quality and compliance.

Submitted code, prompts, and artifacts will undergo a detailed review to verify adherence to challenge rules.

Image

Live Challenge Day
Workflow

  • Participants will receive an original set of questions generated by DataMorgana.
  • Teams must submit the answers generated by their RAG system alongside the Falcon prompts used, within a predefined time window.
  • A dry test session will occur a few days before the Live Challenge Day, allowing participants to validate their submission and upload processes.
Image

Additional
Information

Comprehensive details can be found in the Challenge Details, while operational instructions will be shared directly with selected participants. Prepare to showcase your skills and push the boundaries of RAG system capabilities!

SIGIR’2025 LiveRAG
Challenge

First Prize $5000
Second Prize $3000
Third Prize $2000
Icon

If no academic team ranks among the top three, the first-ranking academic team in the top ten participants will be considered for the third prize.

Organization

Organizing Team

  • David Carmel Technology Innovation Institute
  • Simone Filice Technology Innovation Institute
  • Mehdi Ghissassi AI71
  • Hakim Hacid Technology Innovation Institute
  • Guy Horowitz Technology Innovation Institute
  • Zohar Karnin Technology Innovation Institute
  • Liane Lewin-Eytan Technology Innovation Institute
  • Yoelle Maarek Technology Innovation Institute
  • Ran Tavory Technology Innovation Institute
  • Oren Somekh Technology Innovation Institute

Program Committee Members

  • Charles L. A. Clarke University of Waterloo
  • Yi Chang Jilin University
  • Ido Guy Meta
  • Oren Kurland Technion, Israel Institute of Technology
  • Yiqun Liu Tsinghua University
  • Antonio Mallia Pinecone
  • Marc Najork Google Deepmind
  • Fabrizio Silvestri Sapienza Università di Roma
  • Ian Soboroff NIST
  • Emine Yilmaz University College London and Amazon
  • Elad Yom-Tov Bar-Ilan University

Important Dates

Application
Submission Deadline

February 24, 2025

Application Submission
Notifications and Opening
of Resources

March 12, 2025

Live Challenge
Day

May 12

Short Paper Submission Deadline

May 19

Short Paper
Notifications

May 29

SIGIR LiveRAG
workshop

July 17, 2025

Contact Us

For any question about the challenge, send mail to