QPP++ 2025: Query Performance Prediction and its Applications in the Era of Large Language Models

Call for Papers

Themes and Topics

Given the rapid advancement of LLMs, we present examples of topics we encourage authors to explore in this workshop:

Predicting the performance of LLM-based retrievers/re-rankers, or generative AI systems. Most QPP studies still focus on predicting the performance of rankers based on small-scale pre-trained language models (e.g., BERT or T5). However, little work has explored predicting the performance of newly emerged LLM-based retrievers/re-rankers , or more broadly, LLM-based generative AI systems. Therefore, we aim to ad- dress questions including, but not limited to, the following:

To what extent does the performance of existing QPP methods (designed for retrievers/re-rankers using small-scale pre-trained languages) generalise to LLM-based retrievers and re-rankers?
How can we effectively predict the performance of emerging new re-ranking paradigms based on LLMs, such as pair-wise or list-wise re-rankers?
How can we predict the performance of generative AI systems? E.g., how to predict the text generation quality of an LLM in response to a prompt?

Leveraging the capabilities of LLMs to enhance QPP quality. LLMs have been applied to a wide range of natural language processing (NLP) and IR tasks, achieving numerous state-of-the-art results. However, few studies (e.g., QPP-GenRE) have explored leveraging LLM to model QPP. We aim to address questions including, but not limited to, the following:

What kind of features from LLMs (e.g., embedding) can we use for QPP?
How well a QPP model based on LLMs performs when predicting the ranking quality of an LLM-based retriever/re-ranker or the text generation quality of an LLM?

Applying QPP to benefit various downstream tasks. Most studies evaluate QPP methods only using metrics not directly correlated to downstream tasks, e.g., linear correlation coefficients. However, it is under-explored whether QPP methods evaluated in such a way can be effectively applied to benefit downstream tasks, especially when it comes to applying them to important tasks in the era of LLMs, e.g., retrieval-augmented generation (RAG). So we aim to address questions including, but not limited to, the following:

Which downstream tasks (especially in the era of LLMs) does QPP have the potential to benefit? E.g., in RAG, can QPP be used to determine when to rely on the retrieved documents? In LLM-based re-ranking, can QPP be used to determine query-specific re-ranking depths?
How exactly to use QPP to benefit those tasks?

Exploring QPP in the context of multi-modal content. Most studies focus on QPP in the context of text. However, research on QPP in the context of multi-modal content, such as images or even video, remains limited. We aim to address questions including, but not limited to, the following:

QPP for text-to-image search, image-to-image search, or image generation.
QPP in the context of video search/generation.

Exploring multilingual QPP. There’s limited research on QPP for languages other than English. How well do current QPP methods generalise to non-English languages?

Beyond the above aspects, we also welcome submissions on other QPP-related aspects. There is a need for deeper exploration in other QPP-related areas, including, but not limited to, the following: QPP for conversational search, recommendation systems, question answering, and fairness. We aim to stimulate discussions on these topics.

Paper type

We welcome manuscripts about any QPP-related subjects. We welcome a diverse range of submission types:

Original papers. We welcome paper submissions in various types, including research papers, position papers, reproducibility papers, survey papers, and data collection papers. All submissions should be ranged from 4 to 10 pages in length, including references.

Published papers. We welcome papers already accepted at top-tier conferences or in journals, e.g., SIGIR, CIKM, WWW, ECIR, ACL, EMNLP, TOIS, IP&M. Authors should submit 1-page abstract for a published paper, including the full reference where the paper was accepted.

Paper submission & selection

Note that submissions be written in English. For original papers, double submission is not allowed except for papers already on ArXiv.

The papers (.pdf format) should be submitted using the EasyChair submission system at https://easychair.org/conferences/?conf=qpp2025. The paper template can be found at https://ceur-ws.org/HOWTOSUBMIT.html.

The review process is single-blind. Each manuscript will be peer-reviewed by at least three programme committee (PC) members.

Authors of accepted papers will be invited to give oral presentations at the workshop. The accepted papers will be published in the CEUR-WS.org proceedings series. Since the proceedings only accept papers with a minimum of 4 pages, we will combine 1-page abstract submissions into one or more papers for publication.

Extension on TOIS Special Issue

The best papers submitted to the QPP++ 2025 workshop will be invited for an extension on the TOIS Special Issue on "Query Performance Prediction Towards Novel Information Retrieval Paradigms" . The authors of the best papers are invited to implement suggestions and comments provided by the other participants to the workshop and submit an extended version of their work that will be published on the Transactions on Information Systems (TOIS).

Recommended benchmarks & tools

To facilitate workshop discussions, we encourage (though it is not mandatory) authors to use common, publicly available benchmarks. This will help authors share their experimental experiences and insights more effectively. We also plan to set a discussion session for sharing experience on common datasets.