Archive Query Log
Synopsis
Query logs and search engine result pages (SERPs) are essential for research on information retrieval, but search providers like Bing or Google typically do not publish them for privacy and business reasons, while none of the few publicly available query logs combines size, scope, and diversity. At the same time, web archives like the Internet Archive's Wayback Machine have captured the SERPs of countless searches over the last decades.
The Archive Query Log (AQL) mines these archived SERPs. Its first version included 356 million queries across 550 search providers collected over 25 years, enabling diachronic analyses of search engines and promoting transparency and accountability in the search industry [SIGIR 2023] [code] [data] [demo] [pypi] [video].
To facilitate continuous SERP from web archives, for the AQL, we have developed an open SERP mining infrastructure that can be scaled from a single machine to a Kubernetes cluster [ECIR 2026]. Besides contributions for new search providers and parsers, this architecture enables researchers to analyze the AQL data, for example, on predicting the performance of retrieval corpora [QPP++ 2025].
People
- Maik Fröbe
- Lukas Gienapp
- Matthias Hagen
- Jan Heinrich Merker
- Andrew Parry
- Martin Potthast
- Simon Ruth
- Harrisen Scells
- Sebastian Schmidt
- Benno Stein