Overview of the SISAP 2024 Indexing Challenge

Eric S. Tellez; Martin Aumüller; Vladimir Mic

doi:10.1007/978-3-031-75823-2_21

Overview of the SISAP 2024 Indexing Challenge

Eric S. Tellez^*, Martin Aumüller, Vladimir Mic

^*Corresponding author af dette arbejde

Institut for Datalogi

Publikation: Bidrag til bog/antologi/rapport/proceeding › Konferencebidrag i proceedings › Forskning › peer review

Abstract

The SISAP 2024 Indexing Challenge invited replicable and competitive approximate similarity search solutions for datasets of up to 100 million real-valued vectors. Participants are evaluated on the search performance of their implementations under quality constraints. Using a subset of the deep features of a neural network model provided by the LAION-5B dataset, the challenge posed three tasks, each with its unique focus:Task 1, Unrestricted indexing: Conduct a classical approximate nearest neighbors search, ensuring an average recall of at least 0.8 for 30-NN queries.Task 2, Memory-constrained indexing with reranking: Conduct nearest neighbors search in a low-memory setting where the dataset collection is only accessible on disk, ensuring the same quality as in Task 1.Task 3, Memory-constrained indexing without reranking: Conduct nearest neighbor search in a setting where the dataset cannot be accessed at search stage, ensuring an average recall of at least 0.4 for 30-NN queries. Task 1, Unrestricted indexing: Conduct a classical approximate nearest neighbors search, ensuring an average recall of at least 0.8 for 30-NN queries. Task 2, Memory-constrained indexing with reranking: Conduct nearest neighbors search in a low-memory setting where the dataset collection is only accessible on disk, ensuring the same quality as in Task 1. Task 3, Memory-constrained indexing without reranking: Conduct nearest neighbor search in a setting where the dataset cannot be accessed at search stage, ensuring an average recall of at least 0.4 for 30-NN queries. The present paper describes the details of the challenge, the evaluation system that was developed with it, and gives an overview of the submitted solutions.

Originalsprog	Engelsk
Titel	Similarity Search and Applications - 17th International Conference, SISAP 2024, Proceedings
Redaktører	Edgar Chávez, Benjamin Kimia, Jakub Lokoč, Marco Patella, Jan Sedmidubsky
Antal sider	11
Udgivelsessted	Cham
Forlag	Springer
Publikationsdato	2025
Sider	255–265
ISBN (Trykt)	978-3-031-75822-5
ISBN (Elektronisk)	978-3-031-75823-2
DOI	https://doi.org/10.1007/978-3-031-75823-2_21
Status	Udgivet - 2025

Navn	Lecture Notes in Computer Science
Vol/bind	15268
ISSN	0302-9743

Adgang til dokumentet

10.1007/978-3-031-75823-2_21

OpenUrl tilgængelighed

Fuld tekst

Andre filer og links

Link to publication in Scopus

Citationsformater

@inproceedings{c49b9bb34ce149a589e94fdd9cb684cb,

title = "Overview of the SISAP 2024 Indexing Challenge",

abstract = "The SISAP 2024 Indexing Challenge invited replicable and competitive approximate similarity search solutions for datasets of up to 100 million real-valued vectors. Participants are evaluated on the search performance of their implementations under quality constraints. Using a subset of the deep features of a neural network model provided by the LAION-5B dataset, the challenge posed three tasks, each with its unique focus:Task 1, Unrestricted indexing: Conduct a classical approximate nearest neighbors search, ensuring an average recall of at least 0.8 for 30-NN queries.Task 2, Memory-constrained indexing with reranking: Conduct nearest neighbors search in a low-memory setting where the dataset collection is only accessible on disk, ensuring the same quality as in Task 1.Task 3, Memory-constrained indexing without reranking: Conduct nearest neighbor search in a setting where the dataset cannot be accessed at search stage, ensuring an average recall of at least 0.4 for 30-NN queries. Task 1, Unrestricted indexing: Conduct a classical approximate nearest neighbors search, ensuring an average recall of at least 0.8 for 30-NN queries. Task 2, Memory-constrained indexing with reranking: Conduct nearest neighbors search in a low-memory setting where the dataset collection is only accessible on disk, ensuring the same quality as in Task 1. Task 3, Memory-constrained indexing without reranking: Conduct nearest neighbor search in a setting where the dataset cannot be accessed at search stage, ensuring an average recall of at least 0.4 for 30-NN queries. The present paper describes the details of the challenge, the evaluation system that was developed with it, and gives an overview of the submitted solutions.",

keywords = "Approximate nearest neighbor search, Experimental comparison of search methods, Indexing and searching pipelines",

author = "{S. Tellez}, Eric and Martin Aum{\"u}ller and Vladimir Mic",

year = "2025",

doi = "10.1007/978-3-031-75823-2_21",

language = "English",

isbn = "978-3-031-75822-5",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "255–265",

editor = "Edgar Ch{\'a}vez and Benjamin Kimia and Jakub Loko{\v c} and Marco Patella and Jan Sedmidubsky",

booktitle = "Similarity Search and Applications - 17th International Conference, SISAP 2024, Proceedings",

address = "Netherlands",

}

Overview of the SISAP 2024 Indexing Challenge. / S. Tellez, Eric; Aumüller, Martin; Mic, Vladimir.
Similarity Search and Applications - 17th International Conference, SISAP 2024, Proceedings. red. / Edgar Chávez; Benjamin Kimia; Jakub Lokoč; Marco Patella; Jan Sedmidubsky. Cham: Springer, 2025. s. 255–265 (Lecture Notes in Computer Science, Bind 15268).

Publikation: Bidrag til bog/antologi/rapport/proceeding › Konferencebidrag i proceedings › Forskning › peer review

TY - GEN

T1 - Overview of the SISAP 2024 Indexing Challenge

AU - S. Tellez, Eric

AU - Aumüller, Martin

AU - Mic, Vladimir

PY - 2025

Y1 - 2025

N2 - The SISAP 2024 Indexing Challenge invited replicable and competitive approximate similarity search solutions for datasets of up to 100 million real-valued vectors. Participants are evaluated on the search performance of their implementations under quality constraints. Using a subset of the deep features of a neural network model provided by the LAION-5B dataset, the challenge posed three tasks, each with its unique focus:Task 1, Unrestricted indexing: Conduct a classical approximate nearest neighbors search, ensuring an average recall of at least 0.8 for 30-NN queries.Task 2, Memory-constrained indexing with reranking: Conduct nearest neighbors search in a low-memory setting where the dataset collection is only accessible on disk, ensuring the same quality as in Task 1.Task 3, Memory-constrained indexing without reranking: Conduct nearest neighbor search in a setting where the dataset cannot be accessed at search stage, ensuring an average recall of at least 0.4 for 30-NN queries. Task 1, Unrestricted indexing: Conduct a classical approximate nearest neighbors search, ensuring an average recall of at least 0.8 for 30-NN queries. Task 2, Memory-constrained indexing with reranking: Conduct nearest neighbors search in a low-memory setting where the dataset collection is only accessible on disk, ensuring the same quality as in Task 1. Task 3, Memory-constrained indexing without reranking: Conduct nearest neighbor search in a setting where the dataset cannot be accessed at search stage, ensuring an average recall of at least 0.4 for 30-NN queries. The present paper describes the details of the challenge, the evaluation system that was developed with it, and gives an overview of the submitted solutions.

AB - The SISAP 2024 Indexing Challenge invited replicable and competitive approximate similarity search solutions for datasets of up to 100 million real-valued vectors. Participants are evaluated on the search performance of their implementations under quality constraints. Using a subset of the deep features of a neural network model provided by the LAION-5B dataset, the challenge posed three tasks, each with its unique focus:Task 1, Unrestricted indexing: Conduct a classical approximate nearest neighbors search, ensuring an average recall of at least 0.8 for 30-NN queries.Task 2, Memory-constrained indexing with reranking: Conduct nearest neighbors search in a low-memory setting where the dataset collection is only accessible on disk, ensuring the same quality as in Task 1.Task 3, Memory-constrained indexing without reranking: Conduct nearest neighbor search in a setting where the dataset cannot be accessed at search stage, ensuring an average recall of at least 0.4 for 30-NN queries. Task 1, Unrestricted indexing: Conduct a classical approximate nearest neighbors search, ensuring an average recall of at least 0.8 for 30-NN queries. Task 2, Memory-constrained indexing with reranking: Conduct nearest neighbors search in a low-memory setting where the dataset collection is only accessible on disk, ensuring the same quality as in Task 1. Task 3, Memory-constrained indexing without reranking: Conduct nearest neighbor search in a setting where the dataset cannot be accessed at search stage, ensuring an average recall of at least 0.4 for 30-NN queries. The present paper describes the details of the challenge, the evaluation system that was developed with it, and gives an overview of the submitted solutions.

KW - Approximate nearest neighbor search

KW - Experimental comparison of search methods

KW - Indexing and searching pipelines

UR - http://www.scopus.com/inward/record.url?scp=105002727082&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-75823-2_21

DO - 10.1007/978-3-031-75823-2_21

M3 - Article in proceedings

SN - 978-3-031-75822-5

T3 - Lecture Notes in Computer Science

SP - 255

EP - 265

BT - Similarity Search and Applications - 17th International Conference, SISAP 2024, Proceedings

A2 - Chávez, Edgar

A2 - Kimia, Benjamin

A2 - Lokoč, Jakub

A2 - Patella, Marco

A2 - Sedmidubsky, Jan

PB - Springer

CY - Cham

ER -

Overview of the SISAP 2024 Indexing Challenge

Abstract

Adgang til dokumentet

OpenUrl tilgængelighed

Andre filer og links

Fingeraftryk

Citationsformater