Overview of the SISAP 2024 Indexing Challenge

Eric S. Tellez; Martin Aumüller; Vladimir Mic

doi:10.1007/978-3-031-75823-2_21

Overview of the SISAP 2024 Indexing Challenge

Eric S. Tellez^*, Martin Aumüller, Vladimir Mic

^*Corresponding author for this work

Department of Computer Science

Research output: Contribution to book/anthology/report/proceeding › Article in proceedings › Research › peer-review

Abstract

The SISAP 2024 Indexing Challenge invited replicable and competitive approximate similarity search solutions for datasets of up to 100 million real-valued vectors. Participants are evaluated on the search performance of their implementations under quality constraints. Using a subset of the deep features of a neural network model provided by the LAION-5B dataset, the challenge posed three tasks, each with its unique focus:Task 1, Unrestricted indexing: Conduct a classical approximate nearest neighbors search, ensuring an average recall of at least 0.8 for 30-NN queries.Task 2, Memory-constrained indexing with reranking: Conduct nearest neighbors search in a low-memory setting where the dataset collection is only accessible on disk, ensuring the same quality as in Task 1.Task 3, Memory-constrained indexing without reranking: Conduct nearest neighbor search in a setting where the dataset cannot be accessed at search stage, ensuring an average recall of at least 0.4 for 30-NN queries. Task 1, Unrestricted indexing: Conduct a classical approximate nearest neighbors search, ensuring an average recall of at least 0.8 for 30-NN queries. Task 2, Memory-constrained indexing with reranking: Conduct nearest neighbors search in a low-memory setting where the dataset collection is only accessible on disk, ensuring the same quality as in Task 1. Task 3, Memory-constrained indexing without reranking: Conduct nearest neighbor search in a setting where the dataset cannot be accessed at search stage, ensuring an average recall of at least 0.4 for 30-NN queries. The present paper describes the details of the challenge, the evaluation system that was developed with it, and gives an overview of the submitted solutions.

Original language	English
Title of host publication	Similarity Search and Applications - 17th International Conference, SISAP 2024, Proceedings
Editors	Edgar Chávez, Benjamin Kimia, Jakub Lokoč, Marco Patella, Jan Sedmidubsky
Number of pages	11
Place of publication	Cham
Publisher	Springer
Publication date	2025
Pages	255–265
ISBN (Print)	978-3-031-75822-5
ISBN (Electronic)	978-3-031-75823-2
DOIs	https://doi.org/10.1007/978-3-031-75823-2_21
Publication status	Published - 2025

Series	Lecture Notes in Computer Science
Volume	15268
ISSN	0302-9743

Keywords

Approximate nearest neighbor search
Experimental comparison of search methods
Indexing and searching pipelines

Access to Document

10.1007/978-3-031-75823-2_21

Library access?

Check availability with the Royal Danish Library

Cite this

S. Tellez, Eric ; Aumüller, Martin ; Mic, Vladimir. / Overview of the SISAP 2024 Indexing Challenge. Similarity Search and Applications - 17th International Conference, SISAP 2024, Proceedings. editor / Edgar Chávez ; Benjamin Kimia ; Jakub Lokoč ; Marco Patella ; Jan Sedmidubsky. Cham : Springer, 2025. pp. 255–265 (Lecture Notes in Computer Science, Vol. 15268).

@inproceedings{c49b9bb34ce149a589e94fdd9cb684cb,

title = "Overview of the SISAP 2024 Indexing Challenge",

abstract = "The SISAP 2024 Indexing Challenge invited replicable and competitive approximate similarity search solutions for datasets of up to 100 million real-valued vectors. Participants are evaluated on the search performance of their implementations under quality constraints. Using a subset of the deep features of a neural network model provided by the LAION-5B dataset, the challenge posed three tasks, each with its unique focus:Task 1, Unrestricted indexing: Conduct a classical approximate nearest neighbors search, ensuring an average recall of at least 0.8 for 30-NN queries.Task 2, Memory-constrained indexing with reranking: Conduct nearest neighbors search in a low-memory setting where the dataset collection is only accessible on disk, ensuring the same quality as in Task 1.Task 3, Memory-constrained indexing without reranking: Conduct nearest neighbor search in a setting where the dataset cannot be accessed at search stage, ensuring an average recall of at least 0.4 for 30-NN queries. Task 1, Unrestricted indexing: Conduct a classical approximate nearest neighbors search, ensuring an average recall of at least 0.8 for 30-NN queries. Task 2, Memory-constrained indexing with reranking: Conduct nearest neighbors search in a low-memory setting where the dataset collection is only accessible on disk, ensuring the same quality as in Task 1. Task 3, Memory-constrained indexing without reranking: Conduct nearest neighbor search in a setting where the dataset cannot be accessed at search stage, ensuring an average recall of at least 0.4 for 30-NN queries. The present paper describes the details of the challenge, the evaluation system that was developed with it, and gives an overview of the submitted solutions.",

keywords = "Approximate nearest neighbor search, Experimental comparison of search methods, Indexing and searching pipelines",

author = "{S. Tellez}, Eric and Martin Aum{\"u}ller and Vladimir Mic",

year = "2025",

doi = "10.1007/978-3-031-75823-2_21",

language = "English",

isbn = "978-3-031-75822-5",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "255–265",

editor = "Edgar Ch{\'a}vez and Benjamin Kimia and Jakub Loko{\v c} and Marco Patella and Jan Sedmidubsky",

booktitle = "Similarity Search and Applications - 17th International Conference, SISAP 2024, Proceedings",

address = "Netherlands",

}

S. Tellez, E, Aumüller, M & Mic, V 2025, Overview of the SISAP 2024 Indexing Challenge. in E Chávez, B Kimia, J Lokoč, M Patella & J Sedmidubsky (eds), Similarity Search and Applications - 17th International Conference, SISAP 2024, Proceedings. Springer, Cham, Lecture Notes in Computer Science, vol. 15268, pp. 255–265. https://doi.org/10.1007/978-3-031-75823-2_21

Overview of the SISAP 2024 Indexing Challenge. / S. Tellez, Eric; Aumüller, Martin; Mic, Vladimir.
Similarity Search and Applications - 17th International Conference, SISAP 2024, Proceedings. ed. / Edgar Chávez; Benjamin Kimia; Jakub Lokoč; Marco Patella; Jan Sedmidubsky. Cham: Springer, 2025. p. 255–265 (Lecture Notes in Computer Science, Vol. 15268).

Research output: Contribution to book/anthology/report/proceeding › Article in proceedings › Research › peer-review

TY - GEN

T1 - Overview of the SISAP 2024 Indexing Challenge

AU - S. Tellez, Eric

AU - Aumüller, Martin

AU - Mic, Vladimir

PY - 2025

Y1 - 2025

N2 - The SISAP 2024 Indexing Challenge invited replicable and competitive approximate similarity search solutions for datasets of up to 100 million real-valued vectors. Participants are evaluated on the search performance of their implementations under quality constraints. Using a subset of the deep features of a neural network model provided by the LAION-5B dataset, the challenge posed three tasks, each with its unique focus:Task 1, Unrestricted indexing: Conduct a classical approximate nearest neighbors search, ensuring an average recall of at least 0.8 for 30-NN queries.Task 2, Memory-constrained indexing with reranking: Conduct nearest neighbors search in a low-memory setting where the dataset collection is only accessible on disk, ensuring the same quality as in Task 1.Task 3, Memory-constrained indexing without reranking: Conduct nearest neighbor search in a setting where the dataset cannot be accessed at search stage, ensuring an average recall of at least 0.4 for 30-NN queries. Task 1, Unrestricted indexing: Conduct a classical approximate nearest neighbors search, ensuring an average recall of at least 0.8 for 30-NN queries. Task 2, Memory-constrained indexing with reranking: Conduct nearest neighbors search in a low-memory setting where the dataset collection is only accessible on disk, ensuring the same quality as in Task 1. Task 3, Memory-constrained indexing without reranking: Conduct nearest neighbor search in a setting where the dataset cannot be accessed at search stage, ensuring an average recall of at least 0.4 for 30-NN queries. The present paper describes the details of the challenge, the evaluation system that was developed with it, and gives an overview of the submitted solutions.

AB - The SISAP 2024 Indexing Challenge invited replicable and competitive approximate similarity search solutions for datasets of up to 100 million real-valued vectors. Participants are evaluated on the search performance of their implementations under quality constraints. Using a subset of the deep features of a neural network model provided by the LAION-5B dataset, the challenge posed three tasks, each with its unique focus:Task 1, Unrestricted indexing: Conduct a classical approximate nearest neighbors search, ensuring an average recall of at least 0.8 for 30-NN queries.Task 2, Memory-constrained indexing with reranking: Conduct nearest neighbors search in a low-memory setting where the dataset collection is only accessible on disk, ensuring the same quality as in Task 1.Task 3, Memory-constrained indexing without reranking: Conduct nearest neighbor search in a setting where the dataset cannot be accessed at search stage, ensuring an average recall of at least 0.4 for 30-NN queries. Task 1, Unrestricted indexing: Conduct a classical approximate nearest neighbors search, ensuring an average recall of at least 0.8 for 30-NN queries. Task 2, Memory-constrained indexing with reranking: Conduct nearest neighbors search in a low-memory setting where the dataset collection is only accessible on disk, ensuring the same quality as in Task 1. Task 3, Memory-constrained indexing without reranking: Conduct nearest neighbor search in a setting where the dataset cannot be accessed at search stage, ensuring an average recall of at least 0.4 for 30-NN queries. The present paper describes the details of the challenge, the evaluation system that was developed with it, and gives an overview of the submitted solutions.

KW - Approximate nearest neighbor search

KW - Experimental comparison of search methods

KW - Indexing and searching pipelines

UR - http://www.scopus.com/inward/record.url?scp=105002727082&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-75823-2_21

DO - 10.1007/978-3-031-75823-2_21

M3 - Article in proceedings

SN - 978-3-031-75822-5

T3 - Lecture Notes in Computer Science

SP - 255

EP - 265

BT - Similarity Search and Applications - 17th International Conference, SISAP 2024, Proceedings

A2 - Chávez, Edgar

A2 - Kimia, Benjamin

A2 - Lokoč, Jakub

A2 - Patella, Marco

A2 - Sedmidubsky, Jan

PB - Springer

CY - Cham

ER -

Overview of the SISAP 2024 Indexing Challenge

Abstract

Keywords

Access to Document

Library access?

Other files and links

Fingerprint

Cite this