Anchoring Fine-tuning of Sentence Transformer with Semantic Label Information for Efficient Truly Few-shot Classification

TY - GEN

T1 - Anchoring Fine-tuning of Sentence Transformer with Semantic Label Information for Efficient Truly Few-shot Classification

AU - Pauli, Amalie Brogaard

AU - Derczynski, Leon

AU - Assent, Ira

PY - 2023

Y1 - 2023

N2 - Few-shot classification is a powerful technique, but training requires substantial computing power and data. We propose an efficient method with small model sizes and less training data with only 2-8 training instances per class. Our proposed method, AncSetFit, targets low data scenarios by anchoring the task and label information through sentence embeddings in fine-tuning a Sentence Transformer model. It uses contrastive learning and a triplet loss to enforce training instances of a class to be closest to its own textual semantic label information in the embedding space-and thereby learning to embed different class instances more distinct. AncSetFit obtains strong performance in data-sparse scenarios compared to existing methods across SST-5, Emotion detection, and AG News data, even with just two examples per class.

AB - Few-shot classification is a powerful technique, but training requires substantial computing power and data. We propose an efficient method with small model sizes and less training data with only 2-8 training instances per class. Our proposed method, AncSetFit, targets low data scenarios by anchoring the task and label information through sentence embeddings in fine-tuning a Sentence Transformer model. It uses contrastive learning and a triplet loss to enforce training instances of a class to be closest to its own textual semantic label information in the embedding space-and thereby learning to embed different class instances more distinct. AncSetFit obtains strong performance in data-sparse scenarios compared to existing methods across SST-5, Emotion detection, and AG News data, even with just two examples per class.

UR - http://www.scopus.com/inward/record.url?scp=85184823454&partnerID=8YFLogxK

U2 - 10.18653/v1/2023.emnlp-main.692

DO - 10.18653/v1/2023.emnlp-main.692

M3 - Konferencebidrag i proceedings

SP - 11254

EP - 11264

BT - Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

A2 - Bouamor, Houda

A2 - Pino, Juan

A2 - Bali, Kalika

PB - Association for Computational Linguistics

ER -