Semantic textual similarity models evaluation for tasks requiring binary threshold decisions

Abstract

This work evaluates Semantic Textual Similarity (STS) models for tasks requiring binary threshold decisions. The experiment used English and Ukrainian STS datasets, categorizing sentence pairs as successful or challenging based on a similarity score difference threshold. GTE demonstrated higher resistance to typos in English, while MPNet and MiniLM struggled more with lexical and morphological variations in Ukrainian. The analysis highlights the importance of considering task-specific and dataset-related challenges when selecting models for optimal performance.