A Pearson-Spearman approach for evaluating semantic similarity tasks

Abstract

This work evaluates the performance of Semantic Textual Similarity (STS) models across different language pairs and model characteristics. The comparison is conducted using both Pearson and Spearman correlation coefficients to mitigate the limitations of each. The results show that the newer models, GTE and MPNet, achieved the best performance, followed by MiniLM, which has a smaller embedding dimensionality. The findings indicate that model performance is influenced by a combination of factors, rather than a single one. Additionally, the study highlights the challenges of cross-lingual similarity assessment.