Дослідження методів автоматичної обробки текстової інформації для створення корпусів української мови

Abstract

The current development of natural language processing (NLP) models requires large, high-quality, and verified text corpora. For the Ukrainian language, existing resources are mostly static and lack dynamic vocabulary from social networks. This work focuses on developing an automated system for creating a verified Ukrainian language corpus using Telegram data and official news websites. The methodology involves building a robust NLP pipeline for text preprocessing and applying a semantic verification algorithm. By utilizing TF-IDF vectorization and cosine similarity, the system compares unstructured messenger posts with reliable news sources. This approach filters out disinformation and noise, ensuring a high-quality dataset suitable for training modern AI models.