Автоматизація процесу вивчення японської мови на основі методів обробки природної мови

Abstract

This work is devoted to the research of automated Japanese text processing methods to generate optimal flashcards for language learning. The study addresses the complexity of Japanese scriptio continua and agglutinative grammar by comparing various NLP tools, such as MeCab, Juman++, GiNZA, LLMs, and SudachiPy. Based on multi-criteria decision making, SudachiPy was selected as the optimal tokenizer due to its balance of speed, accuracy, and flexible split modes. Furthermore, the system implements the TF-IDF metric for vocabulary prioritization and integrates with the modern FSRS spaced repetition algorithm. This approach significantly reduces the routine burden of "sentence mining" and enhances the overall efficiency of immersion-based language acquisition.