Development of a multimodal bot for digitizing and systematizing handwritten culinary recipes based on OCR and NLP

Abstract

This paper addresses the challenge of digitizing and systematizing handwritten culinary recipes, which remain vulnerable to physical deterioration and inaccessible for search or filtering in their original form. The proposed solution is a multimodal bot that implements an automated processing pipeline combining cloud-based optical character recognition, large language model-based text structuring, and storage in a relational database with fuzzy search support. The OCR pipeline includes image preprocessing using OpenCV and semantic validation through a categorized stem-based algorithm to filter non-culinary content. Text structuring is performed by the Gemini 2.5 Flash model with a self-correction mechanism limited to two iterations. The PostgreSQL database employs full-text search via TSVECTOR and trigram-based fuzzy search via pg_trgm to ensure tolerance to OCR errors and regional vocabulary variants. Experimental evaluation on a dataset of 15 handwritten recipes demonstrated that the structuring module achieves an F1-score of 0.941 for ingredient extraction and 0.848 for cooking steps, with 100% accuracy for dish category and cuisine classification, despite an average word error rate of 44.1% at the OCR stage.