Architecture of an Automatic Speech-Based Production Task Generation Service for Flexible Robotic Systems

Abstract

This work focuses on the architecture of a service that uses language models to convert recognized speech into formalized production tasks for flexible robotic systems. The service integrates voice activity detection, speaker diarization, automatic speech recognition, and a large language model into a single pipeline. The NVIDIA NeMo framework is justified: its pretrained models separate speech fragments from non-speech segments and generate segment labels required for diarization. At the same time, its customization capabilities allow fine-tuning speaker separation accuracy for shop-floor acoustic conditions. Two processing pipelines are described. The first serves a single operator's utterances, while the second handles multi-participant discussions, where the composition of speakers is reconstructed using diarization. A six-step task generation procedure is proposed, encompassing normalization, correction of part and tooling names against a manufacturing operations directory, recognition of the required task, creation of a structured JSON object, its validation, and speaker role verification. It is shown that the task queue in the database becomes the single point of integration between humans and automated equipment. The described approach automates the manual chain of discussion, formulation, and task assignment currently existing in manufacturing environments.