Engineering Resilient Reproducible Analytical Pipelines (RAP)

A Semantic-Based Self-Healing Framework for High-Velocity Heterogeneous Data Streams

Abstract

Mission-critical telemetry systems such as clinical monitoring systems all face critical limitations in data availability, veracity and velocity. High-frequency data pipelines break easily when upstream schemas shift, sensors fail or interfaces change. Traditional pipelines rely on brittle selectors or rigid schemas. When these fail, organizations experience data blackouts, delayed decision-making and loss of situational awareness at critical points. This research proposes a self-healing Reproducible Analytical Pipeline (RAP) framework designed to autonomously mitigate schema drift without manual intervention in real time. Leveraging a container-deployed Python ecosystem and BERT Large Language Model Processing, the framework replaces static schema changes with a dynamic semantic embedding- driven reconciliation. The framework uses a cross-domain generalizable model to work in various industries. Our framework introduces a domain-agnostic ingestion interface that implements validation and normalization logic. The approach enables a unified, cross-domain resilient data ingestion, while reducing pipeline fragility and ensuring the stability of critical, high-velocity analytical workflows in mission-critical environments. To validate this framework, it has been evaluated in a motorsport telemetry environment that shows a critical amount of schema drift. It was evaluated across 9 different hardware environments to validate the technology independence of the system.