Prognosticating Latency in Directed Acyclic Task Graphs within Distributed Execution Frameworks

Abstract

—This paper investigates the challenge of estimating completion duration for computational workflows represented as directed acyclic graphs (DAGs) within distributed processing systems. A compound methodology, integrating analytical formulations with data-driven models trained on sub-graphs, is proposed to address this problem. The approach leverages feature engineering to capture workflow attributes and employs machine learning techniques for predictive modeling. Empirical validation is conducted on complex, real-world applications, and comparative assessments of various predictive models are presented. The results demonstrate the efficacy of the hybrid strategy in approximating workflow execution time, even for graphs of substantial complexity.