BigOPERA: An OPportunistic and Elastic Resource Allocation for Big Data Frameworks

Abstract

The rapid growth of data-intensive applications has driven the need for more scalable, flexible, and sustainable resource management in Big Data (BD) frameworks. Traditional computing infrastructures often rely exclusively on dedicated resources, which can lead to inefficient utilization and increased operational costs. To address this challenge, this thesis explores the integration of opportunistic computing into Apache Spark through a hybrid resource allocation framework named BigOPERA. BigOPERA combines the elasticity of opportunistic nodes, machines not primarily dedicated to the cluster, with the stability of dedicated infrastructure, achieving cost-aware scalability without compromising performance. The system architecture integrates Apache Spark in standalone mode for primary data processing, Docker for containerized task isolation, and HTCondor as the orchestrator for opportunistic resource provisioning.