RT Journal Article
T1 Ignis: An efficient and scalable multi-language Big Data framework
A1 Piñeiro Pomar, César Alfredo
A1 Martínez Castaño, Rodrigo
A1 Pichel Campos, Juan Carlos
K1 Big data
K1 Multi-language
K1 Performance
K1 Scalability
K1 Container
AB Most of the relevant Big Data processing frameworks (e.g., Apache Hadoop, Apache Spark) only support JVM (Java Virtual Machine) languages by default. In order to support non-JVM languages, subprocesses are created and connected to the framework using system pipes. With this technique, the impossibility of managing the data at thread level arises together with an important loss in the performance. To address this problem we introduce Ignis, a new Big Data framework that benefits from an elegant way to create multi-language executors managed through an RPC system. As a consequence, the new system is able to execute natively applications implemented using non-JVM languages. In addition, Ignis allows users to combine in the same application the benefits of implementing each computational task in the best suited programming language without additional overhead. The system runs completely inside Docker containers, isolating the execution environment from the physical machine. A comparison with Apache Spark shows the advantages of our proposal in terms of performance and scalability
PB Elsevier
SN 0167-739X
YR 2020
FD 2020
LK http://hdl.handle.net/10347/24656
UL http://hdl.handle.net/10347/24656
LA eng
NO Future Generation Computer Systems, Volume 105, April 2020, Pages 705-716
NO This work has been supported by MICINN, Spain (RTI2018-093336-B-C21), Xunta de Galicia, Spain (ED431G/08 and ED431C-2018/19) and European Regional Development Fund (ERDF)
DS Minerva
RD 4 may 2026