BIG data management and processing paradigms: a comprehensive survey of architectures, technologies, and future directions
Abstract
The exponential growth of data generated by modern digital systems has fundamentally transformed how organizations store, process, and extract value from information. Big Data — characterized by its volume, velocity, variety, veracity, and value (the 5Vs) — has emerged as a critical research domain within computer science and information systems. This survey provides a systematic and comprehensive review of Big Data management architectures, distributed processing paradigms, storage technologies, and analytical frameworks developed over the past decade. We examine foundational technologies including the Hadoop ecosystem, Apache Spark, NoSQL database systems, data lake architectures, and real-time stream processing platforms. Additionally, we analyze the integration of machine learning pipelines with Big Data infrastructure, cloud-native deployment strategies, and emerging trends such as edge analytics and federated data processing. Our review synthesizes findings from over 150 peer-reviewed publications and evaluates each paradigm according to scalability, fault tolerance, latency, throughput, and ecosystem maturity. We identify critical open challenges and propose a research agenda for future investigation, particularly in the areas of data governance, energy-efficient processing, and privacy-preserving analytics.
Read the paper
Related studies
- Active Learning during Federated Learning for Object Detection
- Fairness-aware federated learning
- Decentralized Federated Learning
- Building a Personally Identifiable Information Recognizer in a Privacy Preserved Manner Using Automated Annotation and Federated Learning
- Preservation of Patient Level Privacy: Federated Classification and Calibration Models
- Efficient explainable federated learning
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.