1. Hadoop:
- Apache Hadoop is an open-source framework for distributed storage and processing of large datasets across clusters of commodity hardware. It includes components like Hadoop Distributed File System (HDFS) for storage and MapReduce for processing.
2. Spark:
- Apache Spark is a fast and general-purpose cluster computing system for big data processing. It provides APIs in Java, Scala, and Python for processing data in-memory and supports a wide range of data processing tasks including batch processing, streaming, machine learning, and graph processing.
3. Hive:
- Apache Hive is a data warehouse infrastructure built on top of Hadoop for querying and analyzing large datasets stored in Hadoop's HDFS. It provides a SQL-like interface called HiveQL, which allows users to write queries to analyze data using familiar SQL syntax.
4. HBase:
- Apache HBase is a distributed, scalable, and column-oriented NoSQL database built on top of Hadoop's HDFS. It provides real-time read/write access to large datasets and is well-suited for random, real-time access to data stored in Hadoop.
5. Kafka:
- Apache Kafka is a distributed streaming platform used for building real-time data pipelines and streaming applications. It is designed to handle high-throughput, fault-tolerant, and scalable ingestion of data streams from various sources.
6. Flink:
- Apache Flink is a distributed stream processing framework for big data analytics. It supports event-driven, real-time processing of streaming data with high throughput and low-latency capabilities.
7. Cassandra:
- Apache Cassandra is a distributed NoSQL database designed for handling large amounts of data across multiple commodity servers without a single point of failure. It provides high availability, linear scalability, and tunable consistency levels.
8. ZooKeeper:
- Apache ZooKeeper is a centralized service for maintaining configuration information, providing distributed synchronization, and ensuring coordination across distributed systems. It is used by many distributed systems as a coordination and consensus service.
9. Parquet:
- Apache Parquet is a columnar storage file format optimized for use with Apache Hadoop-based data processing frameworks. It provides efficient compression and encoding schemes to minimize storage and improve query performance.
10. Arrow:
- Apache Arrow is a cross-language development platform for in-memory data. It provides a standardized language-independent columnar memory format for efficient data interchange between different systems and languages.