The Genesis of Spark
Increased consumer traffic, a variety of new forms of data and greater computations demanded the need for more storage and better performance. Traditional data storage methods including relational database management systems (RDBMSs) and imperative programming techniques were unable to handle the enormous amounts of data and their processing.
Google is the first to overcome below problems-
- Data collection and ingestion
- Data storage and management
- Data processing and transformation
- Data access and retrieval
Google published the white papers in a sequence to solve these issues –
The Google white papers were highly appreciated by the opensource community and served as the inspiration for the design and development of a comparable open source implementation, called
Hadoop.
- Hadoop is an open-source software framework for storing and processing large amounts of data in a distributed computing environment.
- It is designed to handle big data and is based on the MapReduce programming model, which allows for the parallel processing of large datasets.
- Its framework is based on Java programming.
- It facilitates to start with the small clusters and expand the size as you grow.
- It allows the storage capacity of 100's to 1000's of computers and use it as unified storage system
Hive:
- Many solutions have been developed over Hadoop platform by various organizations.
- Some of the widely adopted systems were Hive, Pig & HBase.
- Apache Hive is the most popular adopted component of Hadoop.
Hive offered following core capabilities on Hadoop platform –
- Create
- Databases
- Tables
- Views
- Run SQL Queries
Bringing together, Hadoop as platform and Hive as a database became very popular. But we still had other problems –
Performance - Hive SQL query performing slower than RDBMS SQL query
Ease of Development - writing MapReduce program was difficult
Language Support - MapReduce was only available in JAVA
Storage - expensive than cloud storage
Resource Management - only YARN container support, unable to use other container like Mesos, Docker , Kubernetes , etc
The point is, Hadoop left a lot scope for improvement and as a result Apache Spark came into the existence...!