How Apache Spark Is Transforming Big Data Processing, Development

Apache Spark is an open source data processing engine built for speed, ease of use and sophisticated analytics. Spark is designed to perform both batch processing and new workloads like streaming, interactive queries, and machine learning.

“One of the things is it improved on what was out there in two dimensions at the same time," he said. “So it was both a lot faster—like 10 to 100 times faster—and a lot quicker to program with and easier to use. So you could write 10 times less code. It’s very uncommon that you have something that’s better in both dimensions," he said.