The Future of ETL and the Argument for Spark Augmentation

In managing databases, extract, transform, load (ETL) refers to three separate functions combined into a single programming tool. Fragmentation in the legacy Data Integration market made us wonder whether or not traditional integration tools were becoming obsolete before our very eyes. With this in mind, we asked the crowd whether or not they believed Data Integration as we’ve known it was dying. To our surprise, the answer was a resounding no, and it appears that legacy tools are still being used in many verticals as enterprises prepare for the next wave in data tools.

In a recent presentation at Spark Summit EU, ING’s Chapter Lead in Analytics Bas Geerdink spoke to this very topic, recommending a migration from ETL to Apache Spark for data processing and movement. Geerdink, who is also a certified Spark developer argues that ETL has seen no real technological or market evolution like BI and the data warehouse have in recent years. ETL tools don’t seem to have a major role in the future outside of niche use cases, with this slideshow even referring to these solutions as “ETL Hell.” Make your own conclusion, and click through the presentation to learn more.

 

Follow Tim

Timothy King

Senior Editor at Solutions Review
Tim is Solutions Review's Editorial Director and leads coverage on big data, business intelligence, and data analytics. A 2017 and 2018 Most Influential Business Journalist and 2021 "Who's Who" in data management and data integration, Tim is a recognized influencer and thought leader in enterprise business software. Reach him via tking at solutionsreview dot com.
Timothy King
Follow Tim