Qubole Pipelines Service Enables Streaming Data Pipeline Building
Qubole recently announced the launch of Qubole Pipelines Service, a new solution that enables users to build scalable streaming data pipelines. The offering lets customers build, test, deploy, monitor and manage streaming data pipelines in a single platform. Qubole Pipelines Service is designed to augment an organization’s existing data lake in a manged environment via the public cloud of their choice.
Named to the 2020 DBTA 100 and featured in our list of Data Management Vendors Our Editors Are Tracking, Qubole adds in its official press release: “With Qubole Pipelines Service, businesses can complement their existing data lake with advanced features that help them instantly capture streaming data from various sources, accelerate development of streaming applications and run highly reliable and observable production applications at the lowest cost.”
Qubole Pipelines Service includes a number of built-in connectors, a code generation wizard, dry run framework, and quick-start options that help hasten development. A pipeline can be developed “within a few minutes” without the user having to write any code. Qubole’s newest innovation utilizes Apache Spark Structured Streaming. Qubole has added enhancements like Rocksdb slate storage, direct writes, and memory pressure scheduling for reliability in building and deploying long-running streaming applications.
The solution touts a broad set of APIs and user interfaces for engineers to manage the streaming application lifecycle. Data management and consistency come via Qubole’s ACID framework to compact small files in the background while allowing concurrent read / write operations that don’t hinder performance.
In a statement about the news, Qubole co-founder and CTO Joydeep Sarma said: “The arrival of Pipelines Service not only equips data teams and engineers with the most comprehensive solution to quickly build streaming data pipelines and analyze massive streams of data And it also underscores Qubole’s longstanding tradition of providing businesses with an open data lake platform for batch and streaming analytics.”
Qubole Pipelines Services is now available on AWS, Google Cloud, and Microsoft Azure.
Learn more about Qubole Pipeline Services.