The Role of Fast Data in the Post-Big Data Era

Industry pundits and marketers talk a lot about big data in its various forms, discussing how to best approach what is seen as a growing resource. Data volumes will continue to grow exponentially as new data sources and systems come online, delivering upon the promise for a deeper understanding of what makes business move forward. Because big data remains such a popular buzzword, many view data in only this form, the piles and piles of stored data that data and analytics leaders must dig through in order to learn about their surroundings. So the focus remains on data volumes, ignoring what may be an even more crucial matter, data velocity.

It’s true that data is fast before it’s big. The speed at which data moves is another necessary consideration for data-driven organizations. It seems only natural that business should intertwine big and fast data to achieve an overarching scheme with which to make decisions, both in the moment and historically. As data continues to pile up, new sources open as relevant for collection in many verticals, making the practice of keeping up with this expansion burdensome.

Big data is certainly valuable. No doubt there is an incredible amount of knowledge that can be derived from historical data stores. However, in a world increasingly being flooded with self-service technology, fast data is what drives real-time decision making. When we think about the types of data that users want to analyze on a continual basis, 21st century data sources come to mind. While still expanding, these sources are largely made up of data streaming to and from business applications, sensor networks, on social media, and involving financial transactions. Modern sources of data are proliferating at a rate beyond which even the most forward-thinking technologies can comprehend.

For these reasons, many enterprise companies have turned away from legacy storage and collection technologies in favor of more agile, open source frameworks such as Hadoop and Spark. Fast data provides the digital enterprise with a unique opportunity to analyze data on the fly while it’s ingested. This allows for mass collection of data for later use, but with the caveat that newer feature enhancements provide users with the tools they need to act on an event-driven basis.

Though big data has been top-of-mind in the technology world for years now, it’s probably correct to assume that enterprises are just scratching the surface as to what’s possible with data at this scale. The benefits of utilizing data as it streams in real-time are numerous. If common practice evolves to a point where this becomes the norm enterprise-wide, access to “fresh” data could very well change the game. This data has intrinsic value, since it paints a picture of what is happening at a particular moment in time. Everyone knows that the quicker data can be acted upon, the more likely it is to have long-standing business impact.

The big data movement was largely driven by the demand for scale in the volume and variety of data, leading to an evolutionary new era in the enterprise and a different approach to data management. It seems obvious that the next step would be to make use of fast data, once and for all processing data at the speed of insight and conquering the entirety of the data spectrum.