Eric Herzog’s Jam perspective on the evolution of enterprise data storage for BUDR Insight Jam on December 13.
Lessons are to be learned from the evolution of enterprise storage – lessons that can be applied today. Let’s take a hop, skip and a jump over my four decades in the storage industry through the past, present, and future of enterprise storage.
Whether you’re a CIO, a CISO, an IT director, a tech manager, an engineer or a storage administrator, this article will put storage in a historical perspective in the tech industry, elucidate the fundamentals of storage that everyone needs to know, and share an informed perspective of where storage is headed in the enterprise market.
The key dictum that has traversed the past 40 years of storage has been the constant need for faster, bigger, better, cheaper storage. Be it the 1970’s or today, this dictum has proved to be a truism time and time again. This is due to the critical importance of storage to all organizations, no matter what the size, that have data of any kind – from the simplest types of data to the most complex datasets.
The proliferation of data requires continual innovation in the storage industry and has only made storage so much more significant in the 21st century that it’s hard to underestimate how critical the right levels of cyber storage resilience, reliability, availability, performance, and cost management are to your organization.
Although this article is not a comprehensive, in-depth treatise on every aspect of the history of storage, it will touch on major themes that are recurring and, in many cases, even more, important now than they were 40 years ago.
The Evolution of Enterprise Data Storage
Enterprise Data Storage: Past
The history of data storage for enterprise-sized organizations runs parallel to the history of computers, dating back to the 1950s. Storage started as an adjunct to the IBM mainframe when the first computers came out. Tape was the original vehicle for storage in the early mainframe days. Additionally, inside of the mainframes – and in all computers today and most storage systems was the use of RAM. A few companies even developed very expensive external RAM subsystems that could be used externally for super high-performance workloads in low-quantity tape, which was serving as primary storage.
The invention of the hard disk drive by IBM in the early 1950s in Silicon Valley revolutionized the industry. As part of that, the world saw the rise of hard disk storage subsystems that were incorporated first into mainframes, then to mini-computers, and then to UNIX systems. Over the years, hard disks became common not only inside of computers but as a connected storage subsystem. As hard drives became the predominant form of media, tape became used more often for backup and archiving, though some computer systems still relied on tape for primary storage into the 1980s. Over time, not only were hard disk systems internal to the computers and the associated, connected storage subsystems but the ability to make those hard drives removable began in the 1960s, allowing for additional data availability. Should a removable drive fail, it could be replaced.
Floppy disks, also removable, were eventually used for software applications, data, and backup. The development of floppy drives was a critical element in the evolution of the personal computer (PC). Over time there also emerged the use of optical technology as a storage medium for both servers and PCs. This included both read-write capabilities with optical but also the development in the late 1980s of a specialized optical technology – Write Once Read Many (WORM) that was used for regulatory and compliance. It was designed in such a way that a user could write to it but never erase it. This concept is highly significant because it served as the foundation for the immutable snapshots that exist today in enterprise storage.
Today’s immutable snapshots are, basically, a WORM without using a WORM drive, though delivering the security and safety of data permanence. But WORM technology originally was in the form of a removable cartridge.
Enterprise users invariably wanted more storage capacity that was cheaper and faster. This continual demand in the marketplace only intensified at the turn of the century. The rise of the Internet in the 1990s and the rapid digital transformation in the first two decades of the 21st century created the need for more innovation in storage technologies, including the game-changing technology of Redundant Arrays of Inexpensive Disk (RAID) at the end of the 1980s. Developed at the University of California, Berkeley, RAID added unparalleled performance and availability to hard disk-based external storage subsystems and created a market that today extends to over $30B annually.
Continuing to develop hard drives happened throughout the 40 years and continues today. This involved both changing the physical form factors, technical interfaces to the disks, and the spin speed of those disks. These developments changed overall system speed, system level size, easier and easier removability, and the dramatic expansion of the external storage system market.
As the need for speed continued year after year, hard drives were found to have reached their limits with 15,000 RPM spin speeds, even when deployed in the fastest of storage systems. This led to the need to search for faster media and the development of the use of flash in external storage subsystems. At first, all-flash arrays were tiny, and then the flash manufacturers realized making larger and larger capacity flash modules were essential to their acceptance in the external storage systems market.
When all-flash arrays first appeared, people said that it was a niche technology and that it would never grow. Obviously, that was nowhere close to being true. It’s reminiscent of when 15,000 RPM hard drives came out. People said it was a niche but 15,000 RPM became the standard for high-end storage systems. Now, 15,000 RPM hard drives have been wiped away by flash. As we said at the beginning of the article, the demand for bigger, faster, better, and cheaper continues to prevail year after year.
Another dimension of the evolution of storage is how storage is connected. Originally, storage was connected directly; it was a direct connect with a cable either inside of the computer or directly connected externally. In the early decades of storage, there was no such thing as a storage area network (SAN). But, as the technologies evolved, then there was the emergence of SANs, which allowed enterprises to create a network of storage devices over Ethernet or Fiber Channel to a network of servers.
SAN revolutionized the business of external storage because it allowed an enterprise to take storage from either one storage vendor or multiple storage vendors, put them into a farm of arrays, and the storage farm would be connected over a network into the enterprise’s server farm.
Over the last couple of decades, we have also seen the rise of the cloud as a destination for storing data. The concept of cloud storage has three components to it: private cloud, public cloud, and hybrid cloud. The private cloud is where the enterprise creates its own cloud (on-premises). The public cloud depends on large cloud providers. A hybrid cloud is a mix of public cloud and private cloud. Last decade, there was a huge shift for enterprises toward the public cloud. However, that has been changing in recent years.
Enterprise Data Storage: Present
We are currently operating in the era of hybrid cloud regarding enterprise storage. Today, most leaders and decision-makers at enterprises realize that there needs to be a combination of private and public cloud. There are benefits to both.
The private cloud allows enterprises to maintain control, ensure security, and keep costs lower, especially for high transactional workloads. Too many enterprises have discovered the hidden costs of the public cloud due to the networking costs of moving data back and forth for highly transactional workloads. But the public cloud is excellent for low transactional workloads. The public cloud is also great for backup, archiving, and disaster recovery.
So much of the current state of storage is rooted in the past. There is also a natural evolution of the market that has changed the way users approach storage capabilities. When data reduction technologies first came out, companies charged for it. Today, it’s free. When encryption on systems first came out, everybody charged for it. Today, encryption comes free from every storage vendor. The lesson is that as features get added, they start out at a premium but then get included.
The implication for the present day in the storage field is that there is a whole suite of exciting features that used to be expensive but are now bundled into storage solutions at no extra cost, such as immutable snapshots, replication, AIOps, and advanced security features. Enterprises have more access to rich sets of features, choices in media, and seamless integration than ever before in the history of this industry. Underscoring it all is automation.
Today’s storage is quite different than in the early days, which used command lines and then eventually GUI interfaces. Unlike in the past, autonomous automation has become a defining characteristic of present-day enterprise storage, enabling a set-it-and-forget-it approach that dramatically simplifies storage management while reducing storage OPEX.
When an enterprise adds new servers and new applications that are pointed at primary storage, an advanced storage system automatically configures it. Utilizing intelligent capabilities (such as Infinidat’s Neural Cache) and even AI, storage has evolved to the point where it takes care of itself.
Another defining characteristic of today’s storage is 100 percent availability. In the old days, four 9’s was considered good, and then five 9’s was better. Then the standard became seven 9’s, and everyone was excited. Those days are gone forever. Enterprises will no longer accept anything less than 100% availability for their primary storage. Expectations have changed.
The market’s unquenchable need for digital has driven the need for high availability. Businesses depend on it. The growth of digital has been exponential, causing data creation to explode. The worldwide analyst firm IDC predicts that global data creation and replication will experience a compound annual growth rate of 23 percent between 2021 and 2025, jumping to 181 zettabytes in 2025.
Just to put it all in perspective, the amount of data in existence across the world was 64.2 zettabytes in 2020, and it was 6.5 zettabytes in 2012. A single zettabyte is the equivalent of about 250 billion DVDs storing data.
It’s no wonder why storage needs to continue to evolve.
Enterprise Data Storage: Future
Just as it has for the past 70+ years, the dynamic of “bigger, faster, better, cheaper” will continue to shape enterprise storage in the future. Enterprises will deploy systems with a larger capacity for storage. If they haven’t transitioned to petabyte scale, they will in 2023 and beyond.
Performance will continue to be a differentiator, and enterprise users will want faster speed with lower latency. Advancements in automation will continue to make storage better, smarter, and easier to use. And enterprises will shrug off more expensive storage for more cost-effective storage, gravitating toward flexible consumption models.
A big strategy to redefine the future is storage consolidation. The evolution of enterprise infrastructure is that different arrays have been added at different times over the years, adding complexity and cost – manpower, energy, floor space, rack space, and more.
However, consolidating storage is a compelling way to make storage less expensive, freeing up resources and delivering cost savings. How the market is headed indicates that storage consolidation will be massive in 2023. It will create opportunities for enterprises to upgrade to bigger, faster, better, and cheaper storage for applications and workloads.
Consolidation will also go hand-in-hand with green IT initiatives impacting enterprise storage. Organizations will reduce their carbon footprint, power usage, and inefficiency of space utilization through storage consolidation. Going forward, IT leaders are expected to increase green IT initiatives in data centers worldwide. They are part of greater sustainability efforts that enterprises are ramping up. With fewer storage systems, organizations will also have less to recycle over time, reinforcing a positive impact on the environment.
In addition, we will see the continuation of hybrid cloud as the preferred approach for most enterprises. For the on-prem storage, buyers will look for more choice, having the flexibility to choose the media that is most appropriate for different workloads. Flash (or solid state) may be the best choice for a given enterprise for the most demanding workloads, but the same organization may use HDD for less demanding workloads, providing a better total cost of ownership and tape for very low-cost archival storage.
Enterprises will also increase their demand for guaranteed SLAs, which made news headlines recently but have not been as widely embraced as they will be. The world of the future will only increase risk, so IT decision-makers will need to be smart to grab hold of guarantees in primary storage and secondary storage to support new deployments, whether guarantees for cyber recovery or performance or whatever it is. Enterprises will increasingly shift away from the big incumbents because of the lack of guarantees, and customers will look to alternatives for guaranteed SLAs.
And I’ve saved my most important point for last because this needs to be a key takeaway about the future. Cyber storage resilience will increasingly become a major part of corporate cybersecurity strategies. Without it, enterprises will continue to leave gaps in their security postures, allowing cyber-criminals to wreak havoc. It is vital that enterprises move toward seeing cyber storage resilience as part of cybersecurity. This is part of the evolution of storage, and it will be a linchpin for organizations over the next few years that will make or break enterprises. It’s time for enterprises to secure their future with cyber storage resilience.
- Evolution of Enterprise Data Storage: Hard Disk to Ransomware - December 13, 2022
Eric Herzog
Prior to joining Infinidat, Herzog was CMO and VP of Global Storage Channels at IBM Storage Solutions, where he was responsible for worldwide product marketing and management for IBM’s storage systems, software-defined storage, hyperconverged infrastructure, and global storage channels. Herzog’s executive leadership experience also includes: CMO and Senior VP of Alliances for all-flash storage provider Violin Memory.
