The Future of Software Defined Storage Is Now

The total amount of data created, captured, copied, and consumed globally is forecast to increase rapidly, reaching 64.2 zettabytes in 2020. Over the next five years up to 2025, global data creation is projected to grow to more than 180 zettabytes[1]. In large enterprises, the amount of data generated, stored and processed has also increased rapidly, with the rise in usage of data mining, data lakes, CRM, machine data, and other logs. Businesses have found usefulness in insights obtained from such data for better decision making.

The rapid growth of data has put pressures on businesses to scale up their data storage infrastructures, and in turn consuming larger portions of their CAPEX spending on storage. Fortunately, advancements in technology of HDDs (Hard Disk Drives) has given us higher density storage, and faster transfer speeds. This translates to rapidly falling cost per gigabyte of storage over the years. This chart from Blackblaze[2] shows the rapidly falling cost per gigabyte from 2009 to 2017 :

Despite the rapidly falling prices of HDDs, the price of enterprise storage systems remain stubbornly high. This may be due to a few factors, from proprietary storage systems hardware and software, inefficient use of data protection schemes (eg RAID), the use of more expensive “enterprise” grade drives (eg SAS drives) with better reliability and transfer speeds.

The world’s largest public cloud providers and datacenters, eg the likes of AWS, Google, Facebook, etc, have built massively scalable storage systems without using any of the well-known brands of enterprise storage systems. Yet, such systems have proven to be highly scalable, reliable, and most importantly, affordable. The answer lies in the pervasive use of Software Defined Storage (SDS) technologies.

What exactly is SDS?

To answer this question, we first have to describe what is storage virtualization. According to Wiki[3], storage virtualization is “the process of presenting a logical view of the physical storage resources to a host computer system, treating all storage media (hard disk, optical disk, tape, etc.) in the enterprise as a single pool of storage.” Storage virtualization gives us the flexibility of adding capacity, eg more HDDs, into the system, without disrupting existing data, program configurations, or usage methods. Storage virtualization is akin to the ability of adding new drives (hence more capacity) to “drive C:” of your Windows system, with zero changes to how your program uses Drive C, or how you organize your files.

SDS is the superset of storage virtualization. On top of storage virtualization, SDS implements the full feature set of an enterprise storage system in software, independent of hardware systems. Hence, you can build a storage infrastructure using commonly available server systems and components. You have the ability to choose the best fit of components to suit your usage and applications. You have a choice to use x86 CPUs from Intel or AMD, or even use ARM-based CPUs. You build a hardware system with different combinations of CPUs, memory, HDDs, SSDs, NICs etc to provide the kind of performance you need.

SDS is also designed to be self-healing, with no single point of failure. Any subsystem can fail, without disruption to any applications. In fact, your software application is not even aware of any subsystem failure; it will just continue to work. For example, if a HDD or a node fails, SDS will automatically heal around the failure, rebuilding the system over time using other unused and available resources, and return it to a stable state.

SDS is also massively scalable to petabytes of storage and beyond. This is the concept of hyperscale storage (or scale-out storage). Adding new nodes will not only increase capacity linearly, it will also increase performance linearly. It is a fully distributed system with parallelization for performance. This is why there is no single point of failure.

Because SDS is inherently a distributed and parallel system, it gives us the opportunity to exploit lower costs HDDs, eg SATA instead of SAS drives. The cost per gigabyte of SATA drive is at least 30% lower than SAS. The maximum size of enterprise-grade SATA today (2021) from Seagate is 18TB. The maximum size of enterprise-grade SAS is 10TB. The lifespan (MTBF) between SATA and SAS is comparable. The disadvantage of SATA is lower data transfer speed. SAS drives has the ability to connect to 2 independent hosts, to build high-availability systems.

The disadvantages of SATA drives can be mitigated in a SDS storage infrastructure. Data in an SDS storage infrastructure is distributed over multiple nodes/drives. With parallelization, multiple I/O can be made at the same time to read/write data. With self-healing capability in SDS, there is no need to connect a drive to 2 different hosts. Using higher capacity drives, cost per gigabyte is driven down further.

Why You Should Consider SDS?

The result of a SDS storage infrastructure is that you get the lowest cost per gigabyte of storage without sacrificing reliability or performance. CEPH[4][5] is the most widely deployed SDS in the world, used by many public cloud providers, to provide a universal storage infrastructure for support of block, file and object access. It is an open-sourced project, hence it is free to use. Starview’s implementation of CEPH is an easy-to-use, highly-tuned storage appliance, built on ARM-based CPUs, to deliver storage at extremely attractive price/performance. ARM-based CPUs consume significantly less power and generate less heat, giving you tremendous TCO savings over the life-span of your storage system.

If your application demands high-performance storage IOPs, another SDS solution from Starview is based on NVMe. This has the capability to deliver upto 3M IOPs for your most demanding applications.

Connect With Starview

If you like this article, connect with us over Facebook and/or LinkedIn to receive regular updates about technologies.

[1] Statista – Volume of data/information created, captured, copied, and consumed worldwide from 2010 to 2025 https://www.statista.com/statistics/871513/worldwide-data-created/
[2] Backblaze – Hard Drive Cost Per Gigabyte https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/
[3] Wikipedia – Storage virtualization https://en.wikipedia.org/wiki/Storage_virtualization
[4] Wikipedia – CEPH https://en.wikipedia.org/wiki/Ceph_(software)
[5] CEPH – https://docs.ceph.com/en/pacific/

The Future of Software Defined Storage is Now

Starview Technologies Now Authorized to Distribute Market-Leading Fungible DPU-Based Storage Solutions in Singapore

Is RAID Dead?- Part 1

The Future of Software Defined Storage Is Now

What exactly is SDS?

Why You Should Consider SDS?

Connect With Starview