The seismic shift to AI is creating a huge data storage issue
Figure 1: ChatGPT response (bottom) to a question regarding AI is on the mark for a growing global problem.
Artificial intelligence (AI) is transforming the way we live and work, from GPS and self-driving cars to voice assistants like Siri and Alexa, to Chat GPT. However, this seismic shift to AI is creating a significant data storage issue that needs to be addressed. Currently Chat GPT collects data only from 2021, what will be the impact when older data in cold, archive storage is accessed?
Understanding the consequences
The problem is that AI requires massive amounts of data for learning to memorize and identify patterns and make predictions. Quite often, the data needs to be stored for decades to build up richer longitudinal datasets. So, as AI becomes more prevalent, data storage requirements are growing exponentially.
This data storage issue is becoming a significant challenge for many organizations, especially those that are heavily invested in AI. They need to find ways to store and manage massive amounts of data in a cost-effective and sustainable manner.
A recent article in the Independent suggested that a conversation of 20 – 50 questions with an AI chatbot may ‘drink’ 500ml of water. (1) In fact, the same article explained that in training GPT-3 alone, Microsoft may have consumed a stunning 700,000 litres of water – enough to produce 370 BMW cars or to serve one human their daily water usage for 13 years.
Current cloud solutions = catastrophic planet impacts
The current favored solution for data management is to use cloud storage services. Cloud storage providers like Amazon Web Services (AWS) and Microsoft Azure offer scalable storage solutions that can handle large amounts of data. Cloud storage can be an option since organizations only pay for the storage they use. However, the additional ingress, egress, transport and compute charges to access the data have made many customers question the overall cost-effectiveness of this model.
In addition, the Cloud comes with thunderous sustainability consequences. It’s neither carbon friendly nor water-efficient in the long run. Studies show that in one year, data centers in the US used a total 626 billion litres of water, putting pressure on precious local resources and supply. (2)
Furthermore, research indicates that saving just 1 terabyte (enough data for a small business) will produce 2 tons of carbon annually. To offset this, roughly 90 trees are needed. (3)
The increasing need for lower cost, sustainable and efficient active archiving solutions
Given the trends resulting in exponential data growth, a disruptive solution is immediately required that removes previous trade-offs.
This means we need cost-effective, sustainable, active archiving solutions.
Current solutions are not sufficient for the future needs for AI training. For example, tape storage has been the go-to solution for data archiving. However, tape storage does not help the AI/Machine Learning innovators because it is very slow and inconvenient to access data for training. Public cloud storage rapidly increases the costs because of the high ingress, egress and transport charges.
Swiss Vault Archival storage solutions
Swiss Vault has developed cutting-edge software and hardware solutions to tackle the data storage challenges. The active archive software, Vault File System (VFS), utilizes erasure coding to reduce data volume while increasing data availability and resilience. For example, a data server using VFS can tolerate higher disk losses while keeping the data available for AI/Machine Learning operations.
Using VFS in data centers, clients will immediately see a reduction in servers required for the volume of data. This correlates to lower costs, lower energy consumption and lower carbon footprint. Another intriguing property of VFS is self-healing data, which enables servers to operate until the end of their life, reducing electronic waste.
Swiss Vault’s hardware innovations are also noteworthy, as they are 10 times more energy-efficient and more space-efficient, compared to standard hyperconverged racks.
Deploying Swiss Vault’s software and hardware together will have a significant impact reducing data’s CO2 footprint.
The mission of Swiss Vault is to revolutionize data storage and provide organizations with cost-effective, resilient, and sustainable technology. There is a need for a better way for the world to manage its data, and Swiss Vault is committed to developing and offering innovative solutions.