In this blog post, we are going to dive into some decentralized storage systems and understand how they work.
For a scalable decentralized web, one of the crucial underlying layers is its storage system. In a decentralized storage system, instead of storing all the data in a centralized server, the data is distributed into different chunks and stored inside various nodes of a peer-to-peer (P2P) network. There have been a few successful distributed file-sharing systems like Bit torrent, Napster, etc. in the past. However, these applications were not designed as infrastructure to be built upon. The advantage of using such storage is to serve the same purpose as a decentralized web (i.e., security, privacy, no single point of failure, cost-effective).
Skeps’ architecture is based on private blockchain and decentralized storage systems. Private blockchain provides efficiency and transaction privacy, and decentralized storage helps to store files securely.
In this blog post, we are going to dive into some decentralized storage systems and understand how they work.
IPFS is currently one of the most talked-about technology in the DApps ecosystem because it can replace HTTP. It is a content-addressed storage system – for every object (example files, pictures, video) stored, a unique cryptographic hash is created, and the same hash is required to fetch these objects. Similar to URI on the web, it is a distributed file system that seeks to connect all computing devices with the same system of files. It is inspired by previous successful peer-to-peer systems, including DHTs, BitTorrent, Git, and SFS.
IPFS is also one of the vital components in Skeps’ decentralized architecture. IPFS, along with asymmetric encryption, helps Skeps to share files securely between nodes.
Filecoin digital currency is created to incentivize data storage on the IPFS network. Filecoin protocol provides data storage and retrieval service via a network of independent storage providers. These storage providers do not rely on a single coordinator, where clients pay to store and retrieve data, storage miners earn tokens by offering storage, and retrieval miners earn tokens by serving data.
Swarm is also a distributed storage platform and content distribution service. It is a part of the Ethereum ecosystem for the decentralized web that consists of three components –
It is similar to IPFS in terms of using hash-based content addressing but has a built-in incentive layer. The incentive layer uses peer-to-peer accounting for bandwidth, deposit-based storage incentives, and allows trading resources for payment.
Ethereum Swarm Bee is the second official Ethereum Swarm implementation that uses libp2p as the network layer as opposed to its first implementation, which uses devp2p.
The Storj network is a robust object store that encrypts, shards, and distributes data to nodes around the world for storage. Its core technology is an enforceable peer-to-peer storage contract – a way for two people to agree to exchange some amount of storage for the money. The contract has a time duration over which the renter (storage consumer) periodically checks that the farmer (storage provider) is still available. The renter pays the farmer for each proof it receives and verifies. On completion of the duration of the contract, both parties can renegotiate or end the contract.
Storj aims to replace Amazon S3, which is currently one of the most widely used centralized cloud storage solutions, and hence, uses S3 compatible APIs.
Sia is a decentralized cloud storage platform that provides peers to rent storage from each other. Sia itself stores only the storage contracts formed between parties, defining the terms of their arrangement. A blockchain, similar to Bitcoin, is used for this purpose.
By forming a contract, a storage provider (also known as a host) agrees to store a client’s data and to periodically submit proof of their continued storage until the contract expires. The storage provider compensates for every evidence the client shares and penalizes for missing evidence. Since the evidence are publicly verifiable (and are publicly available in the blockchain), network consensus can be used to enforce storage contracts automatically. The clients do not need to verify storage proofs personally; they can upload their files and let the network do the rest.
The decentralized storage solutions discussed above are used to store static data, but web apps are highly dependent on dynamic data.
Let us discuss some database solutions for a decentralized web.
OrbitDB is a serverless, distributed, and peer-to-peer database (each peer has its own instance of a specific database). OrbitDB uses IPFS as its data storage and IPFS Pubsub to automatically sync databases with peers. A database is replicated between the peers. It automatically results in an up-to-date view of the database upon updates from any peer. In OrbitDB, the data should be stored, partitioned, or sharded based on the access rights for that data. It supports multiple data models, namely Key-Value, Log (append-only log), Feed (same as log database but entries can be removed), Documents (store indexed JSON documents), and counters.
OrbitDB databases are eventually consistent, achieved with conflict-free database merges (CRDTs).
GUN is a real-time, decentralized, offline-first, graph database. It allows for data synchronization to happen seamlessly between all connected nodes by default. The idea behind the GUN is to offer a decentralized database system that offers real-time updates with eventual consistency.
The flexible data storage model allows for tables with relations (MSSQL or MySQL), tree-structured document orientation (MongoDB), or a graph with circular references (Neo4j).
Other Dynamic databases include AvionDB, ThreadDB, and Secure Scuttlebutt.
At Skeps, we are building our customized decentralized database, ChainwolfDB. It is inspired by OrbitDB to use IPFS pubsub as the underlying layer and maintains a mongo instance on each peer to store data. MongoDB, the data layer, provides persistence and dynamic schema for the data to be shared. It uses PGP encryption for syncing data to other nodes in a secure manner. It also supports transactions that help us to make sure that either all requested nodes receive the data or none gets to act on it.