author

author

Whale Research Institute: Chen Hongyi and Wang Fan

Node Research Center: Ma Xuying, Cai Chenxi, Lang Hanwei

Hash Academy: Alfred LJ

Supporting media (in no particular order)

first level title

01. What is sharding

1.1 Fragmentation solves the expansion problem of blockchain

The current blockchain expansion scheme is mainly divided into three different layers. Sharding and DAG (Directed Acyclic Graph) belong to Layer 1 that changes the architecture of the blockchain itself. Sharding is currently attracting a lot of attention. The public chain that focuses on sharding technology is favored by investment institutions. Sharding is also included in Ethereum’s official expansion plan along with Layer 2’s side chains, sub-chains, and state channels. .

1.2 The principle of sharding

Sharding is actually a traditional database technique that divides a large database into smaller, faster, and more manageable parts called data shards. In the public chain, it uses multiple network devices to obtain the function of parallel processing of transfers, thereby decentralizing the workload of those transfer verifications. This will automatically divide the network into many smaller parts, or "shard" processing, so that each small network only needs to run a smaller-scale consensus protocol. Transactions on the network will be divided into different shards, which consist of different nodes on the network. Therefore, each node only needs to process a small fraction of incoming transactions, and a large amount of verification work can be done by parallel processing with other nodes on the network. Splitting the network into shards will allow more transactions to be processed and verified at the same time. Therefore, the sharding technology uses a parallel processing method. The more nodes join, the faster the approval speed in the network will be. Simply put, sharding is to split a large task into multiple small tasks that can be processed in parallel to improve performance.

In the figure, we divide the Ethereum network into two parts. The left part L1 is the existing Ethereum main chain, and the right part L2 is each shard, each of which is an independent account space. Each shard is maintained by a dedicated node, which is equivalent to an independent blockchain. Each shard summarizes its own records and sends them to the main chain. The main chain collects summaries of each shard, and then generates main chain blocks (N+1). But the main chain collects summaries rather than specific transaction details.

first level title

02. Blockchain and sharding technology

2.1 Layers of sharding technology

The current mainstream sharding technology is divided into three levels: network sharding, transaction sharding, and state sharding, and its technical difficulty also increases in turn. The main core is that the nodes in the shards need to achieve consistency and prevent them from being controlled by malicious attackers, and an information transmission mechanism is required between shards to ensure that the status of transactions and smart contracts is consistent between different shards.

1. network fragmentation

Using randomness, the network can use the VRF method to randomly select nodes to form shards to prevent malicious nodes from occupying a certain shard. Consensus among nodes within a shard can be achieved through consensus mechanisms such as pBFT.

2. transaction sharding

1) Ledger sharding: In a UTXO-based system, the system can assign a shard according to the sender's address. This ensures that two double-spend transactions will be verified in the same shard, so the system can easily detect double-spend transactions without any cross-shard communication.

2) Cross-ledger sharding: In a non-UTXO system, in order to prevent double spending, the shards will have to communicate with each other during the verification process. In fact, since a double-spending transaction can occur in any shard, a transaction received by a particular shard will have to communicate with all other shards. This high communication cost between each other may defeat the original purpose of transaction sharding.

3. state sharding

The key to this technology is to separate the entire storage area, so that different shards store different parts; each node is only responsible for hosting its own shard data, rather than storing the complete blockchain state. Once state sharding is proposed, it comes with challenges.

In a state-sharded blockchain, only a portion of the state is kept in a particular shard. Say one of the accounts creates a transaction that pays the other account a sum. The transaction will be processed by the first shard, and once the transaction is verified, information about the new balance of the first account must be shared with the shard it resides on. This may require frequent cross-shard communication and state exchanges if the two accounts are handled by different shards. Ensuring that cross-shard communication does not outweigh the performance gains of state sharding remains an open research question.

The second challenge of state sharding is data availability. For example, for some reason, some specific shards were attacked and took it offline. Since shards do not replicate the full state of the system, the network can no longer validate transactions that depend on offline shards. Therefore, the blockchain is basically unusable in such a situation. The solution to this problem is to maintain archives or make node backups, which can help the system to recover from failures and restore data that is not available. However, this makes the nodes will have to store the entire state of the system, so this may also introduce some centralization risks.

Any sharding mechanism needs to ensure that shards are resilient against attacks and failures; the network must accept new nodes and assign these to different shards in a random fashion. However, in the case of state sharding, reassigning nodes is intractable. During a retuning of the network, problems may arise that disable the entire system before synchronization is complete. In order to prevent system outages, we must gradually adjust the network to ensure that each shard still has enough old nodes before all nodes are cleared. Similarly, once a new node joins a shard, the system must ensure that the node has enough time to synchronize with the shard state.

first level title

03. Fragmentation project analysis

3.1 List of Sharding Projects

first level title

04. Investment logic of sharding projects

4.1 Possibility of technical realization

The technical difficulty of sharding is very high, especially the state sharding has not been well solved in computer science theory, so only the world's top technical teams can hope to make a breakthrough.

4.2 Combination with other expansion technologies

Fragmentation can complement DAG, state channels, etc., and each play their own advantages to achieve overall system expansion.

4.3 Whether the service quality can reach the commercial level

Whether the scalability is improved will ultimately need to be tested by the market. At present, most sharding projects and applications are still far from commercial availability. How to solve the interconnection between different blocks among sharding projects and how to formulate reasonable smart contracts. If a better solution can be designed in these aspects, it can become a highly competitive project in this industry.

4.4 Innovation and rigor of the project

first level title

05. Future outlook of sharding technology

5.1 Technical advantages

Fragmentation is fundamentally different from traditional consensus mechanisms represented by Bitcoin and Ethereum. For sharding technology, while achieving multi-party consensus, it solves the problem of network congestion caused by the increase of nodes. For example, a recent test experiment of zillqa shows that with only 3600 nodes and 6 shards, the transaction performance of 2,000 transactions per second can be achieved. If the existing miners of Ethereum are transplanted to Zilliqa, then Zilliqa The performance can reach more than 1000 times that of Ethereum. It seems to be a simple jump from series to parallel, but it provides a greater room for optimization for the development of blockchain technology.

5.2 Development Obstacles

Due to the advantages of sharding technology, more and more practitioners have begun to pay attention to and support sharding technology. This aspect promotes the optimization of sharding technology and also brings certain problems.

On the one hand, sharding technology focuses on solving performance problems, while some participants are too obsessed with TPS competition, but ignore its security guarantee.

On the other hand, due to the popularity of sharding technology, many people blindly participate in the entrepreneurship of sharding technology, but many people have not figured out the balance between the governance and division of sharding, and simply think that the logic of sharding is Points, this lack of rigor will not only hinder the development of technology, but also hurt the market's confidence in technology, and even have more serious consequences.

first level title

Reference reading:

Reference reading:

20160527 Rootstock releases test network Bitcoin ushers in smart contracts

20170528 ASCH system, an upgraded version of Ethereum

20170708 Cross-chain technical report provided by Vitalik to R3

20180109 In-depth analysis: blockchain cross-chain technology

20180304 The top project "RSK" in 2018 will be launched soon, with the highest popularity and amazing return rate!

3. Commercial reprinting and secondary editing and reprinting are prohibited.

Vitalik Buterin 《Sharding doc》

Vitalik Buterin  《Chain Interoperaability》

Editor: Chen Wenyang

【Reprint Notice】

1. This report is an original work of Jingdata's professional data research and analysis organization [Jingzhun Research Institute], which is protected by the "Copyright Law" and enjoys the right to compile and annotate according to law;

2. Please leave a message in the background for reprinting, and you can reprint after obtaining authorization;

3. Commercial reprinting and secondary editing and reprinting are prohibited.