Author: Jon Charbonneau
Original title: The Hitchhiker's Guide to Ethereum
Core points:
Ethereum is the only major protocol that aims to build a scalable unified settlement and data availability layer
Rollup scales up computation while leveraging Ethereum’s security
All roads lead to an endgame of centralized block production, decentralized trustless block verification, and censorship resistance.
Innovations such as initiator-builder separation and weak statelessness, which bring about a separation of powers (construction and validation) that can achieve scalability without sacrificing security or decentralization goals
MEV is now front and center - much of the design is planned to mitigate its harm and prevent its centralizing tendencies
Danksharding combines multiple approaches to cutting-edge research to provide the scalable base layer needed for Ethereum's rollup-centric roadmap
Table of contents
Table of contents
Part 1 The road to Danksharding
Original Data Sharding Design - Independent Sharding Proposal
Data Availability Sampling (DAS)
KZG Commitment
KZG Promise vs. Fraud Proof
Separation of initiators and builders within the protocol
Censorship Resistance List (crList)
2D KZG Strategy
Danksharding
Danksharding - honest majority verification
Danksharding - rebuilding
Danksharding - Malicious Majority Security with Private Random Sampling
Danksharding - Key Summary
Danksharding——Constraints on Blockchain Expansion
Native danksharding (EIP-4844)
Multidimensional EIP-1559
Part 2 History and State Management
Calldata gas cost reduction and total calldata limit (EIP-4488)
Qualify historical data in execution clients (EIP-4444)
restore historical data
Weak Statelessness
Verkle Tries (Verkle Tries)
status expired
Part 3 Everything is MEV's fault
Today's MEV supply chain
MEV-Boost
Committee Driven MEV Smoothing
Single-slot Finality
Single Secret Leader Election
Part 4 The Merger's Secret
The merged client
Introduction
Introduction
I've been pretty skeptical about the timing of the merger ever since Vitalik said that someone born today has a 50-75% chance of living to 3000 AD, and he hopes to live forever. But, what the hell, let’s have some fun, let’s take this opportunity to take a closer look at Ethereum’s ambitious roadmap.
(after 1000)
This article is not to be taken for granted. If you want a broad and nuanced look at Ethereum's ambitious roadmap, give me an hour and I'll save you months of work.
There is a lot to keep track of in Ethereum research, but it all ties together into one overarching goal — scaling computation without sacrificing decentralized validation.
Vitalik has a famous "endgame" saying, I don't know if you have heard it. He acknowledged that scaling ethereum requires some centralization. In the blockchain, the C letter for centralization is scary, but it is a reality. We just need to control this power with decentralized and trustless verification. There are no compromises here.
Professionals will add to the L1 and beyond. Ethereum remains incredibly secure with simple decentralized validation, while rollup inherits its security from L1. Ethereum then provides settlement and data availability, allowing rollups to scale. All the research here is ultimately aimed at optimizing these two roles, and at the same time, making full verification of blockchains easier than ever.
The following terms are repeated about seven or eight fifty-nine times:
DA – Data Availability Data Availability
DAS – Data Availability Sampling Data Availability Sampling
PBS – Proposer-builder Separation initiator and builder separation
PDS – Proto-danksharding native darksharding
DS – Danksharding an Ethereum sharding design
PoW – Proof of Work workload proof
PoS – Proof of Stake pledge proof
Part 1 The road to Danksharding
Hopefully you've heard that Ethereum has shifted to a rollup-centric roadmap. No more execution shards - Ethereum will instead optimize for data-hungry rollups. This is achieved through data sharding (a bit of Ethereum's plan) or larger blocks (Celestia's plan).
The consensus layer does not interpret the sharded data. It has only one job - making sure the data is available.
I'm going to assume you're familiar with basic concepts like rollups, fraud, and ZK proofs, and why DA (Data Availability) matters. If you're unfamiliar or just need a refresher, see Can's recent Celestia report.
Original Data Sharding Design - Independent Sharding Proposal
The design described here is obsolete, but worth a look for background. For simplicity, I'll call it "sharding 1.0".
Each of the 64 shards has individual proposals and committees that are rotated through from the set of validators. They independently verify that their shard's data is available. It won't be DAS (Data Availability Sampling) initially - it relies on an honest majority of each shard's validator set to fully download the data.
This design introduces unnecessary complexity, worse user experience, and a vector of attack. Reorganizing validators between shards can be risky.
It's also hard to guarantee that voting will be done within a single slot unless you introduce very strict synchronization assumptions. Beacon block proposals require the collection of all individual committee votes, which may be delayed.
(In the original data sharding design, each shard is confirmed by a committee vote. Voting is not always done in a single slot, and a shard can be confirmed up to two epochs)
DS (Danksharding) is completely different. Validators conduct DAS, confirming that all data is available (no more separate shard committees). A dedicated builder (Builder) uses the Beacon block and all fragmented data to create a large block and confirm it. Therefore, PBS (separation of proposals and builders) is necessary for DS to remain decentralized (building that big block together is resource intensive).
Data Availability Sampling (DAS)
Rollups publish a lot of data, but we don't want to burden the nodes to download all the data. This would imply high resource allocation, thereby compromising decentralization.
Instead, DAS allows nodes (even light clients) to easily and securely verify that all data is available without downloading all of it.
Naive solution - just check a random part from the block. If there is no problem, just sign and finish. But what if you miss a transaction that will drain all your ETH to some bad guy? Can this be safe (safu).
Clever solution - erasure code the data first. The data are expanded using Reed-Solomon codes. This means that the data is interpolated as a polynomial, which we can then evaluate elsewhere. This is a bit complicated, so let's break it down.
Don't worry about your math, here's a crash course. (I promise the math isn't that scary here - I had to watch some Khan Academy videos to write these parts, but now even I get it).
A polynomial is an expression that is the addition of a finite number of terms of the form . The number of terms represents the highest index. For example, + + - 4 is a cubic polynomial. You can reconstruct any polynomial of degree from any coordinates lying on the polynomial.
Now look at a concrete example. Below we have four data blocks (to). These data blocks can be mapped to the values of polynomials at a given point. For example, =. Then you find the minimum degree polynomial that satisfies these values. Since this is four data blocks, we can find a polynomial of cubic. We can then extend this data by adding four values ( to ) that lie on the same polynomial.
Remember that key polynomial property - we can reconstruct it from any four points, not just our original four chunks.
Back to our DAS. Now we just need to make sure that any 50% (4/8) of erasure coded data is usable. From this, we can reconstruct the entire data block.
Therefore, an attacker must hide more than 50% of the data blocks to successfully trick the DAS nodes into thinking the data is available when it is not.
After many successful random samplings, the probability that <50% of the data is available is very small. If we successfully sample erasure coded data 30 times, there is a <50% probability that it is usable.
KZG Commitment
OK, so we made a bunch of random samples, and they're all available. But we still have a question - is data erasure coded correctly? Otherwise, maybe the block producers just added 50% garbage when expanding the block, and our sampling was a waste of time. In this case, we won't actually be able to reconstruct the data.
Usually, we just commit large amounts of data by using a Merkle root. This is useful for proving the inclusion of some data in a set.
However, we also need to know that all original and extended data lie on the same low-degree polynomial. Merkel can't prove it. Therefore, if you go with this scheme, you also need fraud proofs to prevent possible mistakes.
(A Merkle root allows us to make commitments on data and extensions of data, but it cannot tell us whether they fall on the same low-degree polynomial)
Developers can solve this problem in two directions:
Celestia is going the fraud proof route. This scheme requires someone to watch, and if a block is incorrectly erasure coded, they submit a fraud proof to alert everyone. This requires standard honest minority assumptions and synchronicity assumptions (i.e., in addition to someone sending me a fraud proof, I also need to assume that I'm online and will receive it within a finite amount of time).
Ethereum and Polygon Avail are taking a new path - KZG commitments (aka Kate commitments). This removes the honest minority and synchronicity assumptions to keep fraud proofs safe (although they still exist and are used for reconstruction, as we'll get to shortly).
Other solutions are not without, but less sought after. For example, you can use ZK-proofs. Unfortunately, they are computationally impractical (for now). However, they are expected to improve over the next few years, so it is likely that Ethereum will switch to STARKs in the future, as KZG promises not to be quantum resistant.
(I don't think you guys are ready for post-quantum STARKs, but your next generation will love it)
Back to KZG commitments - these are a polynomial commitment scheme.
A commitment scheme is just a cryptographic way to make the commitment to some value provable. The best analogy is putting a letter in a locked box and passing it to someone else. The letter cannot be changed once it is in, but it can be opened with a key and attested. You commit to the letter, and the key is the proof.
In our case, we map all original and extended data onto an X,Y grid, and then find the minimum degree polynomial that fits them (this process is called Lagrangian interpolation). This polynomial is what the prover promises.
(KZG commitments allow us to make commitments on data and data extensions and prove that they fall on the same low-degree polynomial)
A few key points:
First there is a polynomial
The prover's commitment to this polynomial forms
This relies on a trusted setup of elliptic curve cryptography. As for how this works, there is a great thread from Bartek.
For any value of this polynomial , the prover can compute a proof
Speaking human words: the prover gives these fragments to any verifier, then the verifier can confirm that the value of a certain point (the value here represents the data behind) is correctly located on the submitted polynomial.
This justifies our extension to the original data since all values lie on the same polynomial
Note: the verifier does not need to use polynomials
Important properties — the size of the commitment to conform to , the proof size to , and the verification time to . Even for the prover, the generation of commitments and proofs has a complexity of only , where is the degree of the polynomial.
To put it bluntly: even if (number of values) increases (i.e., dataset grows with shard size) - the size of commitments and proofs remains constant, and the amount of work required to verify is constant.
Both promise and proof are just an elliptic curve element on Pairing Friendly Curves (here BLS12-381 is used). In this case, they're only 48 bytes each (really tiny).
Therefore, the prover's commitment to a large amount of original and extended data (represented as many values over a polynomial) is still only 48 bytes, and the proof will also be only 48 bytes
In simpler terms: Scales reasonably well
Here, KZG roots (a polynomial commitment) are analogous to Merkle roots (a vector commitment).
The original data are the values of the polynomial at to , then we expand it by evaluating the polynomial at to . All points to are guaranteed to lie on the same polynomial.
In one sentence: DAS allows us to check whether erasure-coded data is available. KZG promises to prove to us that the original data is scaled correctly, and promises all the data.
Well, all algebra ends here.
KZG Promise vs. Fraud Proof
Now that we understand how KZG works, let's step back and compare the two approaches.
The disadvantage of KZG is that it will not be post-quantum secure, and it requires a trusted initialization. This is not a cause for concern. STARKs provide a post-quantum alternative, while trusted initialization (open participation) requires only one honest participant.
The advantage of KZG is that it has lower latency than a fraud proof setup (although, as mentioned earlier, GASPER will not have a fast final result anyway), and it ensures proper code erasure without being introduced into a fraud proof Inherent synchronicity and honesty are few assumptions.
However, given that Ethereum still reintroduces these assumptions in block reconstructions, you don't actually remove them. The DA layer always needs to assume that blocks are initially available, but then nodes need to communicate with each other to put them back together. This reconstruction requires two assumptions:
You have enough nodes (light or heavy) sampling the data that they have enough power to combine the data. This is a rather weak, unavoidable honest minority assumption, and nothing to worry about.
The synchronicity assumption is reintroduced - nodes need to communicate within a certain amount of time before it can be reassembled.
Ethereum validators fully download shard data in PDS (native darksharding), whereas with DS they will only do DAS (download specified rows and columns). Celestia will require validators to download the entire block.
Note that in both cases we need synchronization assumptions for reconstruction. In cases where blocks are only partially available, full nodes must communicate with other nodes to put them together.
If Celestia wanted to move from requiring validators to download the entire data to only doing DAS, then the latency advantage of KZG would come into play (although this shift is not currently planned). Then, they also need to implement the KZG commitment - waiting for fraud proofs means greatly increasing the block interval, and the danger of validators voting for incorrectly encoded blocks will also be high.
I recommend reading the following articles for a deeper understanding of how KZG commitments work:
The (relatively easy to understand) basics of elliptic curve cryptography
Exploring Elliptic Curve Pairings by Vitalik
KZG Polynomial Commitment by Dankrad
Principles of Trusted Initialization by Vitalik
Separation of initiators and builders within the protocol
Today's consensus nodes (miners) and the merged consensus nodes (verifiers) play different roles. They build the actual blocks, which are then submitted to other consensus nodes for validation. Miners "vote" on the previous block, and after the merge, validators will directly vote on whether the block is valid or invalid.
PBS (Separation of Initiators and Builders) separates these - it explicitly creates a new in-protocol builder role. Dedicated builders will put blocks together and bid for originators (validators) to choose their blocks. This counters the centralized power of MEV.
Looking back at Vitalik's "endgame" - all roads lead to centralized block production with trustless and decentralized verification. PBS takes it a step further. We need an honest builder to serve the validity and censorship resistance of the network (two would be more effective), but the validator set needs an honest majority. PBS makes the initiator's role as simple as possible to support the decentralization of validators.
Builders receive a priority fee tip, plus any MEV they can withdraw. In an efficient market, competitive builders would bid for the full value they can extract from a block (minus their amortized costs, such as expensive hardware, etc.). All value will trickle down to the decentralized set of validators - which is exactly what we want.
The exact PBS implementation is still under discussion, but a two-slot PBS might look like this:
Builders commit to block headers while bidding
The beacon block originator selects the winning block header and bid. The originator is unconditionally paid for winning the bid, even if the builder fails to produce the block body.
The committee of attestors confirms the winning block header
The builder discloses the subject of the winning bid
A different committee of witnesses elects the winning body (if the winning builder does not present the body, vote to prove its non-existence).
The initiator is selected from the validator group using the standard RANDAO mechanism. We then use a commitment-disclosure strategy where the full body is not revealed until the block header is confirmed by the committee.
Promise-disclosure is more efficient (sending hundreds of full block bodies can overwhelm the bandwidth of the p2p layer), and it also prevents MEV theft. If a builder submits their full block, another builder can see it, figure out its strategy, incorporate it, and quickly publish a better block. Furthermore, sophisticated initiators can detect the MEV strategy used, replicating it without compensating builders. If this stealing of MEVs becomes a balancing force, it will incentivize a merger of builders and originators, so we use a commitment-disclosure strategy to avoid this.
After the initiator chooses the winning block header, it is confirmed by the committee and hardened in the fork selection rules. The winning builders then publish their winning full "builders block" body. If published in time, the next committee will certify it. If they fail to publish in time, they still have to pay the originator in full (and lose all MEV and fees). This unconditional payment makes the initiator no need to trust the builder.
The downside of this "dual slot" design is latency. Merged blocks will be a fixed 12 seconds, so without introducing any new assumptions, we need 24 seconds for a full block time (two 12-second slots). 8 second slots (16 second block time) seems like a security compromise, but research is ongoing.
Censorship Resistance List (crList)
Unfortunately, PBS gave builders a great deal of ability to review deals. Maybe the builder just didn't like you so they ignored your deal. Maybe they work so well that other builders give up, or maybe they set a high price for blocks because they really don't like you.
crLists prevents this. The specific implementation is again an open design space, but "hybrid PBS" seems to be the most popular. Builders specify a list of all eligible transactions they see in the mempool, and builders will be forced to accept blanket transactions (unless the block is full).
The initiator publishes the crList and the crList digest with all eligible transactions.
Builders create a proposed block body, then submit a bid that includes a hash of the crList digest as proof they have seen it.
The initiator accepts the winning bidder's bid and block header (they haven't seen the block body yet).
Builders publish their blocks, including proof that they have included all transactions in crList, or that the block is full. Otherwise, the block will not be accepted by the fork choice rules.
Witnesses (attestors) check the validity of published principals
There are still some important issues to be resolved here. For example, one dominant economic strategy is for the initiator to submit an empty list. That way, even builders who should have been vetted can win the auction as long as they bid the highest. There are ways around this (or there are others), I'm just emphasizing that the design here is not rock solid.
2D KZG Strategy
We saw how KZG commitments let us commit to data and prove that it was scaled correctly. However, this is a simplification of what Ethereum actually does. It will not commit all data in one KZG commitment - a block will use many KZG commitments.
We already have dedicated builders, so why not just let them create a giant KZG commit? The problem is, this requires a powerful supernode to refactor. We can accept supernode requirements for initial builds, but we need to avoid making assumptions about rebuilds. We need common entities to be able to handle rebuilds, so splitting the KZG commitment into multiple shares is fine. Given the amount of data at hand, reconstruction might even be fairly common, or an underlying assumption in this design.
To make reconstruction easier, each block will consist of m shard data encoded into m KZG commitments. If you're not smart doing this will result in a lot of sampling - you'll be doing DAS on each shard chunk to make sure it's available (requires m*k samples, where k is the number of samples per chunk).
Therefore, Ethereum will use a 2D KZG strategy. We again use the Reed-Solomon code to scale m commitments to 2m commitments.
We make it a 2D policy by extending additional KZG commitments (here 256-511) that lie on the same polynomial as 0-255. Now we just need to do DAS on the above table to ensure the availability of all shard data.
2D sampling requires 75% of the data to be available (as opposed to 50% mentioned earlier), which means we need to draw a more fixed number of samples. The previous simple version of the 1D strategy requires 30 samples, and here it will require 75 samples to ensure that the probability of reconstructing a usable block is consistent.
Sharding 1.0 (corresponding to 1D KZG commitment strategy) only needs 30 samples, but you need to sample 64 shards, and a full check needs 1920 samples. Each sample is 512 B, so it is:
(512 B x 64 slices x 30 samples) / 16 seconds = 60 KB/s bandwidth
In reality, verifiers are chosen randomly rather than checking all shards individually.
Merging blocks with 2D KZG strategy makes full DA verification extremely easy. Only 75 samples need to be selected from a single merged block:
(512 B x 1 block x 75 samples) / 16 seconds = 2.5 KB/s bandwidth
Danksharding
PBS was initially designed to blunt the centralizing forces of MEV on the validator set. However, Dankrad recently took advantage of that design realizing that it unlocked a far better sharding construct – DS.
DS leverages the specialized builder to create a tighter integration of the Beacon Chain execution block and shards. We now have one builder creating the entire block together, one proposer, and one committee voting on it at a time. DS would be infeasible without PBS – regular validators couldn’t handle the massive bandwidth of a block full of rollups’ data blobs:
PBS was originally designed to hedge the centralization power of MEV in the validator group. However, Dankrad recently took advantage of this design and came up with a better sharding scheme - DS (Danksharding).
DS utilizes dedicated builders to achieve tighter integration between Beacon Chain execution blocks and shards. We now have a builder who creates entire blocks; a proposer; and a committee that votes. DS is not feasible without PBS - ordinary builders cannot have huge bandwidth to satisfy blocks containing countless rollup blocks.
Sharding 1.0 includes 64 independent committees and initiators, allowing each shard to raise issues independently. This tighter integration allows us to ensure complete Data Available (DA) all at once. Data is still "sharded" in the black box, but from a practical standpoint, shards start to feel more like chunks of data, which is great.
Danksharding - honest majority verification
Let's take a look at how the verifier proves that the data is trustworthy:
This requires relying on a majority of honest validators - as a single validator, my columns and rows are available, not enough to give me statistical confidence that the entire block is available. We need an honest majority to draw this conclusion. Decentralized verification is important.
Note that this is different from the 75 random samples we discussed earlier. Private random sampling means that individuals with low provisioning will be able to check availability easily (eg, I can run a DAS light node and know blocks are available). However, validators will continue to use a row and column approach to check availability and guide block reconstruction.
Danksharding - rebuilding
As long as 50% of a single row or column is available, it is easily fully reconstructed by a sampled verifier. When they reconstruct any blocks that are missing in a certain row/column, they redistribute those blocks onto orthogonal lines. This helps other validators reconstruct any missing blocks from the rows and columns they intersect as needed.
The safe assumptions for reconstructing a usable block here are:
There are enough nodes performing the sample request so that they collectively have enough data to reconstruct the block
Synchronization assumption between nodes that are broadcasting their respective block shards
So, how many nodes are enough? A rough estimate requires 64,000 individual instances (over 380,000 so far). This is also a very conservative calculation which assumes no crossover of nodes run by the same validator (which is far from the case as nodes are limited to 32 ETH instances). If you sample more than 2 rows and 2 columns, you increase the chances of collectively retrieved due to crossover. This starts to scale quadratically - if validators are running with, say, 10 or 100 validators, the 64,000 requirement can drop by orders of magnitude.
DS can be set to automatically reduce the number of shard blocks if the number of online validators starts to get very low. Therefore, the security assumption will be reduced to a safe level.
Danksharding - Malicious Majority Security with Private Random Sampling
We see that the verification of DS relies on an honest majority to prove the block. I, as an individual, cannot prove that a block is usable by downloading several ranks. However, private random sampling can give this guarantee without trusting anyone. This is what happens when the node discussed earlier examines 75 random samples.
DS will not initially include private random sampling, as this is a very hard problem to solve in terms of networking (PSA: maybe you can help them!).
Note that "private" is important because if an attacker de-anonymizes you, they will be able to spoof a small number of sampled nodes. They can just return the exact chunk of data you asked for and withhold the rest. So you don't know all the data is provided just from your own sampling.
Danksharding - Key Summary
DS is very exciting, not just a good name. It finally realizes Ethereum's vision of a unified settlement and DA layer. This tight coupling of beacon blocks and shards can achieve the effect of not sharding the real ones.
In fact, let's define why it's even considered "sharded". The only sharding here is simply the fact that validators are not responsible for downloading all the data. Nothing else.
So, if you're now questioning whether this is true sharding, you're not crazy. This is why PDS (we'll get to that shortly) isn't considered a "shard" (even though it has "shard" in its name, yes, I know it's confusing). PDS requires each validator to fully download all blocks to prove their availability. DS then introduced sampling, so individual verifiers only download certain pieces of it.
Minimal sharding means a simpler design than sharding 1.0 (so faster delivery, right?). Simplified content includes:
Compared to the sharding 1.0 spec, the DS spec may have hundreds of lines of code less (and thousands less on the client side).
No more shard committees as infrastructure, committees only need to vote on the main chain
There is no need to track individual shard blob confirmations, now they are all confirmed in the main chain, or they are not.
A nice consequence of this is a consolidated fee market for data. Sharding 1.0 will make all this fragmented with different originators making different blocks.
De-sharding committees have also effectively resisted bribery. DS validators vote for the entire block once in each epoch, so the data will be immediately confirmed by 1/32 of the entire validator group (32 places in each epoch). Shard 1.0 validators also vote once per epoch, but each shard has its own committee that needs to be reorganized. Therefore, each shard is only confirmed by 1/2048 validator groups (1/32 divided among 64 shards).
As discussed, blocks combined with a 2D KZG commitment scheme also makes DAS much more efficient. Sharding 1.0 requires 60KB/s of bandwidth to check all DAs of all shards, and DS only needs 2.5KB/s.
There is also an exciting possibility for DS - synchronous calls between ZK-rollup and L1 Ethereum execution. Transactions from shard blocks can be confirmed and written to L1 instantly, since everything is produced in the same beacon chain block. Sharding 1.0 removes this possibility due to separate shard confirmations. This leaves an exciting design space that could be very valuable for things like shared liquidity (e.g., dAMM).
Danksharding——Constraints on Blockchain Expansion
Modular layers scale gracefully — more decentralization leads to more scaling. This is fundamentally different from what we see today. Adding more nodes to the DA layer can safely increase data throughput (ie have more room to allow rollup).
Blockchain scalability is still limited, but we can improve it by orders of magnitude compared to today. A secure and scalable base layer allows the implementation to be scaled rapidly. Improvements in data storage and bandwidth will also increase data throughput over time.
It is certainly possible to exceed the DA throughput envisaged in this article, but it is difficult to say where this maximum will fall. We don't have a clear red line, but can enumerate intervals where supporting certain assumptions starts to become difficult.
Data Storage - This is about DA and data retrievability. The role of the consensus layer is not to guarantee that data can be retrieved indefinitely, its role is to make data available long enough that anyone willing to download it can satisfy our security assumptions. Then, it gets dumped everywhere - which is comfortable, because the history is 1 out of N trust assumptions, and we're not really talking about that much data, that big of a scheme. However, as throughput increases, it can enter uncomfortable territory.
Validator - DAS needs enough nodes to jointly rebuild the block. Otherwise, attackers can sit around and only respond to queries they receive. If these queries provided are not enough to reconstruct the block, the attacker can withhold the rest and we are dead. To safely increase throughput, we need to add more DAS nodes or increase their data bandwidth requirements. For the throughput discussed here, this is not an issue. Still, it might be uncomfortable if the throughput increases by several orders of magnitude over this design.
Note that the builder is not the bottleneck. You need to quickly generate KZG proofs for 32MB of data, so will want a GPU or reasonably powerful CPU with at least 2.5GBit/s of bandwidth. It's a specialized role anyway, and it's a negligible business cost to them.
Native danksharding (EIP-4844)
DS is great, but we have to be patient. PDS is here to tide us over - it has implemented the necessary forward compatibility steps on a tight schedule (targeting the Shanghai hard fork) to provide orders of magnitude scaling during the transition. However, it does not actually implement data sharding (i.e. validators need to download all data individually).
Today's rollup uses L1 calldata for storage, which can last forever on the chain. However, the rollup only requires DA for some short periods of time, so there is plenty of time for anyone interested to download it.
EIP-4844 introduces a new blob-carrying transaction format, where rollup will be used for future data storage. Blobs carry large amounts of data (about 125KB), and they are much cheaper than a similar amount of calldata. Data blobs are pruned off nodes after one month, reducing storage requirements. This allows enough time to satisfy our DA security assumptions.
For extended context, current Ethereum blocks generally average around 90 KB (calldata is about 10 KB of that). PDS frees up more DA bandwidth (target ~1MB, max ~2MB) for blobs as they get pruned after a month. They don't burden nodes all the time.
A blob is a vector of 4096 field elements of 32 bytes each. PDS allows up to 16 blobs per block, while DS will increase this to 256.
PDS DA bandwidth = 4096 x 32 x 16 = 2 MiB per block, target is 1 MiB
DS DA bandwidth = 4096 x 32 x 256 = 32 MiB per block, target is 16 MiB
Each step is an order of magnitude expansion. PDS still requires consensus nodes to fully download the data, so it is conservative. DS distributes the load of storing and propagating data among validators.
Here are some goodies introduced by EIP-4844 on the way to DS:
Transaction format data carrying blobs
KZG commitment to the blob
All execution layer logic required by DS
All execution/consensus cross-validation logic required by DS
Layer separation between beacon block validation and DAS blobs
Most of the beacon block logic required by DS
Self-adjusting independent gas prices for blobs (multidimensional EIP-1559 with index pricing rules)
DS will also add in the future:
PBS
DAS
2D KZG Strategy
Proof-of-custody or similar in-protocol requirement that enables each validator to verify the availability of a specific portion of shard data per block (approximately one month)
Note that these data blobs are introduced as a new transaction type on the executor chain, but they do not impose additional burden on the executor. The EVM only looks at commitments attached to data blocks. The implementation layer changes introduced by EIP-4844 are also forward compatible with DS, no further changes are needed here. The upgrade from PDS to DS only requires changing the consensus layer.
In PDS, data blocks are completely downloaded by consensus clients. Data blocks are now referenced, but not fully encoded in the beacon block body. Instead of embedding the entire content in the body, the blob's content is propagated separately as a "sidecar". Each block has a blob sidecar, which is fully downloaded in PDS and then DAS (Data Availability Sampling) by DS validators.
We discussed earlier how to commit to blobs using KZG polynomial commitments. However, instead of using KZG directly, EIP-4844 implements what we actually use - its versioned hash. This is a single 0x01 byte (representing the version) followed by the last 31 bytes of the KZG's SHA256 hash.
We do this for EVM compatibility and forward compatibility:
EVM Compatibility - KZG promises are 48 bytes, while EVM more naturally uses 32 byte values
Forward compatibility - if we switch from KZG to something else (like STARKs for quantum resistance), promises can continue to be 32 bytes
Multidimensional EIP-1559
PDS ultimately creates a tailor-made data layer - data blocks will get their own unique fee market, with independent floating gas prices and limits. So even if some NFT project sells a bunch of monkey land on L1, your rollup data costs won't go up (although proof settlement costs will). This shows that the main cost of any rollup project today is publishing its data to L1 (rather than proofs).
The gas fee market has not changed, and data blocks have been added as a new market:
The blob fee is charged in gas, but it is a variable amount that adjusts according to its own EIP-1559 mechanism. The long-term average number of blobs per block should be equal to the stated target.
There are actually two parallel auctions here - one for computation and one for DA. This is a huge step forward for efficient resource pricing.
Some interesting designs can be seen. For example, it might be reasonable to change the current gas and blob pricing mechanism from linear EIP-1559 to the new exponential EIP-1559 mechanism. The current implementation does not average out to our target block size. Today's base fee stability is poor, resulting in observed gas usage per block averaging about 3% above target.
Part 2 History and State Management
A quick recap of the basic concepts:
History - everything that has ever happened on the chain. You can put it directly on the hard drive, since it doesn't need fast access. This is 1 of N honest assumptions in the long run.
State - a snapshot of all current account balances, smart contracts, etc. Full nodes (currently) need to have this data in order to validate transactions. It's too big for RAM, and the hard drive is too slow - it fits nicely in an SSD. The high-throughput blockchain allows this state to grow rapidly, far faster than ordinary people can maintain on their laptops. If everyday users can't hold that state, they can't fully verify it, and decentralization is out of the question.
In short, these things can get very large, so it is difficult for you to run a node, if required, the node must maintain this data. If it's too hard to run a node, we normal folks won't do it. This sucks, so we need to make sure this doesn't happen.
Calldata gas cost reduction and total calldata limit (EIP-4488)
PDS is a good step towards DS, and it fulfills many of the final requirements. Implementing the PDS within a reasonable timeframe will allow the DS schedule to be brought forward.
An easier fix to implement is EIP-4488. It's less elegant, but it still solves the urgency of the current fee. Unfortunately, it doesn't give the steps to DS, so inevitably it will be added later. If it starts to feel like PDS is slower than we'd like, it might make sense to get through EIP-4488 (which is just a few lines of code modification) quickly, and then get into PDS six months later. We are free to take our time.
EIP-4488 has two main components:
Reduced calldata cost from 16 gas per byte to 3 gas per byte
Increase the Calldata limit of 1MB per block, plus an additional 300 bytes per transaction (a theoretical maximum of about 1.4MB total)
The limit needs to be increased to prevent the worst - a block full of calldata will reach 18MB, which is far more than Ethereum can handle. EIP-4488 increases the average data capacity of Ethereum, but due to this calldata limit (30 million gas/16 gas per calldata byte = 1.875MB), its burst data capacity will actually decrease slightly.
Sustained load for EIP-4488 is much higher than for PDS, as this is still calldata vs. data blocks (which can be pruned after a month). With EIP-4488, the growth rate will increase meaningfully, but will also create a bottleneck for running nodes. Even if EIP-4444 was implemented concurrently with EIP-4488, it would only reduce execution payload history a year later. A lower sustained load of the PDS is clearly preferable.
Qualify historical data in execution clients (EIP-4444)
EIP-4444 allows customers to choose to locally prune historical data (including header, body, and receipt) older than one year. It stipulates that clients stop providing such pruned historical data at the p2p layer. Pruning historical data allows customers to reduce users' disk storage requirements (currently hundreds of gigabytes and counting).
This thing is already important, but if EIP-4488 is implemented, it will basically be mandatory (because it greatly increases the historical data). We hope this can be done in a relatively short period of time. Eventually some form of history expiration will be needed, so now is a good time to deal with it.
History is needed for full synchronization of the chain, but not for validating new blocks (only state). Thus, once a client has synced to the top of the chain, historical data will only be retrieved if explicitly requested via JSON-RPC or when a peer attempts to sync the chain. With EIP-4444 implemented, we need to find alternative solutions for these.
Clients will not be able to do a "full sync" with devp2p like they do today - instead they will "checkpoint sync" from a weakly subjective checkpoint, which they will see as the genesis block.
Note that weak subjectivity won't be an additional assumption - it's something that's inevitable with the move to PoS. Due to the possibility of remote attacks, this requires the use of effectively weak subjectivity checkpoints for synchronization. The assumption here is that clients will not sync from an invalid or old weakly subjectivity checkpoint. This checkpoint has to be within the period when we start pruning historical data (here within a year), otherwise the p2p layer will not be able to provide the required data.
This will also reduce bandwidth usage on the network as more and more customers adopt a lightweight sync strategy.
restore historical data
EIP-4444 will prune historical data after a year sounds good, while PDS will prune blobs faster (after about a month). These are necessary actions because we cannot require nodes to store all data and remain decentralized.
EIP-4488 - probably needs about 1MB per socket in the long run, adding about 2.5TB of storage per year
PDS - Target ~1MB per socket, adding ~2.5TB of storage per year
DS - Target ~16MB per socket, adding ~40TB of storage per year
But where did the data go? Do we still need them? Yes, but note that losing historical data is not a risk to the protocol - just to the individual application. Therefore, the work of the Ethereum core protocol should not include permanently maintaining all these consensus data.
So, who will store this data? Here are some potential contributors:
Individual and institutional volunteers
Block explorers (like etherscan.io), API providers, and other data services
Third-party indexing protocols such as TheGraph can create incentivized marketplaces where clients pay servers for historical data with Merkle proofs
Clients in the Portal Network (currently under development) can store random parts of the chain history, and the Portal Network will automatically direct data requests to nodes that own the data
BitTorrent, for example, automatically generates and distributes a 7GB file containing blob data for each day's blocks
Certain application protocols (such as rollup) can require their nodes to store parts of the history relevant to their application
The long-term data storage problem is a relatively easy problem because it is one of the N trust assumptions, as we discussed earlier. This problem is many years away from becoming the ultimate limit to blockchain scalability.
Weak Statelessness
Ok, we've got a good handle on managing history, but what about state? This is actually the main bottleneck in improving the TPS of Ethereum at present.
Full nodes take the pre-state root, execute all transactions in a block, and check that the post-state root matches what they provided in the block. In order to know whether these transactions are valid, they currently need to verify the state on the hand.
Going stateless - doesn't need the state at hand to play its role. Ethereum is working towards being "weakly stateless", meaning that state is not required to validate blocks, but is required to build blocks. Validation becomes a pure function - give me a completely isolated block and I can tell you if it works. Basically like this:
Builders still need state due to PBS, which is acceptable - they'll be more centralized high-configuration entities anyway. Our focus is on the decentralization of validators. Weak statelessness creates a little more work for builders and a lot less work for verifiers. Great deal.
We use witnesses to achieve this magical stateless execution. Witnesses are proofs of correct state access and builders will start including these proofs in every block. You don't actually need the entire state to validate a block - you only need the state that was read or affected by the transactions in that block. Builders will start including pieces of state affected by transactions in a given block, and they will use witnesses to attest that they correctly accessed those states.
Let's take an example. Alice wants to send 1 ETH to Bob. In order to validate the block for this transaction, I need to know:
Before transaction - Alice has 1 ETH
Alice's public key - so I can know the signature is correct
Alice's nonce - so I can know transactions were sent in the correct order
After executing the transaction, Bob gained 1 ETH and Alice lost 1 ETH
In a weakly stateless world, builders add the aforementioned witness data to blocks and attest to its accuracy. Validators receive the block, execute it, and decide whether it is valid. That's ok.
From a validator perspective, here are some implications:
Gone is the huge SSD requirement needed to maintain state - a critical bottleneck to scaling today.
Bandwidth requirements will increase a bit since you are now also downloading witness data and proofs. This is a bottleneck for Merkle-Patricia trees, but not a big problem, not the kind of bottleneck encountered by Verkle Tries.
You still execute transactions to fully validate. Statelessness acknowledges the fact that this is currently not the bottleneck in scaling Ethereum.
Weak statelessness also allows Ethereum to relax its self-imposed constraints on execution throughput, and state bloat is no longer a pressing concern. It might be reasonable to increase the gas limit by a factor of 3.
At this point, most users' execution will be on L2, but higher L1 throughput is beneficial even for them. Rollup relies on Ethereum for DA (publishing to shards) and settlement (requires L1 execution). As Ethereum expands its DA layer, the amortized cost of issuing proofs may take up a larger share of the rollup cost (especially for ZK-rollup).
Verkle Tries (Verkle Tries)
We've intentionally skipped how these witnesses work. Ethereum currently uses Merkle-Patricia trees to store state, but the required Merkle proofs are too large for these witnesses to be feasible.
Ethereum will turn to Verkle tries to store state. Verkle proofs are much more efficient, so they can be used as viable witnesses to achieve weak statelessness.
First let's review what a Merkle tree looks like. Every transaction starts with a hash - these hashes at the bottom are called "leafs". All hash values are called "nodes", which are the hash values of the two child nodes below. The resulting hash is the "Merkle root".
This data structure is very helpful in proving the integrity of transactions without downloading the entire tree. For example, if you want to verify that transaction H4 was included, you need H12, H3, and H5678 from the Merkle proof. We have H12345678 from block header. So a lightweight client could ask a full node for these hashes, and hash them together according to the route in the tree. If the result is H12345678, then we have successfully proved that H4 is on the tree.
But the deeper the tree, the longer the route to the bottom, so you need more items to justify. Therefore, shallow and wide trees are more suitable for efficient proofs.
The problem is that if you try to make the Merkle tree wider by adding more children under each node, it will be very inefficient. You need to hash the hash values of all peer nodes together to touch the whole tree, so you need to accept more hash values of peer nodes for Merkle proofs. This will make the size of the proof huge.
This is what efficient vector promises do. Note that the hashes used in Merkle trees are actually vector commitments - they're just bad commitments that can only effectively commit to two elements. So we want vector commitments where we don't need to receive all peers to verify it. Once we have this, we can make the tree wider, and reduce its depth. This is how we achieve efficient proof size - reducing the amount of information that needs to be provided.
A Verkle trie is similar to a Merkle tree, but it commits its children using efficient vector commitments (hence the name "Verkle") instead of simple hashes. So the basic idea is that each node can have many children, but I don't need all of them to verify the proof. This is a proof of constant size regardless of width.
In fact, we've covered a good example of this possibility before - KZG commitments can also be used as vector commitments. In fact, this is what the ethereum developers originally planned to use here. They later turned to the Pedersen commitment to accomplish a similar role. It will be based on an elliptic curve (here referring to Bandersnatch), promising 256 values (much better than two!).
So why not build a tree of depth 1, as wide as possible? This is great for the verifier because he now has a super compact proof. But there is a practical trade-off that verifiers need to be able to compute this proof, and the wider it is the harder it is. Therefore, Verkle tries will lie between the two extremes of 1~256 value width.
status expired
Weak statelessness removes state inflation constraints from validators, but state doesn't magically disappear. Transaction costs are capped, but they impose a permanent tax on the network by increasing state. State growth remains a permanent drag on the network. We need to do something to fix this fundamental problem.
That's why we need state expiration. Long periods of inactivity (like a year or two) get axed, even things that block builders would have included. Active users won't notice anything change, and we can discard heavy state that's no longer needed.
If you need to restore expired status, you just need to show a proof and reactivate it. This falls back to one of N storage assumptions. As long as someone still has the full history (block explorers, etc.), you can get what you need from them.
Weak statelessness will weaken the immediate need for state expiration in the base layer, but is good in the long run, especially as L1 throughput increases. For high-throughput rollups, this would be a more useful tool. The L2 state will grow at such a higher rate that it will drag down even high configuration builders.
Part 3 Everything is MEV's fault
PBS is a necessary condition for the safe implementation of DS (Danksharding), but please remember that its original design is actually to counter the centralized power of MEV. You’ll notice a recurring trend in Ethereum research today — MEV is now front and center in cryptoeconomics.
When designing a blockchain, taking MEV into consideration is key to maintaining security and decentralization. The basic protocol-level methods are:
Mitigate harmful MEVs as much as possible (e.g. single slot finality, single secret leader selection)
Democratize the rest (eg, MEV-Boost, PBS, MEV smoothing)
The remainder must be easily captured and propagated among validators. Otherwise, it will centralize the validator group by being unable to compete with sophisticated searchers. This is exacerbated by MEV being a much higher percentage of validator rewards after the merger (staking issuance is much lower than the inflation given to miners). This cannot be ignored.
Today's MEV supply chain
Today's sequence of events looks like this:
Mining pools play the role of builders here. MEV Seekers forward bundles of transactions (along with their respective bids) to mining pools via Flashbots. The mining pool operator aggregates a complete block and passes the block header to each miner. Miners use PoW to prove the block and give it weight in the fork selection rule.
Flashbots are here to prevent the vertical integration of the entire stack - which would open the door to censorship and other nasty outsiders. When Flashbots were born,
