Data Availability Vision

Data Egress Node or Edge Node (as a collective descriptor for archive, observer, Access node). Edge nodes follow the chain (block headers), and maintain a (small) subset of execution state and event data.

Usage pattern that an Edge Node is designed for:

• online most of the time and continuously following new blocks and ingesting the relevant state and event data
• receive data once, verify correctness, store and index it locally
• many operations are executed on the same stored data
• if an edge node is offline for prolonged periods of time, it needs to catch up which can take some time; during the catchup period the edge node does not offer its normal services (system resources largely required for catch-up and newest data is not yet available)

‍

Light clients are largely stateless. They do not have to maintain any execution state or event data. Though, they might store a negligible amount of data locally, such as some block headers and a few accounts’ states. Data that the light client serves to its user is fetched on demand (or streamed on demand if desired by the client).

Usage pattern that a Light client is designed for:

• very sparse data access, most data is only needed once and never again
• can operate on minimal hardware, e.g. browser plugins, embedded hardware controllers (such as ESP32), or smartphones
• prolonged periods of being offline do not substantially impact the light client

‍

trusted vs trustless

Shipping correct data to the user / developer is paramount. While the Flow platform cannot universally guarantee data availability, we must unconditionally guarantee data correctness. There are two ways for guaranteeing data correctness:

(i) trusted: the user / developer trusts that the data source delivers correct data

(ii) trustless: the data source might deliver incorrect data. The user’s / developer’s software (edge node or light client) must recognize the data as incorrect during the ingestion process and drop it.

Flow is architected to scale beyond 1 million transactions per second. You can read the details in this paper.

1. A short- to mid-term goal is to support
   • the state exceeding 1 petabyte in size (snapshot at one specific height) and
   • the network producing gigabytes of event data every few seconds.
2. Trustless operations, including trustless data egress, is a central value proposition of Flow. In all likelihood, some data egress functionality will be technically intractable to implement in a trustless manner, too large of an engineering lift to implement compared to its utility, or not economically viable for clients to use due to disproportionate cost of the proofs.
   Nevertheless, we strive to keep the data egress functions that are only available via trusted sources as small as possible. The important data egress functionality must be available in a trustless manner. Otherwise, we would substantially weaken the value of Flow as a decentralized, trustless platform, if core data egress functions are available only through trusted/centralized entities.
3. We want to support trusted data egress as well, because omitting proofs substantially reduces hardware and energy consumption. Trust relationships based on legal or economic incentives are very common, and we should allow clients to utilize such to improve efficiency and decrease cost.
4. We want to enable developers to run their own edge nodes at small costs, allowing them to locally access the subset of state and event data that is relevant to them. Edge nodes must eventually allow getting their input data in a trustless manner for all prevalent use cases.
5. We want users to be able to run light clients. Light clients must eventually allow getting their input data in a trustless manner for all prevalent use cases.
6. Low-latency data access.

‍

Implications

‍

In the long-term, trustless data access cannot be contingent on maintaining the full state and/or consuming all events from blocks, as this would necessitate downloading gigabytes of data per minute and maintaining petabytes of state (for the mature network).

Therefore, for long term goal (2), we need to support trustless retrieval of a sub-set of the state and a subset of the event data.

‍

Verification and Access Nodes as tier-1 data replicas

‍

In the long term, it will not be tractable for the Execution nodes to individually supply each node in the data egress network with its desired execution state and event data. This is because there are only 8 Execution nodes in the entire network and presumably thousands of edge nodes and billions of light clients.

Conceptually, there is a well-established solution to this data replication challenge: Execution nodes replicate their data to a larger yet still limited number of “tier-1 data replicas”, which in turn replicate to the data egress network. To make this layered data replication resilient to sibyl bandwidth exhaustion attacks and faulty-data-injection attacks, the tier-1 replication layer should be staked.

Furthermore, we must account for the fact that there is no intrinsic incentive for execution nodes to supply data to the external world. Empirical evidence shows that node operators are generally willing to support the network they are part of through auxiliary services, even without direct monetary compensation, because they indirectly benefit from increased adoption. However, should the free service their node provides become a substantial cost factor, node operators will stop providing such services on a goodwill basis. Hence, it is key to keep the cost of state and event replication low enough for execution nodes so it doesn’t develop into a significant pain point.

‍

This brings us to an important insight:

Execution nodes have an immediate incentive to provide verification nodes with the necessary data. This is because without the verifier’s approvals, execution results are not sealed by consensus nodes and the execution nodes do not receive rewards. Furthermore, the cost of sending data to the verifiers is covered by the Execution node rewards. In the mature protocol, there will be about 60 to 250 verification nodes assigned to recompute each chunk. As part of their verification work, the verification nodes locally regenerate exactly the same data as the execution node. In other words, for each chunk there are at least 2/3·60=40 to 2/3·250=166 additional nodes in the network that possess exactly the data the edge nodes are interested in. Essentially, the Flow network already has the tier-1 data replicas, including an incentive mechanism for the execution nodes to supply them with data.

It is a very reasonable assumption that the cost will be low enough for supplying the edge nodes with data for free, if verification nodes also share some of the load.

Execution and Verification nodes are required by the protocol to only maintain a limited history of data

‍

The data in the execution and verification nodes has only a limited life time. The purpose of execution and verification nodes is to advance the network, but not for long-term data archiving.

Unstaked Observer and Archive Nodes as tier-2 data replicas

‍

Observer and Archive nodes are unstaked. They would get execution state data from the staked Execution, Verification and Access. As discussed above, the implicit benefits of a flourishing ecosystem probably provide significant incentive for Execution, Verification and Access Nodes to provide this service without explicit compensation, if all of these nodes share the operational load for the data egress. Furthermore, Observer and archive nodes should by default also share the data they store locally amongst each other (e.g. using the bitswap protocol, that works quite similar to bit torrent), which further reduces the load on the staked nodes (at least all data that is not absolutely the newest).

Script execution is possible on all Edge nodes (Access, Observer, Archive Node) if desired by the operator

‍

Script execution is an add-on service that can be added to any edge node. The edge node’s owner should be able to configure whether their edge node provides script execution as a service to the community, or whether they only want to offer private script execution.

Replicate data optimistically without waiting for sealing

‍

We aim to make data available to the external clients as soon as it is executed (goal 6.) Waiting for sealing is not a long-term option, as sealing by itself adds some seconds of latency.

As soon as an execution node has committed to its result (execution receipt recorded in block), we have a relatively high assurance for its correctness, because the execution node would be slashed if the data is later found to be incorrect. In most cases, this could be a sufficient guarantee for clients to accept the execution result.

Nevertheless, for safety-critical applications, the clients may want to wait for sealing. We want to make the option available to clients to choose the assurance level that provides an optimal tradeoff between safety and latency for their particular application. In particular, a client should be able to specify whether it is willing to trust some particular execution node(s) and accept their results as soon as they are available.  Another practically useful option could be for client to specify that they desire at least k execution nodes to have committed to the same result before accepting it. A very small number of available conditions would likely be already immensely useful.

Rich support for observing state transitions, i.e. utilizing events

‍

Context:

Flow is a computation platform, with the central goal to maintain an execution state and evolve this state over time through state transitions. The state (at the end of a block) is well observable through cadence scripts. But the state transitions lack observability. We have events that can expose information about the state transition (similar to log messages of conventional software).

‍

Impact (hypothesis)

In many cases, things happening on-chain are relevant for the outside world. But it is hard (conceptually complex and engineering intensive) for developers to build systems that follow what happens on-chain. Thereby, we create an incentive for a developer to move their logic off-chain into an environment they control and that is easily observable for them. Then, Flow is used like a data storage system, where all the logic happens off-chain and only the result is committed back to the chain.

However, moving logic off-chain undermines Flow’s value proposition of composability. If there is little logic on-chain, there is nothing to compose or build on-top. The value of composability critically hinges on a great platform experience for observing state transitions, i.e. supporting data-rich events.

‍

Long-term Vision:

Filtering and indexing as a service on Edge nodes including trustless event retrieval.

The following filtering functions are potentially broadly useful to address the majority of use cases:

• filtering by type of event
• secondary filtering on event fields by their value (bucketing could be a viable approach, similar to prometheus)
• Type hierarchy for events (probably requires additional cadence features). This would allow queries like “give me all NFT-mint events”. Conceptually, this is very similar to interfaces in programming languages, just applied to data structures. When a developer is specifying that their event is ‘of type NFT-mint’, the event must at least contain the data fields as specified in the abstract type (and potentially more).
• streaming support, where clients can register web hooks and respective event filters with an edge node and have the subset of events they are interested streamed to them

‍

Indexing and aggregation of events (see next section)

Native data indexing and aggregation

‍

Established solutions for data indexing in the blockchain space can broadly be divided into the following two categories:

Centralized (example: Alchemy):

• require trust
• subject to protectionism, special interests, oligopolies, etc
• very hardware and energy efficient and thereby also cost efficient
• can cover all use cases that are computationally tractable

‍

Decentralized (example: The Graph)

• allows for decentralized access
• facilitate composability
• incur higher hardware and energy overhead and thereby less cost efficient (larger redundancy is required for safety of decentralized platforms)
• overhead (higher redundancy) will be prohibitive for a noteworthy section of compute-intensive indexing and aggregation use cases

‍

Axiom:

We assume that there is utility in having decentralized data indexing and aggregation due to its decentralized and open nature as well as resilience to cyber attacks and natural disasters.

‍

Common approach for decentralized data indexing and aggregation:

Computational platform (layer-1 blockchain) is bridged to decentralized data indexing network like The Graph.

Downsides:

• bridging incurs high latency
• significant developer overhead, because they have to write code for two ecosystems
• usage overhead: original blockchain and The Graph each have their own token
• indexes and aggregates are not available to smart contracts on the original blockchain, because they live in a different ecosystem

‍

Insights:

• The layer-1 blockchain computes something. Subsequently, The Graph ingests a subset of state + events and performs indexing and aggregation. Essentially, we have a layer-1 compute platform, whose results are ingested by a layer-2 compute platform for the purpose of post-processing.
• This approach of coupling two compute platforms has developed, because blockchains are commonly much too weak to handle the computational loads and data storage required for indexing and aggregation. So people accept the substantial overhead and performance costs of coupling compute platforms. Conceptually, it is just another case of sharding.

‍

Long-term Vision

Flow is architected to handle large computational loads and store a massive state. As opposed to most (all?) other blockchain-based compute platform, Flow will have the power to run data indexing and aggregation natively within the platform. In fact, many indexing and aggregation computations are exceptionally suitable for concurrent execution, which is very effective for Flow’s large execution nodes.

Instead of burdening developers with writing code for two ecosystems, dealing with two tokens, high latency and no proper access to the indexes and aggregates from the original layer-1 smart contracts, we should provide indexing and aggregation functionality natively within Flow. This is very much in the spirit of Flow, because we don’t outsource complexity to the developer but absorb it into the platform. Thereby, the platform solves the problem once, instead of having developers solve the same problem over and over again. We increase utility of the Flow platform and free the developers to focus on their creative ideas instead of spending their time working around the platform limitations.

Current state

‍

• data is only shared to Observer and archive and once sealed
• observer and archive nodes must ingest the full state delta for each block (all or nothing)
• trustless (cryptographically verified) data ingestion for Observer/Archive node
• Observer/Archive nodes offer trusted, poll-based data access API to community

‍

Important observations

Evolution 1 (close to done)

‍

‍

• include Verification nodes as tier-1 data replicas
• Mitigations for data decay (build on top of on-disk Binary Patricia Trie)
• Edge nodes can maintain subset of state and event data in trustless manner (requires trie-based proofs, including proof of data completeness)
• trustless data sharing amongst each other
• trustless streaming of filtered events to community clients
• type system for events allows filtering events by category
• second-level filtering by field values within events (e.g. give me only NFT-mint events, where rarity is “legendary”)
  •    first version can be trusted
  •    trustless version would be nice but might be too complex (potentially requires specific cadence support where developer specifies buckets as part of event’s type definition)

• trustless script execution
• community client can select their desired level of security:
  •   require sealed result
  •   unsealed result is ok, but only if there are consistent receipts from k different execution nodes (value for k is determined by community client)
  •   unsealed result is ok as long as it is from an execution node that the client trusts (list of trusted ENs is specified by client)

‍

When envisioning light clients running on embedded hardware, including smartphones, there are three main challenges to solve:

1. The amount of CPU cycles, memory, and storage that a light client can reasonably consume is extremely limited. You probably prefer to use your smartphone storage for photos rather than gigabytes of chain data.‍
2. For a light client to be useful to an average end user, it has to deal with being offline for prolonged periods of time. Even if the light client were allowed to validate chain progress constantly by the operating system and user behavior, doing so would deplete batteries unacceptably quickly.‍
3. Lastly, the light client must be able to catch up on multiple days of chain progress quickly and efficiently. Downloading all blocks is not an option. Imagine returning from a three-day camping trip, eagerly trying to check whether the trades for some NBA Top Shot Moments have succeeded – but your phone takes 20 minutes to download and validate all the block headers it missed… and runs out of battery.

‍

These features are enabled through Consensus Follower. In a nutshell, the consensus follower is a library for tracking consensus progress trustlessly. The Consensus Follower independently applies consensus rules and locally asserts block finality. Essentially, it shields the higher-level processing logic from any malicious nodes sending invalid blocks. Unlike full consensus participants, the Followers do not need to validate block payloads in full — instead they can validate block headers only, utilizing the verification work already done by the consensus nodes.

Flow partitions time into Epochs, each one week long. Within this week, the set of staked nodes authorized to participate in network remains unchanged. If a node is slashed, the network might revoke their right to propose blocks for example, but no new nodes are added mid epoch.

At the epoch transition, the nodes authorized to run Epoch N hand the control of the blockchain over to the nodes for Epoch N+1. An intriguing feature of the Consensus Follower is that it only needs a few blocks from Epoch N, to assert the validity of any block header from Epoch N+1. The figure below illustrates this process. If you are interested how this works in more detail, we recommend this article.

‍

‍

While roughly 460,000 blocks are finalized during an epoch, a consensus Follower only needs to retrieve 0.005% of those blocks. Thereby, the Follower can fast-forward through months or even years of blocks consuming negligible computational and networking resources.

Furthermore, consider an Epoch (past or current) for which the Follower knows the authorized nodes. The Follower can trustlessly retrieve the state root hash for at an height belonging to the epoch without needing to know any ancestor blocks, by downloading a small chain segment and asserting its validity and finality. When knowing the state root hash at some block, it becomes possible to selectively retrieve part of the state in a trustless manner utilizing Merkle proofs.