For internal network functionalities, NEAR uses the following technologies within the protocol:
- Thresholded proof-of-stake
- DoomSlug
- Nightshade Sharding
- Aurora Engine
- Flux Protocol Oracle
In Q3 2022, NEAR introduced a JavaScript Software Development Kit (SDK), enabling developers to construct smart contracts on the NEAR blockchain using JavaScript. Additionally, in Q4 2022, NEAR established a partnership with Google Cloud to provide technical support to all NEAR grant recipients. This partnership with Google Cloud will provide the necessary infrastructure for NEAR's Remote Procedure Call (RPC) node provider, Pagoda, a Web3 startup platform developed by NEAR.
Consensus Algorithm
For a decentralized network of nodes/computers to function correctly, the independent network participants must agree over a shared state (e.g., who owns what on a blockchain). While doing this, the network should remain fault-tolerant with valid consensus despite imperfect information or malicious actors (Byzantine Fault Tolerance). Different blockchains implement different methods, but all are attempting to create a “consensus algorithm” that best fits their chain.
Public blockchain or distributed computer design uses consensus algorithms to convince nodes in a decentralized system to agree on the next valid state. Within the context of public blockchains—such as Bitcoin and Ethereum—this method signals that at least 51% of network nodes agree on the network's global state. In addition, a consensus algorithm often guarantees (probabilistic or deterministic) that network nodes can reach a consensus on the next valid state, even if a minimum number of nodes in the system are adversarial. In Bitcoin's case, the consensus algorithm is Nakamoto Consensus. For Ethereum, it's known as “Ethash.”
Nakamoto’s consensus requires waiting for the creation of several additional blocks to ensure transactions can't be reverted. As a result, Nakamoto chains have high uptime (they don’t go down or stall), but low transaction speed due to their probabilistic finalization guarantee. This is because Nakamoto Consensus requires waiting for “enough” blocks to be mined on top of the block, including the user’s transaction, that reorganizing or reverting the blockchain becomes economically impractical. This ensures some “economic certainty,” but never theoretic/deterministic certainty.
One issue with allowing anyone to participate in the consensus of an open network is that one malicious actor can create endless nodes, thereby creating multiple identities, as seen by the blockchain. If one person could create enough nodes, they could theoretically control the network known as a “Sybil.” For this reason, blockchains also need a Sybil-resistance mechanism in addition to its Consensus algorithm.
On the other hand, a Sybil-resistance mechanism is a process through which a decentralized system deters Sybil attacks. A Sybil assault occurs when a single node can flood the network with several identities and use them to obtain an outsized amount of power.
Ideally, each node in a decentralized system would represent one vote. If a node can impersonate multiple other nodes and get 125, 1,000, or more than 10,000 votes instead of one, then the system is vulnerable to assault. Sybil attacks are often deterred by requiring nodes to show proof of a difficult-to-fake resource (unlike online identities, which are easy to forge).
Proof-of-work - Hash power (e.g.,
Proof-of-stake - Tokens (e.g.,
Proof-of-space Time - Disk space (e.g., Chia, Filecoin, etc.)
Proof-of-authority - (e.g., Algorand
NEAR protocol uses a specific form of PoS consensus called “Thresholded proof-of-stake” (TPoS). As of 2023, there is ~470 million NEAR staked (43% of the total supply or ~57% of the circulating supply). The staking total is spread across 216 total validators.
TPoS helps the protocol increase network participation by distributing rewards more equitably, placing a maximum reward per shard based on the amount of NEAR that's staked. This helps encourage more participation throughout the network, disincentivizing larger validator pools, and secures larger parts of the network as shards with low stake counts offer higher rewards to willing validators. However, the effectiveness of TPoS could be argued. As of 2023, the top 8 staking pools in Near control ~39% of the stake, enough to disrupt any PoS chain. In early 2022, ~6 pools controlled over 33%, so it seems to be directionally improving, albeit quite concentrated.
NEAR staking distribution.
To achieve network consensus NEAR attempts to establish a large pool of participants (witnesses) to be elected as part of the consensus process. These witnesses are elected over a specific interval of time (defaulted to 24 hours) that's split into block slots (defaulted to 1,440 slots or one per minute) with 1,024 witnesses per block.
Thresholded proof of stake visualization. Source: NEAR Protocol\Medium
Per NEAR, this equates to 1,474,560 open individual witness seats. The value of each seat is entirely determined by the amount of total NEAR that's staked and locked by witnesses wishing to participate in consensus. Anyone wanting to participate in consensus must lock their designated amount of NEAR for three days.
For example,1,474,560 individual witnesses all staked 10 NEAR; each seat would cost exactly 10 NEAR. If 10 individuals stake 1,474,560 tokens each, every seat would still cost 10 NEAR, but each participant would be rewarded 147,456 seats. This enables greater network participation but does little to increase decentralization.
More specifically, incentivizing increased network participation theoretically helps distribute block validation of the NEAR Protocol. However, decentralization refers to the distribution of decision-making power across many entities. NEAR’s TPoS consensus algorithm doesn’t do this effectively. Seats are rewarded proportionately to the amount of NEAR staked per participant, so those staking large amounts of NEAR will always control the majority of the available seats.
DoomSlug Block Production
NEAR introduced its block production algorithm, DoomSlug, through a white paper released on December 31, 2019, by Alex Skidanov. DoomSlug offers advantages over block production models like PBFT or Tendermint as it enables a single round of communication between block producers to be sufficient to mint blocks.
DoomSlug functions by having participants (block producers) take turns minting and broadcasting new blocks. When a new block is received at height , participants can then endorse the block assigned to the next participant at height and so on. If there’s a delay (a predetermined period) between blocks, a secondary message could be sent to the next participant () with a recommendation to skip the previous block.
Comparing DoomSlug visually. Source: NEAR Protocol
Once a participant has either endorsements or skip messages from at least half the total participants, that block can then be produced and added to the chain. This comes with the unique and favorable advantage of preventing the network from ever stalling. No matter how slow the network is, a block endorsed or skipped by 50% of participants will eventually be either minted or skipped.
Implementation Complexity
Comparatively, DoomSlug simplifies the messaging rounds required to finalize blocks versus Tendermint or PBFT. That said, NEAR admits that because DoomSlug finalizes blocks faster than it produces them, forks are always possible until the block’s minted. On Tendermint, as long as one-third of all block producers are never corrupted, the protocol can't fork.
With DoomSlug, NEAR had to ensure its network logic was prepared to discourage forks. NEAR accomplishes this by logically identifying the longest chain as the most legitimate, though this comes with increased implementation complexity in block production. So, in this instance, DoomSlug eliminates the possibility of the network stalling but fails to reduce complexity.
Nightshade Sharding
NEAR Protocol achieves scalability through sharding, a process that distributes computational stress by partitioning data on the network into smaller pieces called “shards.” In traditional sharded blockchain models, each shard is its parallel blockchain with its validator set. NEAR has implemented a slightly modified version referred to as “Nightshade
To further grok a sharded blockchain architecture, it is critical to understand the role nodes play in a blockchain and the resource burden over time. Nodes (typically) perform three tasks in a decentralized blockchain:
- Compute: Process new transactions
- Consensus: Relay valid blocks to the other nodes in the network
- Storage: Store the state/history of the entire network
However, each of these activities places an increasing demand on nodes. As the volume of transactions increases, more computing power is required to process them. Relaying transactions and blocks increases network bandwidth requirements. As the amount of data to store increases, more storage is required.
In sharded blockchains, each shard stores only its local state and only relays transactions associated with its state. While this does reduce the resource load on nodes and improves scalability, it introduces increased complexity in the form of cross-shard interoperability. All the shards must know where all/specific states reside at all times and coordinate in validator selection/slashing. Designs like Ethereum and Polkadot have a Beacon/Relay chain at the center to help coordinate these things.
Source: Xangle
Differentiation from Beacon Chain
Nightshade takes a step away from the Beacon Chain model of sharding. In Beacon chains, multiple blockchains are by different validators, with each chain being its shard. This creates separation within the overall security of the total system as validators are separated per chain (shard). So, if there are 10 shards in a system with each shard having its validator, to achieve the same level of security as a non-sharded chain, each shard requires 10x the validators.
Contrasting Beacon Chain and Nightshade. Source: NEAR Protocol
Instead, Nightshade operates as a single blockchain, as each block contains all transactions from all shards while simultaneously adjusting all shards' whole states. No one node downloads the full state of the network. This is how the NEAR protocol performs dynamic sharding.
This has three advantages that benefit NEAR:
- There’s less opportunity for the network to fork, granting higher security
- The network operates faster as participants are only responsible for their shards
- Increased cross-shard composability
Nightshade lists the transactions within a block and splits them into chunks (one chunk:one shard). Generally, NEAR ideally has one chunk per block, though it's possible that chunks are missing due to network delays. This functionality corresponds with DoomSlug in ensuring that blocks can still be validated and added to the chain even when data is missing.
In Ethereum's case, every transaction must be confirmed by the beacon chain, which limits scalability based on the throughput of the beacon chain. The interblock sharding method of Near does not rely on a single chain for transaction validation and, therefore, does not suffer this limitation.
Stages of Nightshade
The Nightshade sharding implementation is being rolled out in stages. Nightshade was initially released as Simple-Nightshade, a phase 0 launch in November 2021 that enabled the basics of Nightshade to begin providing transactions four times as fast for validators. Phase 1 was recently completed in Q3 2022 and introduced a new form of validator on NEAR, the chunk-only producers. Hypothetically, this will add decentralization to the network by countering the network’s previous 100 validator limit.
Phase 2 is the full release of Nightshade for validators in which both the state and processing of the state will be 100% sharded. This eliminates the need for validators to track all shards, reducing the hardware requirements to make the network far more accessible to those who wish to participate directly. This rollout is projected sometime in 2023, though developers are notably behind schedule.
Lastly, Phase 3 is projected for some time in 2023 and will include the ability for the network to split and merge shards entirely based on resource use dynamically. This means that less used shards can be combined with other under-utilized shards to help the network increase overall efficiency.
Octopus Network
Octopus Network is a NEAR-based multichain crypto network that supports the launching of appchains—application-specific blockchains. The network is a bootstrapping solution for appchains to significantly lower capital costs, essentially operating as a web3 accelerator built on the NEAR blockchain network.
The Octopus Network is secured through the OCT token. OCT is used by validators native to the Octopus chain to secure the network and receive economic incentives. This doesn’t give OCT token holders any governing power or benefits for appchains built using the Octopus Network, just the native chain itself.
