NOMT - Scaling Blockchains with a High-Throughput State Database
Esta página aún no está disponible en tu idioma.
NOMT: Scaling Blockchains with a High-Throughput State Database
Blockchain scalability has long been a critical challenge in the industry. At the Sub0 Reset 2024 event, Rob Habermeier, co-founder of Polkadot and a key contributor to its development, unveiled NOMT—a high-throughput state database optimized for modern SSDs. This innovative solution promises to revolutionize blockchain node performance by addressing the limitations of traditional state databases.
Introduction to NOMT
NOMT (Nearly Optimal Merkle Tree) is a permissively licensed, single-state, merklized key-value database designed specifically for blockchain nodes. It aims to optimize fast read access, rapid witnessing, and swift updating of state data. By supporting efficient Merkle multi-proofs of reads and updates, NOMT facilitates full block state proofs, making it an invaluable tool for interoperability and scalability in blockchain networks.
The Vision Behind NOMT
Habermeier emphasized that the vision for NOMT is to become the primary state database driver for high-performance blockchain nodes globally. The key motivation is to enable blockchains to scale much further without necessitating nodes with exorbitant amounts of RAM. Instead, NOMT leverages the advancements in modern SSDs to achieve high throughput and low latency.
Original PresentationThe Importance of State Databases in Blockchains
State databases are critical components of blockchain nodes. They store essential data such as addresses, account balances, smart contract code, and various other state information that smart contracts maintain. Every node maintains its own copy of the state database, ensuring synchronization through consensus protocols. Traditional databases like RocksDB, LMDB, and LevelDB, although robust, are general-purpose and not optimized for the specific needs of blockchain state management.
Key Features of NOMT
Specialized for Blockchain Use Cases
NOMT is specialized for blockchain applications, encapsulating all state merklization within the database itself. Unlike general-purpose databases that require additional layers for Merkle tree implementations, NOMT integrates this functionality natively. This specialization allows for optimized storage and retrieval of state data, significantly improving performance.
Optimized for Modern SSDs
Modern SSDs offer high throughput and low latency, especially for random access operations. NOMT is designed to exploit these characteristics, optimizing disk I/O patterns to maximize performance. By packing as much useful data into 4KB pages—the standard SSD read/write unit—NOMT ensures efficient use of storage and I/O bandwidth.
Supporting Large State Sizes and High Throughput
One of the core advantages of NOMT is its ability to handle large state sizes efficiently. It is designed to support billions of accounts on consumer-grade 4TB SSDs. This scalability ensures that as blockchain applications grow and user adoption increases, the underlying infrastructure can accommodate the expanding state without degradation in performance.
Full Block State Proofs and Interoperability
NOMT’s support for efficient Merkle multi-proofs enables the generation of full block state proofs. This capability is crucial for light clients, interoperability solutions, and blockchain bridges. By providing authenticated state trees, NOMT ensures that any piece of state data can be proven to be part of the state at a given block, facilitating secure cross-chain interactions.
Technical Overview
Binary Merkle Trees
NOMT utilizes binary Merkle trees for its state representation. This simple yet effective structure allows for efficient verification of state data. Each node in the tree represents a hash of its child nodes, culminating in a single root hash that represents the entire state.
Dual Database Architecture
The architecture of NOMT comprises two specialized databases:
- Bitcask: Stores the Merkle tree nodes (branches and leaves), optimized for sequential writes and fast retrieval.
- B-Tree: Manages the key-value store, mapping keys to values with efficient read and write operations.
This separation allows each database to be optimized for its specific access patterns, enhancing overall performance.
Direct I/O and Asynchronous Operations
To maximize IOPS (Input/Output Operations Per Second), NOMT employs direct I/O operations, bypassing the operating system’s caching mechanisms. This approach reduces latency and improves throughput. Additionally, NOMT takes advantage of asynchronous I/O capabilities, particularly on Linux systems using IO_uring, to handle multiple I/O requests concurrently.
Memory and Storage Efficiency
NOMT is designed to be memory-efficient, with an in-memory index that requires approximately 200MB per billion state items. The disk space usage scales linearly with the number of state items, with an estimated 150GB needed for a billion items. This efficiency ensures that NOMT can run effectively on consumer-grade hardware.
Leveraging Advances in SSD Technology
Habermeier highlighted the rapid advancements in SSD technology, noting that SSDs are becoming faster, larger, and more affordable. Modern SSDs offer:
- High IOPS: Capable of handling millions of random I/O operations per second.
- Low Latency: Read and write operations with latency as low as 40 microseconds.
- Large Capacity: Multi-terabyte SSDs are now accessible to consumers.
By aligning NOMT’s design with these hardware improvements, blockchain nodes can achieve higher performance without the prohibitive costs associated with scaling RAM.
Comparing SSDs to RAM Scaling
Scaling blockchain nodes by increasing RAM is costly and inefficient. High-capacity RAM modules are expensive and have limitations in maximum capacity. In contrast, SSDs provide a more cost-effective and scalable solution. NOMT’s optimization for SSDs allows blockchain nodes to handle large states and high transaction throughput without requiring terabytes of RAM.
Potential Performance Metrics
NOMT aims to achieve:
- 25,000 Transactions Per Second (TPS): On a state with one billion accounts using consumer-grade hardware.
- Low Hardware Requirements: Nodes can operate effectively with 4GB of RAM and a 256GB SSD.
- High Scalability: Supporting tens of billions of state items on a 4TB SSD.
These metrics position NOMT as a viable solution for blockchain networks seeking to scale to millions of users while maintaining high performance.
Future Directions and Industry Impact
The development of NOMT is funded by a grant from Sovereign Labs, with plans to integrate it into the next version of their SDK. The goal is for NOMT to power blockchain throughput across various networks and platforms. By providing a specialized, high-performance state database, NOMT has the potential to become a foundational component in blockchain infrastructure.
Habermeier expressed hope that NOMT would be adopted in substrate-based chains, Ethereum nodes, and other blockchain SDKs. Its permissive licensing and open-source nature encourage widespread adoption and collaboration.
Conclusion
NOMT represents a significant advancement in blockchain state database technology. By addressing the limitations of existing general-purpose databases and harnessing the capabilities of modern SSDs, NOMT offers a path toward scalable, high-throughput blockchain networks. Rob Habermeier’s presentation at Sub0 Reset 2024 underscored the importance of specialized infrastructure in meeting the growing demands of blockchain applications.
As the blockchain industry continues to evolve, solutions like NOMT will play a crucial role in enabling networks to scale efficiently and securely. The integration of NOMT into blockchain nodes promises not only improved performance but also enhanced interoperability and user experience.