Keywords

1 Introduction

Scalability is one of the well-known bottlenecks in the development of blockchain technology. The scalability of a blockchain refers to its ability to process more transactions per unit of time. Many blockchain platforms, such as Bitcoin [1] and Ethereum (ETH) [2] (about seven transactions per second for Bitcoin and about fifteen for Ethereum), are inefficient in processing transactions per unit of time. The scalability of blockchain has become increasingly important as blockchain is being used in different scenarios [3,4,5]. Blockchain platforms process transactions much slower than VISA, which processes 56,000 transactions per second [6]. The scalability of blockchain has been studied extensively to increase transaction rates [7, 8]. Shading technology is the most potential technology to improve blockchain scalability among many scaling schemes. For blockchains to become scalable, various sharding solutions have been introduced in [9,10,11,12,13].

Sharding is implemented on-chain, making it part of the layer-1 scaling solution [7]. Scaling on layer 1 has two directions, vertical scaling and horizontal scaling [14, 15]. Vertical scaling of a chain involves increasing the transaction capacity within each block or shortening the block period. Vertical scaling increases transaction throughput, but it requires more bandwidth from nodes. As a result, vertical scaling methods are not feasible for nodes with limited bandwidth. Sharding is a horizontal scaling scheme that divides the blockchain into multiple parts. Each independent chain only responds to its transaction mining. As the number of shards increases, transaction throughput increases linearly [16].

Blockchain sharding increases transaction throughput, but cross-shard transactions also increase during sharding. Due to excessive cross-shard transactions, the sharding technique cannot scale efficiently in the blockchain [17, 18]. Transactions between senders and recipients on different shards are termed cross-shard transactions, while intra-shard transactions are transactions within the same shard. Cross-shard transactions require multi-phase protocols to verify their authenticity. A multi-phase protocol typically consists of a prepare and a commit phase [19]. Since cross-shard transactions take longer to process and require more processing power than intra-shard transactions, sharding in the blockchain is not as effective in improving scalability when cross-shard transactions are involved. In [10], more than 95% of transactions are cross-shard transactions. Thus, reducing cross-shard transactions with efficient shard allocation strategies is a direct way to optimize sharding.

This paper intends to reduce the transactions between shards by allocating frequent blockchain node pairs to the same shard. We transform the problem of obtaining the least number of cross-shard transactions into a graph partition problem. A weighted and undirected graph is constructed using the number of transactions between nodes. Then, we propose generating the community result for assigning nodes to shards using a community detection algorithm. We demonstrate our scheme by testing its reliability and determining that cross-shard transactions are reduced to 20%. The following is a summary of our contributions.

  • We convert the blockchain sharding challenge into a graph partitioning problem. The sharding optimization can be transformed into the community detection problem, where nodes are tightly linked with nodes in the same community and loosely connected with nodes in other communities.

  • We propose a new sharding scheme using community-detection algorithms. The scheme is achieved by designing an undirected and weighted graph to capture transactions where the weight of an edge represents the number of transactions between the endpoints. The graph is then divided into communities by running the community-detection algorithm, and each community forms a shard.

  • Validated by comprehensive experiments on Ethereum transaction data, the proposed sharding scheme can significantly reduce the number of cross-shard transactions, e.g., from 80% to 20%, compared with popular random sharding schemes. Experiment results also confirm the stability of the proposed sharding scheme.

This paper is organized as follows. Section 2 introduces related blockchain sharding works. Section 3 depicts the proposed community detection-based sharding scheme. The experiment results are presented in Sect. 4, followed by conclusions in Sect. 5.

2 Related Work

Assigning nodes to shards is commonly accomplished by adding Distributed Randomness Generation (DRG) protocols to enhance the randomness of the assignment process. Elastico [9], Omniledger [10], and RapidChain [11] all use DRG protocols to randomly allocate nodes to shards. In [9], nodes are assigned to a shard according to the last few bits of the solution of the Proof of Work (PoW) puzzle. The PoW puzzle is derived from the combination of the epoch randomness, which is generated with a DRG protocol [20], node identity, and nonce.

Kokoris et al. [10] develope Omniledger, a distributed ledger where validators are randomly assigned to shards and synchronize the previous shard state. Each shard has a leader who is elected by a verified random function. The leader generates a randomness output for deriving a random permutation with other nodes by initiating the RandHound DRG [21] protocol. In subsequent allocations, no more than \(\frac{1}{3}\) nodes at the beginning of the permutation are shuffled randomly to other shards. As with [9, 10], Zamani et al. [11] propose a sharding-based blockchain protocol RapidChain. RapidChain uses different random allocation methods for nodes. In [11], the reference committee (a special shard) uses the Feldman Verifiable Secret Sharing (VSS) DRG [22] protocol to generate unbiased random outputs. The PoW puzzles are generated by random outputs and need to be solved by the nodes to join the system. Each node is then randomly assigned to each shard by a member of the reference committee executing Commensal Cuckoo rule [23]. With the Commensal Cuckoo, each node is mapped in a random position of Interval [0, 1) based on its identity by a hash function. Interval [0, 1) are then divided into regions representing shards. As new nodes are added, existing nodes in the same region are moved to new random regions.

Without a DRG protocol, node allocation to shards can also be achieved according to addresses’ prefixes, such as Monoxide [12]. In addition to allocating nodes to shards based on addresses, nodes can be divided by voting from other nodes. For instance, Chainspace [13] permits nodes to move between shards based on the votes of other nodes. Through a smart contract called ManageShards, voting is carried out. By deploying different shards on different channels, Fabric [24, 25] address sharding can be achieved. A trusted entity is used for cross-shard transactions.

Nodes can be assigned to shards using effective partitioning methods for reducing cross-shard transactions. Classical graph partitioning algorithms like Kernighan-Lin [26] can be used to optimize the sharding process. The Kernighan-Lin algorithm divides a graph into two communities of known size and exchanges any two vertices between them to obtain two communities with the smallest cut set size. The Kernighan-Lin algorithm causes multiple partitions since it divides two communities at once. Nevertheless, a multi-partitioned system will increase algorithm complexity.

Existing blockchain sharding schemes, including random sharding, voting-based sharding, and channel-based sharding, have not tried to improve the scalability of sharded blockchains by reducing the number of cross-shard transactions. Graph partitioning and community detection algorithms are promising to reduce cross-shard transactions but have yet to be developed for blockchain sharding schemes, especially for blockchains craving high scalability.

3 Proposed Community Detection-Based Sharding Scheme

We propose a novel blockchain sharding scheme using the community detection algorithm. An adjacency matrix representing transactions (TXs) between node pairs is used as the input parameter for the system model we presented. Our model uses the Louvain algorithm in community detection [27] to obtain communities that respond to the sharding result. All nodes are assumed to be trustworthy in the model. Under our assumption, cross-shard transactions are reduced to a lower frequency based on our model. Therefore, our community detection-based sharding model is more suitable for permissioned chains since security issue is out of scope. Notations used in this paper are collected in Table 1.

The blockchain sharding network is reviewed as an undirected weighted graph \(G = (V, E)\). The set of vertices V represents node addresses, and the set of edge E represents the transaction number between node pairs. This G is incorporated into the community detection algorithm using the adjacency matrix format.

Figure 1 illustrates the sharding system we use to reduce \(R_c\). This system performs four stages: Graph generation, Community detection, Community-based sharding, and Chain extension. The following Eq. 1 calculates the ratio of cross-shard transactions.

$$\begin{aligned} R_c = \frac{\phi _c}{\phi _i+\phi _c}. \end{aligned}$$
(1)
figure a
Fig. 1.
figure 1

The flow diagram of the proposed Re-sharding system comprising the following four stages. Graph generation: generate an adjacency matrix according to the number of transactions between nodes. Community detection: run the community detection algorithm to identify node communities so that nodes frequently exchange with nodes in the same community and less transact with nodes in different communities. Community-based sharding: allocate nodes to shards according to detected communities, where nodes in the same community are in the same shard, and implement the sharding result. Chain extension: extend the chains in parallel.

Full size image
Table 1. notation and definition in the blockchain sharding system.
Full size table

STAGE 1: Graph Generation (Algorithm 1 lines 1–7). The model generates an adjacency matrix \(\mathcal A\) based on the number of transactions between nodes. As shown in Algorithm 1, the adjacency matrix is initialized with zero. The algorithmic flow starts with obtaining the dataset of the node \(\mathcal N\) that sends and receives transactions from the selected block epoch. Then, node dataset \(\mathcal N\) are sorted based on the number of transactions to a sorted node dataset \(\mathcal N_s\). An adjacency matrix \(\mathcal A\) is generated after traversing all node sets. The size of the \(\mathcal A\) depends on the node number. The i-th row and the j-th column in matrix \(a_{i,j}\) represents the number of transactions between nodes \(N_i\) and \(N_j\). Matrix diagonal elements represent the number of transactions between nodes and themselves.

STAGE 2: Community Detection (Algorithm 1 lines 8–9). A community set \(\mathcal C\) is generated by the community detection algorithm using an adjacency matrix \(\mathcal A\) and a fixed number of shards s as input parameters. An algorithm for detecting communities facilitates more frequent exchanges between nodes within a community and less exchange between nodes from different communities. To measure the quality of the sharding result, we define a cut weight ratio r. Cut weight ratio is calculated by dividing the sum of edge weights crossing communities by the sum of all edge weights.

STAGE 3: Community-based Sharding (Algorithm 1 lines 10–11). Based on the optimized outputs of the community set \(\mathcal C\) from previous stages, sorted nodes are assigned to the new communities and the filled shard set \(\mathcal S\) is returned. Filled shards set \(\mathcal S\) with sorted nodes and a cut weight ratio r are returned at the end of each epoch.

STAGE 4: Chain Extension (Algorithm 1 lines 12–21). As the new shards returns, each chain mines blocks and extends in parallel until it reaches the end of an epoch. According to returned sharding results, the number of intra-chain transactions \(\phi _{i}\) and the number of cross-chain transactions \(\phi _{c}\) are separately counted. According to Eq. 1, the statistical results of both transactions are used to calculate a ratio of cross-shard transactions \(R_c\). We set a threshold \(\rho \) for cross-shard transactions in our system. The threshold determines whether or not a node needs to be reassigned during a block epoch \(\mathcal E_{t}\). A node is not re-allocated if \(R_c\) is less than the threshold. Otherwise, nodes are re-allocated to other shards based on the community detection algorithm. Additionally, the value of \(\rho \) is based on the largest \(R_c\).

4 Experiments

4.1 Experiment Settings

An experimental framework is implemented in a local environment (MacBook Pro with 2.5 GHz Quad-Core Intel Core i7 and 16 GB memory) to evaluate a proposed blockchain sharding scheme. We create a virtual machine using Conda on Visual Studio Code and implement our framework in Python 3.10.44. Ethereum block data ranging from 13.7 million to 15.04 million are downloaded and sorted from the Ethereum public endpoint [28]. There are 1.34 million blocks in the captured block range. Experiments are conducted on consecutive 100,000 blocks randomly selected from captured block ranges. Our randomly selected block period is between 14 million and 14.9 million. We divide the selected block period into ten equal parts, each containing 10,000 blocks.

4.2 Data Overview

We have 762,203 node addresses and 3,735,641 transactions during our first test block epoch. The community detection algorithm uses the top 90 addressesFootnote 1 sending the most transaction numbers out of 762,203 addresses. We define K as the number of addresses with the most transactions in subsequent tests. After sorting the first 90 nodes, we see that the address with the most transactions belongs to OpenSeaFootnote 2 In our first tested 10,000 blocks, OpenSea generates 154,879 transactions. Besides OpenSea, big companies like CoinbaseFootnote 3 and UniswapFootnote 4 also have large numbers of transactions on Ethereum. According to the adjacency matrix, 31 nodes do not have any transactions with any of the first 90 nodes. The row sum of all the 31 sender addresses in the matrix is zero. Thus we remove these 31 addresses, and the top 90 most transacted addresses form a 59\(\times \)59 adjacency matrix.

Fig. 2.
figure 2

Testing Ethereum address in sharding framework under unbalanced random allocation method, balanced random allocation method, and community detection-based allocation method while fixing tested addresses number \(K=90\) and shard number \(s=4\).

Full size image

4.3 Sharding Analysis

The comparison between a community detection-based sharding method and a random sharding method is shown in Fig. 2. In Fig. 2(a), the vertical axis represents the number of cross-shard transactions, while the horizontal axis represents the Ethereum block sequence number. Figure 2(b) differs from Fig. 2(a) in that the vertical axis represents the ratio of cross-shard transactions, while the horizontal axis remains the same. During the first block epoch, we divide the 59 addresses into four shards, which generate 76,369 transactions. As a result of adopting the community detection algorithm, 65,703 transactions are intra-shard transactions, and 10,666 are cross-shard transactions.

To compare with the community detection-based sharding scheme, we simulate a process of randomly assigning addresses, similar to the process of nodes randomly assigned to varying shards in Omniledger [10]. Nodes can be randomly allocated into shards of the same size. Alternatively, nodes can be randomly divided into shards of different sizes. Therefore, we develop two methods for randomly allocating nodes to shards in the experiments. Keeping each shard with the same number of nodes is balanced sharding. Sharding with more or fewer nodes in each shard is unbalanced.

As shown in Fig. 2(a), the proposed sharding scheme keeps the number of cross-shard transactions around 10,000. Additionally, there is a transaction peak of around 14.05 million blocks. In the peak block, the random balanced sharding method generates 70,455 cross-shard transactions, while the unbalanced sharding method generates 54,629 cross-shard transactions. Community detection-based sharding generates only 14,972 cross-shard transactions, approximately one-fifth of the number generated by random sharding. With our proposed sharding technique, the peaks of shard lines appear relatively flat because it is based on a community detection algorithm. Figure 2(b) shows that random balanced and unbalanced sharding leads to a high ratio of cross-shard transactions, i.e., around 80% and 60% respectively. Compared with random sharding with balanced shards, the ratio of cross-shard transactions is reduced from 80% to 20% by implementing the community-detected sharding rather than random sharding.

Fig. 3.
figure 3

Cross-shard TX ratio, the top 10 to 90 most traded addresses. Varying K within\(\{10,30,50,70,90\}\), and fixing \(s=4\).

Full size image

We explore our proposed scheme by changing the numbers of addresses or numbers of shards. According to Fig. 3, we demonstrate changes in the ratio of cross-shard transactions when the number of addresses varies. Table 2 presents the number of deleted addresses and the number of remaining addresses for each K value.

We evaluate our proposed community detection-based sharding approach by varying ETH most traded top K addresses from the range \(\{ 10, 30, 50 ,70, 90 \}\) and fixing shard number \(s = 4\). After traversing ten epochs, experiments on the ratio of \(\phi _c\) in Fig. 3 reveal an interesting pattern of change. We see that a decrease in the ratio of \(\phi _c\) occurs with an increase in K from 10 to 50, and an increase in the ratio of \(\phi _c\) occurs with an increase in K from 50 to 90. The ratio of cross-shards is lowest when \(K= 50\).

Table 2. Testes ETH top K addresses and remaining addresses
Full size table
Fig. 4.
figure 4

Cross-shard TX ratio, while varying shards number within \(\{4, 6, 8, 10\}\), and fixing \(K=90\).

Full size image
Fig. 5.
figure 5

Variation of cross-shard TX ratios in 1.3 million blocks.

Full size image

In addition to varying the top K, we test sharding performance by changing the shard number s from the range \(\{ 4, 6, 8, 10 \}\) and fixed K = 90. The observation in Fig. 4 indicates that an increase in s leads to an increase in \(\phi _c\). Although the s increases with \(\phi _c\), the s and the \(\phi _c\) do not follow a linear relationship. A significant change in the ratio of \(\phi _c\) occurs when the s is increased from 4 to 6, followed by a continually smaller change as the s increases. Sharding results show that fewer shards result in a lower ratio of cross-shard transactions. Also, cross-shard transactions will reach upper bounds as more shards are added.

Figure 5 shows the test results of our proposed sharding method from the block range of 13.7 million to 15.04 million. We display the overall ratio of cross-shard transactions during the tested period. Community detection-based sharding reduces transactions between shards to 20% and stays stable.

5 Conclusion

We presented a community detection-based sharding scheme to reduce cross-shard transactions. We converted blockchain data into an undirected and weighted graph format and executed a community-detection algorithm to get communities. The assignment of nodes to communities represented the division of nodes into shards. The superiority of this sharding approach was demonstrated by evaluating it on a real-world Ethereum dataset. The community detection-based sharding scheme reduced the ratio of cross-shard transactions to 20% and maintained stability.