What is Sharding? Guide to this Ethereum Scaling Concept Explained


As the scaling debate in cryptocurrencies continues, some potential solutions have actually been in development for quite some time now. Specifically, in the case of Ethereum, where a large focus is placed on decentralization and security at the expense of scalability, the application of sharding in conjunction with implementing Proof of Stake consensus is seen as the much needed mechanism through which the network can scale to practical levels for applications while still retaining its decentralization and security.

Sharding is a complex topic, especially when applied to a decentralized, peer to peer network such as Ethereum where the global state of the network constantly is updated. So what exactly is sharding and how can it help blockchain networks to scale?

Sharding and Distributed Computing Background

Sharding is actually much older than blockchain technology and has been implemented in a variety of systems from business database optimizations to Google’s global Spanner database. Essentially, sharding is a particular method for horizontally partitioning data within a database. More generally, the database is broken into little pieces called “shards”, that when aggregated together form the original database.

In distributed blockchain networks, the network consists of a series of nodes connected in a peer to peer format, with no central authority. As is the case with current blockchain systems, each node stores all states of the network and processes all of the transactions. While this provides the high level security through decentralization, especially in Proof of Work systems such as Bitcoin and Ethereum, it leads to legitimate scaling problems.

Using Ethereum as an example, a full node in the Ethereum network stores the entire state of the blockchain, including account balances, storage, and contract code. Unfortunately, as the network increases in size at an exponential pace, the consensus only increases linearly. This limitation is due to the communication needed between the nodes needed to reach consensus.

Nodes in the network do not have special privileges and every node in the network stores and processes every transaction. As a result, in a network the size of Ethereum’s, issues such as high gas costs and longer transaction confirmation times become noticeable problems when the network is strained. The network is only as fast as the individual nodes rather than the sum of its parts.

Sharding helps to alleviate these issues by providing an interesting, yet complex solution. The concept involves grouping subsets of nodes into shards which in turn process transactions specific to that shard. It allows the system to process many transactions in parallel, thus significantly increasing throughput.

A simpler way to put it would to be imagining the division of the United States into states. While each state (a shard in this case) is part of the larger United States (Ethereum network), they have their own specific rules, boundaries, and subsets of populations. However, they do share a universal language and culture as part of their larger network that makes up the country.

Or even better, in Vitalik Buterin’s own words:

 “Imagine that Ethereum has been split into thousands of islands. Each island can do its own thing. Each of the islands has its own unique features and everyone belonging on that island i.e., the accounts, can interact with each other AND they can freely indulge in all its features. If they want to contact other islands, they will have to use some sort of protocol.”

As you can see, the concept of fragmenting the network into more efficient pieces allows the network to function as the sum of its parts, rather than being limited by the speed of each individual node.

How Does Sharding Work in Blockchains?

We will continue to use Ethereum as an example in this as it is the most well-known and arduous sharding attempts in the blockchain arena, as the Ethereum developers are implementing what is known as “state sharding”.

The current state of the Ethereum blockchain is known as the “global state” and is what everyone can see when they look at the blockchain at a specific instance. The tricky part in implementing sharding in Ethereum is that by sharding the nodes into smaller subsets, these subsets need to be able to process specific sets of transactions while simultaneously updating the state of the network, all while ensuring it is valid.

Sharding in Ethereum is supposed to be implemented in a two phase rollout, more than likely after Proof of Stake is implemented in the network. Phase one will be the data layer consisting of the consensus of what data is in the shards. Phase two is the state layer. All of this is very fluid, so a general breakdown of how it may work is below.

Ethereum breaks down the network into specific shards. Each shard is assigned a specific group of transactions that is determined by grouping specific accounts (including smart contracts) into a shard. Each transaction group has a header and a body that consist of the following.

  • The shard ID of the transaction group
  • Assignment of validators through random sampling (verify the transactions in the shard)
  • State Root (state of the merkle root of the shard before and after transactions added)

Body

  • All of the transactions that belong to the transaction group that are part of the specific shard.

Transactions are specific to each shard and occur between accounts native to that shard. When transactions are verified, the state of the network changes and account balances, storage, etc are updated. In order for the transaction group to verify as valid, the pre-state root of the transaction group must match the shard root in the global state. If they match, the transaction group is validated and the global state is updated through the particular shard ID state root.

Instead of only containing a state root, each block of the Ethereum blockchain now contains both a state root and the transaction group root. The transaction group root is the merkle root of all of the transaction groups from the specific shards for that block of transactions. Basically, there is a merkle root of all of the different shards that contain the updated and verified transaction groups. This root is stored in the blockchain along with the updated state root.

The employment of merkle tree concepts in this structure is vital to ensuring validity of the blockchain. Understanding how a merkle tree and specifically a merkle root work, can help you to grasp these concepts much more easily.

What is a Merkle Tree

Read: What is a Merkle Tree ?

Consensus within a shard is reached through a Proof of Stake consensus of randomly selected nodes that are applied to a shard for specific consensus round. This not only provides finality to the consensus, which is necessary within the shards, but also provides a particular defense to an attack that a Proof of Work blockchain would be susceptible to in this instance.

The hash power required to overrun a specific shard in a PoW sharded network is drastically reduced and the ability for a malicious actor to take over a shard through computational power is feasible. Through this, the bad actor could attack other shards through the communication protocol which is one of the more complicated and important features of sharding architecture. Random sampling selection of the validators within a shard manages to stifle this type of attack since a bad actor will not know which shard they are being placed in before they are actually placed in it. Further, random sampling will be used to select the validators that are actually validating from that random validating set.

The communication protocol is vital to the sharding architecture functioning correctly in the system. You can think of the communication protocol as the universal language that is consistent among the states as part of the larger United States. However, designing this protocol is highly challenging and needs to be performed so that it is only used when necessary. It becomes necessary when a specific node requires information that is not stored within its own shard and needs to find the shard with the requisite information. This communication is known as cross-shard communication.

The cross-shard communication is achieved through applying the concept of transaction receipts. The receipt for a transaction is stored in a merkle root that can be easily verified but that is not part of the state root. The shard receiving a transaction from another shard checks the merkle root to ensure that the receipt has not been spent. Essentially, the receipts are stored in a shared memory that can be verified by other shards, but not altered. Therefore, through a distributed storage of receipts, shards are able to communicate with each other.

Sharding Moving Forward

Sharding in Ethereum is expected to be implemented after the Casper PoS upgrade. Recently, there have been some developments regarding Ethereum 2.0 which involve implementing both Casper and sharding. Casper seems to be slated for 2019 while sharding will follow in 2020 or 2021.

Sharding has also been implemented in a few other platforms, most notably Zilliqa. However, Zilliqa does not implement state sharding at this time and instead focuses on providing a high throughput blockchain by utilizing transaction and computational sharding.

Zilliqa

Read our Guide to Zilliqa

Conclusion

Sharding serves to offer some promising solutions to the elephant in the room of blockchain platforms right now, scalability. While Bitcoin’s lightning network is in the testing phase and has been showing some very promising progress so far, Ethereum’s solution brings with it some unique challenges as it is pegged as a world computer that is Turing complete.

Sharding will directly work only at the protocol level, so to the end user or dapp developer it may not be necessarily relevant to learn about. Regardless, Ethereum’s attempt at state sharding for a vast, decentralized network is an impressive endeavor and will be an enormous feat of accomplishment if successfully implemented.


Be the first to comment

Leave a Reply

Your email address will not be published.


*


Privacy Policy

WideBitcoin.com is committed to safeguarding your privacy. Contact us at if you have any questions or problems regarding the use of your Personal Data and we will gladly assist you.

By using this site or/and our services, you consent to the Processing of your Personal Data as described in this Privacy Policy.

Table of Contents

  1. Definitions used in this Policy
  2. Data protection principles we follow
  3. What rights do you have regarding your Personal Data
  4. What Personal Data we gather about you
  5. How we use your Personal Data
  6. Who else has access to your Personal Data
  7. How we secure your data
  8. Information about cookies
  9. Contact information

Definitions

Personal Data – any information relating to an identified or identifiable natural person.
Processing – any operation or set of operations which is performed on Personal Data or on sets of Personal Data.
Data subject – a natural person whose Personal Data is being Processed.
Child – a natural person under 16 years of age.
We/us (either capitalized or not)

Data Protection Principles

We promise to follow the following data protection principles:

  • Processing is lawful, fair, transparent. Our Processing activities have lawful grounds. We always consider your rights before Processing Personal Data. We will provide you information regarding Processing upon request.
  • Processing is limited to the purpose. Our Processing activities fit the purpose for which Personal Data was gathered.
  • Processing is done with minimal data. We only gather and Process the minimal amount of Personal Data required for any purpose.
  • Processing is limited with a time period. We will not store your personal data for longer than needed.
  • We will do our best to ensure the accuracy of data.
  • We will do our best to ensure the integrity and confidentiality of data.

Data Subject’s rights

The Data Subject has the following rights:

  1. Right to information – meaning you have to right to know whether your Personal Data is being processed; what data is gathered, from where it is obtained and why and by whom it is processed.
  2. Right to access – meaning you have the right to access the data collected from/about you. This includes your right to request and obtain a copy of your Personal Data gathered.
  3. Right to rectification – meaning you have the right to request rectification or erasure of your Personal Data that is inaccurate or incomplete.
  4. Right to erasure – meaning in certain circumstances you can request for your Personal Data to be erased from our records.
  5. Right to restrict processing – meaning where certain conditions apply, you have the right to restrict the Processing of your Personal Data.
  6. Right to object to processing – meaning in certain cases you have the right to object to Processing of your Personal Data, for example in the case of direct marketing.
  7. Right to object to automated Processing – meaning you have the right to object to automated Processing, including profiling; and not to be subject to a decision based solely on automated Processing. This right you can exercise whenever there is an outcome of the profiling that produces legal effects concerning or significantly affecting you.
  8. Right to data portability – you have the right to obtain your Personal Data in a machine-readable format or if it is feasible, as a direct transfer from one Processor to another.
  9. Right to lodge a complaint – in the event that we refuse your request under the Rights of Access, we will provide you with a reason as to why. If you are not satisfied with the way your request has been handled please contact us.
  10. Right for the help of supervisory authority – meaning you have the right for the help of a supervisory authority and the right for other legal remedies such as claiming damages.
  11. Right to withdraw consent – you have the right withdraw any given consent for Processing of your Personal Data.

Data we gather

Information you have provided us with
This might be your e-mail address, name, billing address, home address etc – mainly information that is necessary for delivering you a product/service or to enhance your customer experience with us. We save the information you provide us with in order for you to comment or perform other activities on the website. This information includes, for example, your name and e-mail address.

Information automatically collected about you
This includes information that is automatically stored by cookies and other session tools. For example, your shopping cart information, your IP address, your shopping history (if there is any) etc. This information is used to improve your customer experience. When you use our services or look at the contents of our website, your activities may be logged.

Information from our partners
We gather information from our trusted partners with confirmation that they have legal grounds to share that information with us. This is either information you have provided them directly with or that they have gathered about you on other legal grounds. See the list of our partners here.

Publicly available information
We might gather information about you that is publicly available.

How we use your Personal Data

We use your Personal Data in order to:

  • provide our service to you. This includes for example registering your account; providing you with other products and services that you have requested; providing you with promotional items at your request and communicating with you in relation to those products and services; communicating and interacting with you; and notifying you of changes to any services.
  • enhance your customer experience;
  • fulfil an obligation under law or contract;

We use your Personal Data on legitimate grounds and/or with your Consent.

On the grounds of entering into a contract or fulfilling contractual obligations, we Process your Personal Data for the following purposes:

  • to identify you;
  • to provide you a service or to send/offer you a product;
  • to communicate either for sales or invoicing;

On the ground of legitimate interest, we Process your Personal Data for the following purposes:

  • to send you personalized offers* (from us and/or our carefully selected partners);
  • to administer and analyse our client base (purchasing behaviour and history) in order to improve the quality, variety, and availability of products/ services offered/provided;
  • to conduct questionnaires concerning client satisfaction;

As long as you have not informed us otherwise, we consider offering you products/services that are similar or same to your purchasing history/browsing behaviour to be our legitimate interest.

With your consent we Process your Personal Data for the following purposes:

  • to send you newsletters and campaign offers (from us and/or our carefully selected partners);
  • for other purposes we have asked your consent for;

We Process your Personal Data in order to fulfil obligation rising from law and/or use your Personal Data for options provided by law. We reserve the right to anonymise Personal Data gathered and to use any such data. We will use data outside the scope of this Policy only when it is anonymised. We save your billing information and other information gathered about you for as long as needed for accounting purposes or other obligations deriving from law, but not longer than 1 year.

We might process your Personal Data for additional purposes that are not mentioned here, but are compatible with the original purpose for which the data was gathered. To do this, we will ensure that:

  • the link between purposes, context and nature of Personal Data is suitable for further Processing;
  • the further Processing would not harm your interests and
  • there would be appropriate safeguard for Processing.

We will inform you of any further Processing and purposes.

Who else can access your Personal Data

We do not share your Personal Data with strangers. Personal Data about you is in some cases provided to our trusted partners in order to either make providing the service to you possible or to enhance your customer experience. We share your data with:

Our processing partners:

  • facebook.com
  • google.com
  • bing.com
  • twitter.com
  • pinterest.com

Our business partners:

  • facebook.com
  • google.com
  • bing.com
  • twitter.com
  • pinterest.com

Connected third parties:

  • facebook.com
  • google.com
  • bing.com
  • twitter.com
  • pinterest.com

We only work with Processing partners who are able to ensure adequate level of protection to your Personal Data. We disclose your Personal Data to third parties or public officials when we are legally obliged to do so. We might disclose your Personal Data to third parties if you have consented to it or if there are other legal grounds for it.

How we secure your data

We do our best to keep your Personal Data safe. We use safe protocols for communication and transferring data (such as HTTPS). We use anonymising and pseudonymising where suitable. We monitor our systems for possible vulnerabilities and attacks.

Even though we try our best we can not guarantee the security of information. However, we promise to notify suitable authorities of data breaches. We will also notify you if there is a threat to your rights or interests. We will do everything we reasonably can to prevent security breaches and to assist authorities should any breaches occur.

If you have an account with us, note that you have to keep your username and password secret.

Children

We do not intend to collect or knowingly collect information from children. We do not target children with our services.

Cookies and other technologies we use

We use cookies and/or similar technologies to analyse customer behaviour, administer the website, track users’ movements, and to collect information about users. This is done in order to personalize and enhance your experience with us.

A cookie is a tiny text file stored on your computer. Cookies store information that is used to help make sites work. Only we can access the cookies created by our website. You can control your cookies at the browser level. Choosing to disable cookies may hinder your use of certain functions.

We use cookies for the following purposes:

  • Necessary cookies – these cookies are required for you to be able to use some important features on our website, such as logging in. These cookies don’t collect any personal information.
  • Functionality cookies – these cookies provide functionality that makes using our service more convenient and makes providing more personalised features possible. For example, they might remember your name and e-mail in comment forms so you don’t have to re-enter this information next time when commenting.
  • Analytics cookies – these cookies are used to track the use and performance of our website and services
  • Advertising cookies – these cookies are used to deliver advertisements that are relevant to you and to your interests. In addition, they are used to limit the number of times you see an advertisement. They are usually placed to the website by advertising networks with the website operator’s permission. These cookies remember that you have visited a website and this information is shared with other organisations such as advertisers. Often targeting or advertising cookies will be linked to site functionality provided by the other organisation.

You can remove cookies stored in your computer via your browser settings. Alternatively, you can control some 3rd party cookies by using a privacy enhancement platform such as optout.aboutads.info or youronlinechoices.com. For more information about cookies, visit allaboutcookies.org.

We use Google Analytics to measure traffic on our website. Google has their own Privacy Policy which you can review here. If you’d like to opt out of tracking by Google Analytics, visit the Google Analytics opt-out page.

Read more about cookies on our Cookie Policy

Contact Information

email: contact@widebitcoin.com

Changes to this Privacy Policy

We reserve the right to make change to this Privacy Policy.

You can configure your Internet browser, by changing its options, to stop accepting cookies completely or to prompt you before accepting a cookie from the website you visit. If you do not accept cookies, however, you may not be able to use all portions of the WideBitcoin Websites or all functionality of the Services.

Please note that disabling these technologies may interfere with the performance and features of the Services.

You may also disable cookies on the WideBitcoin Sites by modifying your settings here:

Visitor comments may be checked through an automated spam detection service.

Last Update: May 25, 2018