In my earlier tutorial, I demonstrated how to use the Python library hashlib to create a sha256 hash function. Now, using Python, I am going to demonstrate the principle of blockchain mining. Again using BitCoin as my model, I will be trying to find a nonce value that will result in a hash value below a predetermined target.
We will start by simply enumerating an integer through our sha 256 hash function until we find a hash with 4 leading zeros.
I used a while loop, passing the variable “y” through my hashing function each time the loop runs. I then inspect the first 4 digits [:4] of my hash value. If the first four digits equal 0000 then I exit the loop by setting the found variable to 1
(*note, a hash value is a string – hence the need for quotes around ‘0000’)
As you can see in the version above, it took 88445 iterations to find an acceptable hash value
Now, using the basic example of a blockchain I gave in an earlier lesson, let’s simulate mining a block
You’ll see, I am now combining the block number, nonce, data, and previous hash of my simulated block and passing it through my encryption function. Just like in BitCoin, the only value I change per iteration is the Nonce. I keep passing my block through the hashing function until I find the Nonce that gives me a hash below the target.
Now, let’s lower the target value to 6 leading zeros. This should result in a longer runtime to get your hash
To measure the run time difference, let’s add some time stamps to our code
So, I am using the timestamp function twice. D1 will be our start time, d2 will be our end time, and I am subtracting d1 from d2 to get our elapsed time. In the example below, my elapsed time was 5 secs
Now, let’s bump the target down to 7 leading zeros. Now this brings my elapsed computing time to 20 minutes. That is a considerable commitment of resources. You can see why they call it a “proof of work” now.
This was, without a question, the most confusing aspect of blockchain to me when I first tried to learn it. Maybe it was because all the buzz around BitCoin mining. I usually find that when a topic becomes popular, misinformation spreads just as fast, if not faster, than actual information.
In an attempt to explain this topic I am going to be using BitCoin as my primary example. I am doing this mainly because I am sure that is where most of you first heard of the term blockchain mining. I will however, begin with a more generic explanation.
First off, let us separate blockchain mining from the idea of financial reward. Yes, in BitCoin, the miner who successfully mines a block is rewarded with BitCoin, but that is not a required part of a blockchain environment.
Mining, at its simplest form, just means successfully adding a new block to the blockchain.
So why can’t you just add a new block like you add a new element to a list or array in any other programming language? This has to do with the decentralized nature of blockchain. Since there is no centralized authority, blockchains rely on group consensus to verify that a new block added to the chain is valid. Keep in mind, this group is made up of anonymous nodes all over the world who do not know each other, and have no good reason to trust one another.
So, if you were to add a new block to the chain, the rest of the chain would need a mechanism in place that gives them time to update their copy of the chain and verify the block you added is good. So effectively we need a pause button.
How do we do that fairly and in a random manner that doesn’t allow for gaming of the system? We force anyone wishing to add a new block to show what is known as proof of work. Proof of work is proof that the person wishing to add the new block has completed a complex mathematical puzzle that required some level of resource allocation on their end. This effectively means they have skin in the game. There is a “cost” associated on their end. This “cost” deters the typical denial of service or blasting type attachk.
In BitCoin the mining process goes like this:
Bob wants to buy a guitar from Philip. They agree on a price of .2 BitCoin and through what is known as their BitCoin wallet a transaction is sent out to the BitCoin universe. A BitCoin wallet is a software client that the person trading with BitCoin uses. From the end user’s point of view, it is a lot like a Paypal type interaction.
Once the transaction is out in the ether, nodes (computers) known as BitCoin miners verify the transaction using a set of established rules built into the BitCoin software they are all running. These rules verify sender and receiver public keys, timestamps, etc. Once verified, the transaction is put into a queue.
Next the bitcoin miners build a new block. The goal is to make their new block the next block in the chain. This block with contain the following items:
Block number – just the next number in the line
Previous Hash – this the hash value of the current last block in the chain
Transactions – they will fill the block with verified transactions from the queue
On a side note – BitCoin only releases a block every 10 minutes. This is a design decision made by the makers of BitCoin, this is not a requirement of Blockchain
Once they have all of that information, they pass their block into a hashing algorithm and get a hash for their new block – This is where mining gets interesting
You see, not any old hash will do. In order for the block to be added to the chain, the hash must be less than the target hash.
The target hash is established by the initial creator of the blockchain. It can be whatever they chose. In the case of BitCoin, the target hash is actually programmed to drop lower and lower every couple thousand blocks.
Okay, so what is the big deal about trying to get below the hash?
Let’s consider this problem using a 6 digit number. (keep in mind that sha256 uses a 64 digit hexadecimal number – much bigger).
So we all agree that a hash is nothing more than creating a random number. Sure it looks funny in hexadecimal, but it can be converted to a base 10 number we all understand.
So, when I pass anything into my imaginary 6 digit hashing function, I can expect 1 million results, (if I only consider positive numbers) – 000,000 to 999,999
Now, let’s set a target for our hash. Let’s say that in order for a hash to be under the target, it must have a first digit of 0. I know what you are thinking – big deal! Well, actually, it kind of is. We have just gone from 1 million possible hashes to 100,000. We have effectively made 9 out of 10 available hashes invalid. We can now only accept 000,000 – 099,999. Now, let’s make the first 3 digits of our target 0. So now we have gone from 1 million possible hashes to 1000. 1000/1,000,000 = .001. So every time you run a hashing function, you now have a 0.1% chance of getting a valid hash.
Now think about our 64 digit hexadecimal number. It has a maximum value of 18,446,744,073,709,551,615 when converted to base 10. So If I made a requirement for the first 6 digits to be 0, we are now looking at: 99,999,999,999,999/18,446,744,073,709,551,615
Every time you run your hashing function, you have a 0.00054% chance of getting one below the target. So as you can see, in order to get a hash below the target, you will most likely have to use a brute force approach.
And that is what miners do. In fact the current difficultly related to finding a hash below the target in BitCoin has led to hundreds of thousands of nodes teaming up together to find a good hash in a brute force manner.
The way they find the hash is through the Nonce. You see, if you pass the word “dog” through my imaginary hash, you will get 123456. If you pass it to my hash 1 million times, you will get 123456 1 million times. So how it working in Blockchain, is we add the nonce to the hash, so dog+1 will give us 879602 and dog+2 will give us 258665. We will repeat this, enumerating the nonce until we get a good hash: dog+28549 gives us 000587. The nonce that gave us that hash 28549 – is called the golden nonce.
And in the world of BitCoin, once you have a golden nonce, you have your proof of work. You can now place your block on the chain.
It is called a proof of work, because other nodes on the blockchain can very quickly verify your work. They just pass your block with your golden nonce and they will get a hash of 000587, which is below the target. When enough nodes have verified your proof of work ( a consensus), your block becomes locked into the chain, it can no longer be changed or removed.
Okay, so what about the reward BitCoin miners get? Well, built into the BitCoin algorithm, each block mined with worth an ever decreasing number of BitCoin. At the time of this writing, I believe a mined block is worth 12.5 BitCoins. On top of that, the miners also get transaction fees from the people wishing to buy something with BitCoin. In the example above, Bob might offer up .1 BitCoin as a transaction fee as encouragement for some miner to put their transaction into the Blockchain. Keep in mind, there is no bank here, no centralized entity. So the transaction fee is the fee you pay help encourage total strangers to use their time and electricity to verify and move your transaction into the blockchain.
I promise to create another lesson diving deeper into transaction fees, until then, I hope this offers up at basic understanding of how mining works.
If you are at all like me, reading about a concept is one thing. Actually practicing it though, that helps me to actually understand it. If you have been reading my blockchain tutorial, or if you came from an outside tutorial, then you have undoubtedly read enough about cryptographic hashes.
For this example, I am using the Anaconda Python 3 distribution.
Like most things in Python, creating a hash is as simple as importing a library someone has already created for us. In this case, that library is: hashlib
So our first step is to import hashlib
Now let us take a moment to learn the syntax require to create a cryptographic hash with hashlib. In this example, I am using the SHA 256 hashing algorithm. I am using this because it is the same algorithm used by BitCoin.
Here is the syntax used
To understand the syntax, we are calling the hashlib method sha256(): hashlib.sha256()
Inside the brackets, we are entering the string we want to encode in the hash. Yes it must be a string for this function to work.
Still inside the brackets we use the method .encode() to (surprise, surprise) ENCODE the string as a hash
Finally, I added the method .hexdigest() to have the algorithm return our hash in hexadecimal format. This format will help in understanding future lessons on blockchain mining.
So in the example below, you can see that I assigned the variable x the string ‘doggy’. I then passed x to our hash function. The output can be seen below.
Now a hash can hold much more than just a simple word. Below, I have passed the Gettysburg Address to the hashing function.
(**note the ”’ ”’ triple quotes. Those are used in Python if your string takes up more than one line **)
Now I try passing a number. You will notice I get an error.
To avoid the error, I turn the integer 8 into a string with the str() function
Below I concatenation a string and an integer.
Last I want to show the avalanche effect of the hash function.
By simply changing the first letter from an uppercase T to a lowercase t the hash changes completely. This is a requirement for hashing functions. If the hash did not change dramatically from a small change to the string, it would be easy to reverse engineer the hash. This is known as the avalanche effect.
Let’s make the report at little more readable and interactive.
We are going to do a drill down on the names in our report. This mean when the report first opens, you will only see one row for each name. If you want to see the details of that person, you can click on a + by their name and the details for that person will pop down like an accordion.
Below your report design window, you should see another window called Row Groups
Go to the area marked Details (green arrow) and right click – Group Properties…
Click Visibility > Hide > Display can be toggled.
(Note I am selecting Name1 not Name. If you remember Name was the column we hid. Name1 is the column created when we created the Parent Group)
Now when you open your report, you will see it is in drill down format.
Our data is currently in table form, but is otherwise still nothing more than a raw data dump. Let’s make our report a little nicer with some grouping. Right click on the data row (not header) in your table. Mouse over to Add Group and select Parent Group…
In the new window, select[Name] from the drop down.
Click Add group headerand Add group footer boxes. Now click OK
Now a group has been added to your report.
If you click on preview, you will now see the table is grouped by Names
But you will notice we now have 2 columns showing the Name, one – our new grouping column and the other – the original column. This is redundant. To get rid of it, go back to Design, right click on the second name column and select Column Visibility…
When the new window opens up, click Hide.
Now when you look at the report now, you will see the second Name column is now hidden.
Next, let’s set up a running total for Hours spent on each job. To do so, right click on the Hourstext box and selectAdd Total
Now when we go back to preview, we will see at total in the group footer for each person
Now what if we wanted an average instead? Right click on the textbox that says [Sum(Hours)] and select Expression
You can just type = Avg(Fields!Hours.Value) in the expression builder box, but if you don’t know the code, you can use information in the boxes below. As you can see in the example below, if you go toCommon Functions > Aggregate you will see the code for lots of functions like Average, count, standard deviation.
Now when you go to preview, you will see an average.
But now we have a new problem. If you are trying to average something like work hours, odds are you will not need to go out to 10 decimal places. So a number like 8.272727272727 is pretty much ridiculous for a report like this.
Now go to Number > Number and set the Decimal Places to 2
So if you look at it again, you will see you only have 2 decimal points now.
Another concept you need to be familiar with to understand Blockchain is the concept of a P2p distributed network. This is the physical architecture that allows Blockchain to work and provides a blockchain with redundancy.
The P2P in P2P distributed network stands for peer to peer, indicating a network comprised of peers. What do I mean by that? The majority of computer networks in place right now are what is known as Server/Client networks.
In the picture below, the center square represents a server, with the boxes around it representing nodes (or in your case, the computer/tablet/phone you are reading this on). When you want to view a web page, you send a request from your node to a server. The server will then respond with the requested information.
While this works well, it does have some drawbacks. First off, since the server is central point of communication and the holder of all the information (webpages, databases, etc), if the server goes down, the network is essentially dead. This is the whole idea behind one of the more successful methods of cyber attack – the Denial of Service in which a server is targeted with more traffic than it can handle, shutting it down. You will often see it called a Distributed Denial of Server of DDoS as in order to hit the server with enough traffic to break it, hackers use multiple computers synced to deliver enough requests to the server all at the same time, overwhelming it. In other words, the attack is “distributed” across multiple computers.
Blockchain does not use a server client approach. Instead it uses a P2p or peer to peer network to function. In a peer to peer, the nodes (laptops, tablets, etc) all talk directly to each other. Instead of a server holding all the information, the data that makes up the blockchain is instead distributed across all the different nodes. So the more nodes that are part of the blockchain, the more copies of it that exist.
This works great for redundancy as even if you took out a couple of nodes in the network, it would still be able to function as normal. And as we will see a future lesson, even if you were able to hack in and corrupt the blockchain in one of the nodes, the fact that copies of it exist on all the other nodes protect it from corruption.
This architecture is also at the heart of philosophy around crypto-currencies like BitCoin. Unlike traditional banking systems that have centralized management, BitCoin is programmed with a deflationary policy that no one person (or group of person) can control. This is an interesting economic experiment unfurling before all of us. And I for one am curious to see how it plays out. While organizations like the Fed (in the United States) have done a relatively good job of keeping the US dollar strong, poor centralized economic management has spelled disaster in countries like Venezuela and Zimbabwe.
I’ll discuss more on the economic theory behind BitCoin in later lessons. It is enough for now for you to know that the P2P decentralized nature of the network is all part of the design in ensuring no one person can make such drastic changes.
If you want to learn more about networks and how they interact, here are some further resources:
The next concept we need to cover to properly understand blockchain is the concept of an Immutable Ledger. To translate this term into something that looks more like English, an Immutable Ledger simply means a record that cannot be changed.
The idea behind all of this is data security and proof that the data has not been altered. Why are we so concerned here? In a blockchain application like BitCoin, we are tracking transactions of money. Imagine if you sent me an electronic funds transfer for $100. How would you feel if I hacked into your bank and changed that $100, to $100,000? (For the record, if you tried that with my account, the computer would just laugh at you. The only time you’ll see a number that big associated with me is when you are looking at my student loans LOL).
Anyway, back to the point, you want to make sure that when you set up a transfer for $100, no one can alter it. With blockchain, that is true even if you want it to be altered (you made a mistake). If you want to fix an error, you will have to add another transaction to the blockchain to correct the issue. This is actually good accounting practice, as once an entry is made into a ledger, it should never be removed or altered.
Think of this like purchasing a car. If you go to your neighbor and buy his used car for $2000. You give him the money, and he signs over a title to you. The title is proof of ownership. To ensure that your neighbor cannot just take the car back, you take the title down to the Department of Motor Vehicles and have the title registered in your name. Now you have a record of your transaction should the ownership of the vehicle ever come into question.
So how does blockchain ensure immutability of the ledger? It all resides in the concept of the hash. If hacker tries to alter anything in the block below, its hash will change. Now the hash will no long match the previous hash in the second block. So, the hacker would have to change the next block, and the block after that, etc.
And even if they were able to pull that off, remember that the blockchain resides on multiple computers. The hacker would need to make all of these changes simultaneously. When you consider the millions of nodes that make up a blockchain environment like BitCoin, you will see that would be impossible.
In the next lesson, we will be looking at peer to peer distributed networks
To fully understand blockchain, it helps to have a good understanding of what is known as a cryptographic hash. It is this hash that is at the very core of how blockchain works.
If you read my introduction to blockchain lesson, you would see that the Hash is part of every block. It serves as a unique identifier for the block. That is the general idea for all Hashes, whether cryptographic or not.
It would actually be more proper for me to refer to it as a Hash Function. It is a function where you pass in text, a document, a picture, anything digital, and the function will return a unique identifying number.
Hash functions were used even before cryptography was an issue. You may have heard the term Hash Tables, this is where computer programmers would store a table full of hashes indicating text or documents, instead of having to store large documents in memory.
The Hash tables worked kind of like this:
You would pass some text to the Hash Function and it would transform it to a hash.
Ben -> 0123
Data Science -> 4871
Analytics is a great field of study -> 2580
These hashes were then placed in a table and when the program needed to access the information, it used the Hash to look it up – kind of like an index in the back of a book.
Now the Hash Functions used by Blockchain are cryptographic, so they are a little different than just a simple hash table. In order for a hash to be cryptographic, it needs to follow some basic rules.
It has to be 1 way. If you pass the text “Analytics” to a cryptographic hashing function it will return a hash(let’s say 1234). It will return the same hash every time you pass the text “Analytics” to it (again: 1234). So the hash function knows what hash to create for that text. But, we want to ensure that if someone has the hash 1234, they cannot reverse it to obtain the text (they can’t reverse engineer it)
Think of it like a finger print. If I have person, I can always obtain a fingerprint. However, if all I have is a fingerprint, I cannot produce a person from it. A finger print on its own won’t let me derive information like eye color or hair color of the individual who left print behind.
The hash function needs to be fast. You will understand better when we get to mining, but blockchain miners are passing millions of a hashes a second. With a slow hash function, the entire concept would fail.
The hash needs to ensure that similar items do not receive similar hashes. The example below shows what we do not want.
With the hashes 1234 and 1235 being so close as well as the text being so close, it would make it possible to reverse engineer a hash. For example, if you knew that Analytics4all was 1236, you might be able to back track the hash until you hit analytics at 1234
Instead we want something like this:
You see the similar text do not get similar hashes. Let’s look at it through numbers
Now different blockchain applications will use different cryptographic hash functions, For example, Etherium uses MD-5. BitCoin, however uses SHA-256, so that is what I will focus on.
SHA-256 was created by the NSA and as of this writing, it has not been cracked. The code for SHA-256 is open source, so anyone can make use of it.
A SHA-256 HASH is 64 hexidecimal digits long (64 digits * 4 bits per digit = 256).
Below, I pass the text Analytics to the hash generator and I get a 64 digit hash returned
Now when I just add the number 4 to the end of my text, the hash completely changes. The two hashes do not look anything alike. Again, this is designed to help reduce the chance of someone reverse engineering the hash.
Just to drive the point home. Here is removed the s from the end of the text. Note the new hash is again completely different.
Go to the website. Try out some hashing for yourself.
In the next lesson I will cover the concept of an imputable ledger. This will take us one step closer to understanding Blockchain.
Unless you’ve been living under a rock, you have undoubtedly heard the term blockchain being batted around. Most likely you’ve heard of it in relation to cryptocurrency like BitCoin, but blockchain is quickly moving into many other areas. Much like the Big Data craze though, my personal experience has been the more you hear someone utter the term blockchain, the less that person actually knows about it. It is the Dunning-Kruger Effect in action.
In this lesson, I am going to introduce you to the concept of blockchain. I have boiled it down to its simplest concepts, and I will be speaking very broadly about the subject. In future lessons I will dive deeper into the more technical aspects of blockchain, providing much more in the way of specifics.
For those already familiar with blockchain, I am aware that I am glossing over some rather important concepts here, but my goal in this lesson is to provide a simple, easily understood tutorial. The goal of my website is to provide an accessible education into many of the complex concepts surrounding analytics and data science to everyone, regardless of past experience or education. I promise a deeper dive in the future, but for now, let us start simply.
What is a blockchain?
At the most basic level, a blockchain is a collection of data kept in a list.
What makes these lists so interesting is that:
They are connected using cryptography
They are distributed amongst multiple computers, providing a redundant method of protection
To understand how they work, let us start by looking at a block
Starting from the top
Block #: is just the number of the block in the chain, first block # is 1, second is 2, so on
Nonce: stands for “number used only once”. I am going to cover the Nonce in depth in the lesson on Mining, but for right now, just be aware that it is a number and every block needs a Nonce
Data:This is where the data is stored. In BitCoin this is often filled with transaction information
Previous Hash: The hash of the block before this one
Hash:I will cover hashes in depth in a future lesson two. For now, just know this the cryptographic part of blockchain. A hash is a code number that identifies the block. An easy way to conceptualize it is to think of it like a VIN on a car. The VIN (or vehicle identification number) can be used to tell you the make, model, color, and many other characteristics of a car. The hash (in blockchain) will tell you everything found in the block (I know I am way over simplifying here, but hey we have to start somewhere, and I promise a future lesson on hash)
Now let’s add a second block to our chain
Notice the previous hash in block 2 is the same as the hash in block 1. This, you will soon see, is part of what makes blockchain so secure.
In the picture below, you can see if someone tried to go into Block 1 and make a change, the Hash for block 1 would change. Any change to the first four fields in a block will cause the Hash to chance since the block is no longer the same anymore. The Hash is like a VIN or a fingerprint, it can only represent a single individual block. And once you change any aspect of a block, it is no longer the same individual block anymore.
So as you can see, if the Hash in the first block changes, it will no longer match the Previous Hash in the second block. When this happens the blockchain is broken. So if a hacker tried to alter a transaction in block 1, the chain would break.
Okay, so what is to prevent the hacker from just changing the second block? Aside from hashing issues that we will discuss in future lessons? The other deterrent is found in peer to peer sharing of blockchains
In most real world applications of blockchain, the chain will not reside on only one computer, instead will be replicated across multiple computers.
So now if a hacker tries to change the third block on one computer.
The third blocks hash will change, breaking its connection from the fourth block
And even more importantly, the chain will no longer look like the one on the second computer. Now in real life, this will be spread across thousands of computers. So to determine which blockchain is correct, they look to see what iteration the majority of computers say is correct.
So as seen below, 3 of the 4 computers show 4 blue squares, while only one was a yellow square. So based on the vote of the majority, the final result will be 4 blue squares.
So there you have it, blockchain in its most simplistic form. In future lessons I’ll dive deeper in the different concepts to show how it works from the inside out.
Click below for an interesting link for BitCoin information