Here is a code block to create a database if you want to play along
create database sandbox;
use sandbox;
CREATE TABLE employee_id (
emp_nm varchar(30) not null,
emp_id varchar(8),
b_emp_id varchar(8),
PRIMARY KEY(emp_nm) );
While loading data from Excel files and CVS files is pretty straightforward in Python, getting database from a SQL database is really a skill you should learn. SQL databases can store much more data than an Excel file and are used pretty much everywhere now.
This lesson will focus on MySQL, I have another lesson for SQL Server as well. The syntax does vary a little and if you are working with Oracle or Teradata, your syntax may change a bit as well. Luckily the Internet has plenty of resources to point you in the right direction.
Now, for MySQL, lets get started. First you are going to want to install mysql.connector (if you have the Anaconda distribution that I recommend, it comes with this pre-installed.
If you don’t have it installed, you can get it using pip or conda – I have a video showing you how to install a module here: video link — skip ahead to the 7 minute mark where I explain how to install modules
Once install, import it into your notebook
Now lets make a connection to our server: syntax should be self explanatory – localhost simply means its installed on my computer — otherwise you would provide a network path to the server there.
Let’s start by looking around- First I want a name off all the databases in my instance
Now in this example we will be working with the Sandbox database — I provided code at the top of the lesson you can paste and run in your MySQL instance to create a Sandbox database
Now lets add a new element to our connection string – database
And query the table names in the Sandbox database
Using the .execute() method, we can pass SQL commands to the MySQL instance
Below I am creating a table -> passing data to the table -> and committing the data
Without the commit() command, the data will not be saved into the table.
To add multiple rows to the table, I can use the executemany() method
Now let’s query our table. Note the .fetchall() method — this brings in all the rows from the query. I can iterate through list by running a simple for loop
I can also use the command .column_names after my cursor to return the column names of the last table I queried
Finally, we can use Pandas to put this database table into a dataframe we can work with
Decorators are functions that “wrap” around other functions, changing their behavior without changing the function it self. If you have not seen my Closure lesson, you might want to visit it first (Closure)
If you already read it, let us move forward. Let’s start by building a simple function that simply prints my name:
Now we need to build the decorator function. You may notice it is a Closure
Now let’s us the dec() (decorator) to alter what my_name() returns
By passing the function as a argument through the decorator function, my name is now printed out now with text above and below.
Now we can build another decorator function, and try it out. Notice the output differs from the output above.
Next, notice I can nest the decorator functions together
Now, instead of the syntax about, you can simply add the decorator name with a @ in front and it will apply the decorator.
In the second box, you will notice I can stack or nest the 2 decorators I made.
In my humble opinion as a Data Scientist, I think the nesting function would be the most usefulness out of this. Again, I am not a software engineer. They may have more of use for this functionality.
This is a more advanced programming topic, and honestly not one I make much use of myself. However, I got a request to make a lesson on Closures and Decorators. I am going to make it into two lessons to try to make it a bit clearer.
Now to understand closures and decorators, the first thing we need to discuss is nested functions. In programming, nested functions means wrapping a function inside of another function. Here is a very simple example of a nested function:
Now if I just call this with outerFunction(‘Hello World’), the argument x gets passed into the innerFunction which is in turn called by the outerFunction. The result is shown below:
Now why would anyone do this? Outside of just being able to show you that you can? I don’t know. There are reasons for nested functions that involve hiding variables from the code outside of the functions. That is a bit beyond the scope of this lesson. I will try to tackle it when I create my Object Oriented Programming lessons.
For now, let us get on to Closures. Let’s look at the code below:
It is very close to the first example, except notice 2 little changes. First, instead of just calling the innerFunction at the end of the function, we are returning it. Notice there is no () when we return innerFunction
Also notice we are passing the outerFunction to a variable and when we call the var we add () to the end: var()
What we have done now is created an official Closure, let me show you why that matters below.
First, notice I delete the outerFunction. When I try calling it returns an error saying outerFunction is not defined (meaning it doesn’t exist).
But, look what happens when I called var(). It is still there. The closure commits the value to memory when you create it. That is not something that would happen from a normal function or even nesting function.
SO WHY THE HELL SHOULD I CARE????
So why this is cool, and why you might use this in the future, is imagine a function that does processer heavy multi-step calculations. If you just keep calling the function as usual, each time you call it, it has to run the whole chain of calculations again. However, once you’ve called it as closure, you don’t have to run the calculations again, you just get the value.
So, how about a more concrete example
Here I created a closure that uses two functions which each take 1 argument, the arguments are then multiplied together.
Notice, I created a variable called double where I sent the outer function mult_by() a value of 2.
When I call double() I can pass it a value for (y)
I can create as many of these instances as I want. Look below, created one for Triple, and notice double is still functional
I’ll continue this in the Decorators lesson.
Just the main thing to keep in mind about closures, is that they improve program performance by no requiring the function to be run each time.
If you are still a little confused after reading this, don’t feel bad. It took me a few tries to finally understand closures myself.
Data Analysts get a bad wrap. With the advent of the Data Scientist, Data Analysts are often viewed as Data Scientists lite, however I feel that is not the honest case. Truth is, there is a lot of overlap between the two fields. I will dive deeper into what a Data Scientist is in a future article, but just know my opinion is the definition of Data Scientist as a job is still a bit fuzzy and I think the job title may eventually be broken into a few different titles to better define the differences.
Data Analyst
So what does a Data Analyst do?
A lot actually. You could put 10 data analysts into a room and you would get ten different answers to this question. So the best I can do here is make sweeping generalities. As the old saying goes “Your results may vary”
In general, data analysts perform statistical analysis, create reporting, run ad-hoc queries from data warehouses, create data visualizations, create and maintain dashboards, perform data mining, and create machine learning models (yes, ML is not only for data scientists). Their assignments are business driven. A data analysts is either embedded with a business unit (financial planning, fraud, risk management, cyber security, etc.) or working in a centralized reporting/analytics team. They use their skills to provide reporting and analytics for the business.
Tools used by Data Analysts
SQL – MySql, SQL Server, Oracle, Teradata, Postgres – whether simply querying a data warehouse or creating and managing a local data mart, data analysts need to be advanced SQL programmers
Visualization tools – Tableau, Qlik, Power BI, Excel, analysts use these tools to create visualizations and dashboards
Python/R – Data analysts should be familiar with languages like Python or R to help manage data and perform statistical analysis or build machine learning models
Spreadsheets – Excel, Google Sheets, Smart Sheets are used to create reports, and pivot tables used to analyze the data
ETL tools – SSIS, Alteryx, Talend, Knime, these tools are design to move data to and from databases, CSV files, and spreadsheets. Until the data is in a usable format, analysis cannot be performed.
Educational Requirements
Typically a data analyst position will ask for a bachelors degrees, preferably in computer science, statistics, database management or even business. While the barrier to entry for a data analyst job is generally not as high as a data scientist, that does not mean you cannot make a meaningful and well paid career as a data analyst. Also, the demand for data professionals seems to keep going up and up and it most likely will for the foreseeable future.
We are using a module that does not come with the Anaconda distribution. To get this code, open up your Anaconda Prompt and type the following code:
> pip install pyqrcode
Next, you will import pyqrcode into python. Also, since the output of this program will be a png picture file, I am running pwd to determine where on my machine the file will be saved.
Next pick a URL you want your QRCode to go to. I chose my website. I assigned my website address to a variable url.
Then, using the pyqrcode module and its method create(), place your variable in the parenthesis ().
Note printing the QR code you created will not display it, but it will give you the basic information on it. To turn it into a QR code, we will need to covert it to a png.
*Note: Some of you may need to download the png module if you get an error. Just go to pip install png and add import png to the top of the script.
Otherwise, just give you file a name, and scale — the larger the scale, the larger your QR code will be.
Finally, go to your directory where the file was saved and open it.
Scanning the QR Code below should take you to my website’s main page.
Selenium is a Python package that allows you to control web browsers through Python. In this tutorial (and the following tutorials), we will be connecting to Googles Chrome browser, Selenium does work with other browsers as well.
First you will need to download Selenium, you can use the following commands depending on your Python distribution
c:\> Pip install selenium
c:\> Conda install selenium
If you are on a work computer or dealing with a restrictive VPN, the offline install option may help you: Selenium_Install_Offline
Next you need to download the driver that let’s you manage Microsoft Edge through Python.
Start by determining what version of Edge you have on your computer
Click the three horizontal lines in the upper right corner > Help and Feedback>About Microsoft Edge
Search for msedgedriver to download the file that matches your Edge version. (note, this is something you will need to do every time Edge is updated, so get used to it.)
Open up the zipfile you downloaded, you will find a file called msedgedriver.exe
Put it somewhere you can find, put in the following code to let Python know where to find it.
from selenium import webdriver
opts = webdriver.Edge()
dr = webdriver.Edge('C:/Users/larsobe/Desktop/msedgedriver.exe',chrome_options=opts)
Now to see if this works, use the following line, (you can try another website if you choose)
Note the message Edge is being controlled by automated test software.
For your first project in the course, I am giving you the code below. You will notice I am placing 2 dictionaries in a list:
d = {'Name': 'Ben', 'Age': 35}
e = {'Name': 'Christine', 'Age': 30}
a = []
a.append(d)
a.append(e)
print(a)
Your challenge, should you choose to accept it, will be to run this code on your machine or in the test python browser (click on the blue arrow below)
Try your Python code in the free console
I then want you to
1) Print out the second dictionary from the list
2) Print out the name from the first dictionary
3) Print out both ages
Note: I have not shown how to work with dictionaries inside a list. So consider this a stretch project. Don’t be afraid to use Google to help find an answer.
Project Answer Notebook Download Available at Course Page below:
