Category: Analytics

Top 5 Data Visualization Tools on the Market

On By Ben Larson Ph.D.In Analytics, Qlik, Tableau, VizualizationLeave a comment

Data visualization is an essential aspect of data analysis and communication. Data visualization tools enable users to transform data into charts, graphs, and other visual representations that are easier to understand and interpret. In this article, we will look at some of the top data visualization tools available in the market.

Photo by Luke Chesser on Unsplash

Tools are listed in no particular order

1. Tableau

Tableau is a powerful data visualization tool that enables users to create interactive dashboards, reports, and charts. It has a user-friendly interface, which allows users to drag and drop data to create visuals quickly. Tableau is known for its robust features, including data blending, mapping, and real-time collaboration. It also has a vibrant community, which makes it easy to find resources and solutions to any challenge.

2. Power BI

Power BI is a popular data visualization tool developed by Microsoft. It enables users to create interactive dashboards and reports that can be shared across an organization. Power BI has a user-friendly interface and offers a wide range of features, including data modeling, forecasting, and natural language processing. It also integrates seamlessly with other Microsoft products like Excel, SharePoint, and Teams.

3. QlikView

QlikView is a business intelligence tool that enables users to create interactive visualizations, reports, and dashboards. It has an intuitive interface that allows users to drag and drop data and create charts and graphs quickly. QlikView also offers advanced features like data modeling, association analysis, and collaboration capabilities.

4. D3.js

D3.js is a data visualization library that allows users to create custom visualizations using web standards like HTML, CSS, and SVG. It provides a high degree of flexibility, allowing users to create unique visualizations that match their specific needs. D3.js has a steep learning curve, but its versatility and customization options make it a favorite among developers.

5. Google Data Studio

Google Data Studio is a free data visualization tool that enables users to create interactive reports and dashboards. It integrates with Google Analytics and other Google products, making it easy to gather and analyze data. Google Data Studio also offers collaboration capabilities, allowing teams to work together on reports and dashboards.

Conclusion

In conclusion, data visualization tools play a crucial role in helping organizations make sense of their data. The tools mentioned above are just a few of the many available in the market. When choosing a data visualization tool, it’s essential to consider factors like ease of use, features, and cost. Ultimately, the right tool will depend on the specific needs of your organization.

Top 7 skills a Data Analyst have should have?

On By Ben Larson Ph.D.In Analytics, Careers in Data, UncategorizedLeave a comment

Data analysis has become an integral part of business operations in the digital age. As companies collect and store vast amounts of data, they need skilled professionals to extract insights and make data-driven decisions. Data analysts play a crucial role in this process, using their expertise to analyze data and draw insights that inform business decisions. Here are the top skills a data analyst should have to excel in this field.

Photo by path digital on Unsplash

1. Strong quantitative skills:

Data analysts need to be comfortable with numbers and statistics. They should have a solid foundation in mathematics and be proficient in tools such as Excel and statistical programming languages like R or Python. Understanding mathematical concepts such as probability, regression analysis, and hypothesis testing is essential to be able to analyze data accurately and draw meaningful conclusions.

2. Data visualization skills:

Data analysts must be able to communicate their insights effectively. They should be proficient in creating data visualizations, such as charts and graphs, to help others understand complex data sets. Knowledge of data visualization tools like Tableau, Power BI, or QlikView can help in creating interactive dashboards and presentations to communicate insights.

3. Strong problem solving skills:

Data analysis is all about solving problems. Data analysts must have a strong analytical mind to identify patterns and insights in data that can help businesses solve problems. They should have the ability to think creatively, identify gaps in data, and come up with strategies to fill those gaps.

4. Attention to detail:

Data analysis requires meticulous attention to detail. Data analysts must be able to identify and correct errors in data to ensure accuracy in their analysis. They must be skilled in data cleaning and data preparation, and be able to ensure data consistency across multiple sources.

5. Business acumen:

Data analysts should have a strong understanding of the business they are working for. They must be able to connect the insights drawn from data analysis to the larger business objectives and strategy. They must also be able to communicate their findings in a way that is understandable to non-technical stakeholders.

6. Communication skills:

Data analysts must be able to effectively communicate their insights to stakeholders. They must be skilled in creating clear and concise reports, presentations, and visualizations that convey the insights drawn from data analysis. They should also be able to work collaboratively with others, including non-technical stakeholders, to identify business problems and develop solutions.

7. Continuous learning:

The field of data analysis is constantly evolving, and data analysts must be willing to continuously learn and adapt to new technologies and techniques. They should be passionate about exploring new tools and techniques, and be willing to experiment with new approaches to problem-solving.

Conclusion

In conclusion, data analysis is a complex field that requires a combination of technical skills, business acumen, and problem-solving abilities. Data analysts must be comfortable with numbers and statistics, have strong analytical skills, and be able to communicate their insights effectively. They must be able to work collaboratively with others, including non-technical stakeholders, to identify business problems and develop solutions. Finally, they must be willing to continuously learn and adapt to new technologies and techniques to stay ahead in the field.

Data Jobs: What does a DBA do?

On By Ben Larson Ph.D.In Analytics, Careers in Data1 Comment

A database administrator, commonly referred to as a DBA, is a professional who is responsible for managing, maintaining, and optimizing a database. Databases are an essential component of most organizations as they store critical information that is required for daily operations.

The role of a DBA is to ensure that the database is running smoothly and efficiently, while also being available to users at all times. This requires a combination of technical and interpersonal skills, as well as a deep understanding of database management systems.

Here are some of the key responsibilities of a DBA:

  1. Installing, configuring, and upgrading database management systems: A DBA is responsible for setting up new database systems and ensuring that they are configured properly. This also includes upgrading existing systems to the latest version.
  2. Database security: A DBA is responsible for implementing security measures to protect the database from unauthorized access, theft, or corruption. This includes setting up user accounts, defining access privileges, and implementing encryption and backup systems.
  3. Performance tuning: A DBA is responsible for optimizing the performance of the database by analyzing queries and indexes, as well as adjusting parameters to ensure that the database is running as efficiently as possible.
  4. Data backup and recovery: A DBA is responsible for backing up the database regularly and testing the recovery process to ensure that critical data can be restored in the event of a failure.
  5. Monitoring database activity: A DBA is responsible for monitoring the database for performance and usage trends, as well as identifying any potential problems or issues that may arise.
  6. Data migration: A DBA is responsible for moving data from one database to another, either for the purpose of upgrading systems or for consolidating multiple databases into one.
  7. Troubleshooting: A DBA is responsible for solving problems that arise with the database, whether it is a performance issue or a software bug. This requires a deep understanding of the database management system and the ability to work quickly and effectively to resolve the problem.

In conclusion, a DBA is a critical member of any organization that relies on a database for its operations. The role requires a combination of technical expertise, interpersonal skills, and a deep understanding of database management systems. A DBA is responsible for ensuring that the database is running smoothly, efficiently, and securely, and is available to users at all times.

What is a Document Database: NoSQL

On By Ben Larson Ph.D.In Analytics, databasesLeave a comment

Document databases, also known as document-oriented databases or NoSQL databases, are a type of non-relational database that store data in a semi-structured format as documents. Unlike traditional relational databases which store data in tables, with rows and columns, document databases store data in flexible, nested structures that can more closely match the data’s natural hierarchical relationships.

Some of the most popular document databases include MongoDB, Couchbase, and RavenDB. These databases are often used in web and mobile applications that need to store large amounts of semi-structured or unstructured data, such as user profiles, product catalogs, and social media feeds.

Advantages of Document Databases

  1. Flexibility: One of the biggest advantages of document databases is their flexibility. They allow you to store data in any format, without the need to define a schema beforehand. This makes it easier to add new fields or nested structures as your application evolves, without having to make changes to the database schema.
  2. Performance: Document databases are designed for high performance when dealing with large amounts of semi-structured or unstructured data. They use indexing, caching, and other techniques to speed up queries and provide quick access to the data.
  3. Scalability: Document databases are highly scalable, making it easy to add more capacity as your application grows. They can be scaled out horizontally, by adding more nodes to the database cluster, or vertically, by increasing the resources of the individual nodes.
  4. Easy to use: Document databases are designed to be easy to use, with simple APIs for storing, retrieving, and querying data. They often come with user-friendly tools for managing the database, such as web-based management consoles or command line tools.

Disadvantages of Document Databases

  1. Lack of consistency: Since document databases do not enforce a strict schema, there is a risk of data inconsistencies and data anomalies. This can make it difficult to ensure the quality and integrity of the data over time.
  2. Lack of transactions: Many document databases do not support transactions, which can make it difficult to ensure that related data updates are made atomically. This can lead to problems such as data loss or inconsistent data in the event of a crash or network failure.
  3. Complexity: While document databases can be easy to use for simple applications, they can become complex to manage and maintain as the data grows. This is especially true for large-scale applications with complex data structures and relationships.

In conclusion, document databases are a type of NoSQL database that are well-suited for storing large amounts of semi-structured or unstructured data. They offer advantages such as flexibility, performance, scalability, and ease of use, but also come with disadvantages such as a lack of consistency, lack of transactions, and increased complexity as the data grows. When choosing a document database, it’s important to consider your specific use case and requirements, and weigh the advantages and disadvantages carefully.

Data Jobs: Data Engineer

On By Ben Larson Ph.D.In Analytics, Careers in DataLeave a comment

In today’s data-driven world, data engineers play a crucial role in ensuring that data is collected, stored, processed, and made available for analysis and decision making. Data engineers are responsible for the design, construction, and maintenance of the infrastructure that supports the collection and processing of vast amounts of data. They are the foundation of the data ecosystem, making it possible for data scientists, analysts, and business users to derive insights and make informed decisions.

The role of a data engineer is to build and maintain the pipelines that transfer data from various sources to a centralized repository. They are responsible for ensuring that the data is cleaned, transformed, and made ready for analysis. This requires a deep understanding of data management, data warehousing, and the use of big data technologies such as Hadoop, Spark, and NoSQL databases.

Data engineers work closely with data scientists and analysts to understand the data requirements and design systems that meet those needs. They also ensure that the data is properly secured, backed up, and protected from unauthorized access. Data engineers must be able to write efficient and scalable code, debug and resolve issues, and optimize systems for performance and scalability.

One of the most important responsibilities of data engineers is to ensure that data is easily accessible and usable. This requires the development of data APIs and integration with other systems such as data visualization tools and business intelligence platforms. Data engineers must also be able to monitor the data pipelines and systems to ensure they are functioning as expected and make adjustments as needed.

Data engineering is a rapidly evolving field, and data engineers must be able to stay up-to-date with the latest technologies and trends. This requires continuous learning and a willingness to experiment with new tools and approaches. Additionally, data engineers must have excellent communication skills, as they often work in cross-functional teams and must be able to effectively communicate technical concepts to non-technical stakeholders.

In conclusion, data engineers play a vital role in the data ecosystem by building and maintaining the infrastructure that supports data collection and processing. They ensure that data is accessible and usable, and they work closely with data scientists and analysts to support data-driven decision making. If you are interested in pursuing a career in data engineering, you should have a strong foundation in computer science and programming, as well as experience with big data technologies and data management.

Database Design: Generalization and Specialization

On By Ben Larson Ph.D.In Analytics, Data Architecture, data modeling, databases, Free CoursesLeave a comment

•How do we handle the situation when we have multiple classes but realize the classes have a significant amount of information in common?

•How do we handle the situation when we have a single class but realize that there are differences among the different objects in that class that may drive us to break up the class into two classes?

Specialization

When the majority of the information about two types of objects are the same, but there exists some different specialized data

•Let’s say this class already exists for a small startup company

•All employees have an ID, firstName, lastName, and salary

•The startup has grown enough that they now want to hire consultants

•Instead of salary, consultants have an hourly rate

•Subclasses (or inherited classes) contain the specialized information

•Permanent and Consultant are both subclasses of Employee

•Superclasses (top classes) should be as general as possible

•Future changes to the superclass would affect all subclasses

•Easy to add additional subclasses

Generalization

When you have 2 or more existing classes and realize they have some information in common

Adding a superclass and pulling out the common information from the 2 subclasses into the superclass (the same as specialization only in the opposite direction)

Inheritance

•Generalization and Specialization are examples of inheritance

•SubClassA and SubClassB are both specialized types of SuperClass

•SubClassA and SubCLassB will have all the attributes of SuperClass in addition to their own attributes

Database Design: The Development Process

On By Ben Larson Ph.D.In Analytics, Data Architecture, data modeling, databases1 Comment

When start on the design of a database there are few questions you need to ask right away:

  • Not to be dismissive, but do not simply build a database to the specs of a request
  • Take stock of the data at hand, ask around if there is related data that also might be included
  • Think about what users will be doing with the data
  • As best you can, try to anticipate other uses for the data that come up
  • Focus on the type of data you are dealing with, and how it can be best used/store
  • Don’t only focus on the current use of the data, consider the data may find other uses as well

Now, let’s look at database development through the lens of a software development cycle

Keep in mind, this process is not simply linear. This process, in the real world, goes through many iterations. Change management is good idea to put in place. It helps you to deal with issues like:

  1. New data sources
  2. New requirements from business users
  3. Technology platform changes in your organization

To get started, we need to come up with a problem statement, something that can describe the problem we are trying to solve. Use cases are commonly used when developing problem statements. Use cases help to demonstrate how a user will be interacting with your database:

  • Transactional – think like a cash register, adding new transactions to database, updating inventor
  • Reporting base – data doesn’t change often, people want to look more at aggregate numbers over the details of each transaction. (End of the day reporting – what is the final sales total from the cash register)

You can use Unified Modeling Language (UML) to create your use case. UML uses a series of diagramming techniques to help visualize system design. Below, the stick figure represents users and each oval will show one of the tasks the user hopes to be able to use the database for. UML is great for mock-ups, but you should also include a detailed document that provides deeper insight into the tasks below.

Now imagine we want to create a database to store plant data:

Using this data, we can come up with a few basic use cases

Use case 1: Enter (or edit) all the data we have about each plant; that is, plant ID, genus, species, common name, and uses.

Use case 2: Find or report information about a plant (or every plant) and see what it is useful for.

Use case 3: Specify a use and find the appropriate plants (or report for all uses). 

How does our existing table design handle our use cases:

1.Can we maintain data?  Yes

2.Given a plant can we return the uses?  Yes

3.Given a use can we return the plant?  Not really

We need to re-examine this data from a class point of view

We are dealing with 2 classes in the Plant data:

1.Plant

2.Use

•One plant can have many uses

•This is an example of a relationship between classes

•Relationships between classes are represented with a line in UML

•We need more details besides just a line to know how classes relate

•The pair of numbers at each end of the line indicates how many objects of one class can be associated with a particular object of the other class 

•The first number is the minimum number (0 or 1 to indicate whether there must be a related object)

•The second number is the greatest number of related objects (1 or n to represent many)

Design Phase

•You spend a lot of time in the top 2 quadrants

•It is an iterative process, not simply linear

•Keep reviewing the model to ensure it satisfies the problem

•Once you have a model that works, move on to design

•For the database world, design involves:

Converting our Class Diagram into Database Objects: Tables, Keys, and Relationships

Application Phase

•Now that you have a database foundation in place, the Application phase builds upon it by adding additional functionality for the users

•The Application phase satisfies the Use Cases:

Input Use Cases will most likely be satisfied with Forms

Output Use Cases will most likely be satisfied with Reports

Back to Main Course Page: Course

Database Design: First Things to Consider

On By Ben Larson Ph.D.In Analytics, Data Architecture, data modeling, databases1 Comment

First lets start with some basic terminology

Some Relational Database basics before we begin:

Database = A collection of structured tables designed to store data

Table = A database object that stores data by organizing it into rows and columns

Field = Each column in a database table is called a field

Record = Each row in a database table is called a record

We want to create databases that will:

1.Store our data

2.Let us interact and ask questions of the data

3.Retrieve or export the data

In order to do this, we need to have a properly designed database

Example Situation #1:

A teacher puts out a sign up sheet during back to school night to allow parents to donate various classroom supplies.  The parents of the class fill out the sign up sheet and it looks like this:

The next day the teacher types up the donation sheet and puts it in Excel.  The teacher decides that he wants to try to build a database to keep track of what each person donated.  After noticing that at most someone is donating 3 items he builds a database table that looks like this:

In Excel:

Database:

The teacher is pleased.  He set out to create a database to keep track of what each parent donated.  Given a parent name he can easily see their donations.  However, when he wants to see all the parents that have donated tissues he ran into some problems. 

Problems:

•The table answers questions in 1 direction only (given a parent you can easily return donations)

•To answer the question in the other direction (what parents donated tissues?) you need to search across 3 different fields

•The table was designed without recognizing we are dealing with 2 different classes

With a parent name provided as input (ex: Greg Carter), a record in the table can be identified.  Once we have identified a record, the values for all fields in that record can be retrieved.

So, this table works to answer questions like this:

“What donations did Greg Carter provide?”

Example Situation #1 Takeaways:

1.Do not let an existing form dictate your table design (a form can be simple like a paper signup sheet or more complex like the interface of an application)

2.You must get to know your data to be able to correctly identify how many classes of data you have

3.Be careful when designing your table to answer a particular question – will you ever want to ask that question in the opposite direction?

4.Don’t rush to simply load given data into a table – take the time to think through design

Repeated Information:

•Another common problem is storing the same piece of information several times

•If you find this happening, you may be placing too much emphasis on an existing form

Example:

A business’s paper order form has customer name, address, phone number.  We wouldn’t want to create a database table to keep storing all the information for every single order.  It is inefficient and will most likely lead to inconsistencies and future problems. 

If you see repeated information in a table, it should be a huge red flag for you to re-examine the design. 

Designing for a single report:

•You can’t allow an existing form to determine your table design

•Similarly, you can’t allow a report to determine your table design

•Think about reports as nicely presented data coming out of your database

Back to Main Course Page: Course

Databases: What are they and why do we need them?

On By Ben Larson Ph.D.In Analytics, data modeling, databases1 Comment

Data scientist, data architect, data engineer, database administrator, data analyst: all of these jobs have 1 big thing in common – they all work with databases. So what exactly are databases?

A database is a software platform designed to store, manage, and manipulate data. They come in different flavors from small desktop based solutions to large enterprise databases sprawling across 100’s or 1000’s of servers.

Databases are used for all types of business and personal needs. Some common examples are transactional databases > like the ones run on cash registers in a retail store. They track inventory, prices, and sales.

Data warehouses: they are used by data analysts and business intelligence teams to run reports and build dashboards against. Also, there are unstructured databases that hold information like documents, images, audio, and free text.

From a very broad view, databases can be broken down into two categories: structure and unstructured. Another way you may describe it is SQL vs NoSQL. While not all structured databases run SQL, the vast majority do, so for the sake of this article, we will use SQL and unstructured interchangeably.

Structured SQL databases are still king of the data world at the moment. Most companies use them for both day to day functions as well as reporting and advanced analytics. These databases fall under the umbrella of relational databases.

RDBMS = Relational Database Management System

RDBMS databases stores data in one or more tables formatted into rows and columns (think Excel or Google Sheets). These tables are typically set up to handle some specific data, like an HR employee list or a student grade table. The tables can be joined using SQL to create rich datasets.

In the picture above, we have 2 tables. Using the ID from the first table, you can determine who the instructors are for each of the classes in table 2. This is, at the most basic level, a relational database.

Relational databases are normalized, meaning the duplicates are removed and big tables are broken up in to small more specific tables. This is done to save disk space as well as improve performance. I promise, I will have a whole write up dedicated solely to data normalization, but for now, just know it is a property of RDMS.

Other factors of Relation database design include:

ACID – set of rules to guide database design

Atomicity – integrity of an entire database transaction

Ex: if you are updating a person’s first name, the entire value must get updated to be successful (not just the first letter); no view of partial transactions in progress

Consistency – only data that pass validation rules are permitted

Ex: email address requires an @ otherwise it should not be written to the database

Isolation – the ability for a database to process multiple transactions simultaneously

Ex: if you are performing 2 updates, the first must complete before the second

Durability – once data is saved it should remain saved even when the machine shuts down

Ex: if a machine shuts down in the middle of an update, it should be rolled back; if an update is completed and a machine shuts down then the update should exist on the restart

Non-relational databases (NoSQL) are designed to fill in the gaps left by RDMS. These databases scale easily. Their loose approach to storing and managing data allow them to scale across multiple servers, allowing them to grow with demand and shrink as demand shrinks. They have a more flexible design, allowing for storage of data not easily handled by a RDMS such as: images, audio, documents.

When it comes to growth and scaling, there is a clear difference between the two approaches

The following articles will be based on RDMS, mostly due to the fact they are still the primary database in use today, and also they are easier to conceptualize. Don’t worry though, everything you learn here will help you to later understand the world of NoSQL unstructured data.

Back to Main Course Page: Course

Data Jobs: What do Data Scientists do?

On By Ben Larson Ph.D.In Analytics, Careers in DataLeave a comment

While generalizing any profession is difficult since so many factors can come into play: (academic versus corporate — large company versus start-up), I find the role of Data Scientist can truly only be described by breaking it apart into three very different roles. For lack of better terms I will name these roles: Academic/Research Data Scientist, Applied Data Scientist, and finally Data Analyst/Scientist. These roles require different skill sets and different mindset that I will discuss below.

First off, I would like to put my bias out front, I am an Applied Data Scientist. Now, that does not mean I feel that my current role is superior to any other data scientist role out there. Also, I work for a very large company with lots of resources, so I also want to point out that the roles I will describe below can overlap depending on your working environment.

Academic/Research Data Scientist

These are the people who design new machine learning algorithms and push the boundaries of data science and AI. When you see advances in self driving cars and computer vision, these are the people behind those advances. These individuals work either in a university setting or as part of a research team for companies like Google, Facebook, Tesla. Most either hold a PhD or are currently working on one. These data scientists actively develop and conduct research experiments that are written up and published in scientific journals. These are the pure scientists among the data science world.

To be amongst this crowd of data scientists, you need to be at the top of your game with advanced mathematics and programming skills. If you really love diving deep into the field and can handle the often glacial pace research moves at, this could be the job for you. The biggest drawback is that these jobs are limited. There are only so many academic or industry research positions out there. Most data scientists today instead fall into the other two categories.

Applied Data Scientist

Applied Data Scientists are a bit more pragmatic. They work for a company that drives their goals, instead of being funded by research grants. They do not have the luxury of time often afforded those working on grants, so the solutions they build need to go into production sooner rather than later. (Now to be fair to researchers, applied data scientists also don’t have to deal with the headaches surrounding grant proposals).

Typically, data scientists working in industry (not part of a research team) are not out developing new algorithms or trying to push the limits of machine learning. Instead, they use tools created by others to explore and derive meaning from data that can be acted upon. When the boss wants actionable data and they need it now. Most applied data scientists keep a few algorithms on hand that they know work for certain scenarios and spend most of their time gathering, cleaning, and prepping the data to build out the models.

I kind of like this position. I view myself more of an applied scientist. Even while going through my PhD, I always leaned towards applied versus theoretical.

To work as an applied data scientist, a candidate should have a master’s degree or at least 6 years industry experience. They should be inquisitive and honestly interested in the domain in which they work in. I work in cyber security, I have spent a lot of time researching and studying the field so I can identify opportunities to provide a data driven solution. Candidates should also have a wide ranging skill set beyond just ML. An applied data scientist should be well versed in tools such as dashboard development, optimization modeling, forecasting, and simulation modeling.

Data Analyst/Scientist

This is probably the most common data science position right now. What this position is calling for is basically a top level analyst familiar with data science tools. Now I am not denigrating this positon. What these data scientists do is every bit as challenging and important as the other two I described above. They are expected to operate as a data scientist while also handling analyst or business intelligence duties as well.

People in this position are not less than or incapable of doing the other roles described above, they are instead part of an organization that either does not have a fully developed data science program or they are working outside the data science organization. These data scientists are often embedded in a department providing data management and analysis expertise. Truly jacks of all trades, these data scientist often stand-up and manage data warehouses or data marts for their department. They can be expected to handle reporting duties as well as machine learning model development.

Conclusion

To sum this all up, data science is a broad and still evolving field. Solid industry definitions are not common place and titles often do not represent actual job duties. However, most data scientists can be generalized under one of the three roles I discussed above: Academic/Research Data Scientist, Applied Data Scientist, or Data Analysts/Scientists. Neither role is inherently better or more important, but the differences in the rows can definitely attract different individuals.