Tag: data jobs

Data Jobs: Data Engineer

On By Ben Larson Ph.D.In Analytics, Careers in DataLeave a comment

In today’s data-driven world, data engineers play a crucial role in ensuring that data is collected, stored, processed, and made available for analysis and decision making. Data engineers are responsible for the design, construction, and maintenance of the infrastructure that supports the collection and processing of vast amounts of data. They are the foundation of the data ecosystem, making it possible for data scientists, analysts, and business users to derive insights and make informed decisions.

The role of a data engineer is to build and maintain the pipelines that transfer data from various sources to a centralized repository. They are responsible for ensuring that the data is cleaned, transformed, and made ready for analysis. This requires a deep understanding of data management, data warehousing, and the use of big data technologies such as Hadoop, Spark, and NoSQL databases.

Data engineers work closely with data scientists and analysts to understand the data requirements and design systems that meet those needs. They also ensure that the data is properly secured, backed up, and protected from unauthorized access. Data engineers must be able to write efficient and scalable code, debug and resolve issues, and optimize systems for performance and scalability.

One of the most important responsibilities of data engineers is to ensure that data is easily accessible and usable. This requires the development of data APIs and integration with other systems such as data visualization tools and business intelligence platforms. Data engineers must also be able to monitor the data pipelines and systems to ensure they are functioning as expected and make adjustments as needed.

Data engineering is a rapidly evolving field, and data engineers must be able to stay up-to-date with the latest technologies and trends. This requires continuous learning and a willingness to experiment with new tools and approaches. Additionally, data engineers must have excellent communication skills, as they often work in cross-functional teams and must be able to effectively communicate technical concepts to non-technical stakeholders.

In conclusion, data engineers play a vital role in the data ecosystem by building and maintaining the infrastructure that supports data collection and processing. They ensure that data is accessible and usable, and they work closely with data scientists and analysts to support data-driven decision making. If you are interested in pursuing a career in data engineering, you should have a strong foundation in computer science and programming, as well as experience with big data technologies and data management.

Data Jobs: What does a Data Architect do?

On By Ben Larson Ph.D.In Analytics, Careers in Data, UncategorizedLeave a comment

If experience has taught me anything, it is that while companies and organizations have gotten much better at collecting data, most of this data is still just stored away in unstructured data stores. So while the data is “technically” there, it is not a whole lot of use to anyone trying to build a report or create a machine learning model. In order to get actually use out of all the data being stored, it needs to organized into some sort usable structure: Enter the Data Architect.

Data Architect is a job title I have held on multiple occasions throughout my career. My best description when explaining the job to other people is that I was kind of like a data janitor. You see, everyone takes their data and dumps it into the storage closet. So you find some billing data on the second shelf, tucked away behind employee HR records. The floor is cluttered with server logs and six years worth of expense reports stored as PDFs with completely useless names like er123.pdf.

As a data architect, it was my job to try to organize this mess and put the data into some sort of structure that lends itself to reporting or modeling. So data architects have to be well versed in data modeling, data storage, and data governance.

Data Modeling

ERD diagram showing an HR database design

Data modeling is basically taking raw data dumps and organizing them into structure that fit the needs of company. It could involve creating an HR database like above or creating a series of aggregated tables designed for reporting or dashboarding. It is the job of the data architect to best fit business needs to a data platform: be it a transactional database, a data warehouse, or perhaps a data lake.

Data Storage

Data architects also need to address data storage. While I often defer to the server crew in the IT department, as a data architect I do advise on to how and where to store data. Cloud infrastructure and cheaper faster disk storage has made a lot of data storage decisions easier, but it is good to have a working understanding of storage platforms.

Data Governance

Data governance is all about how the data is managed from a regulatory and security standpoint. It is the practice of deciding who can have access to what data. Can some data be “outside facing” versus should some data sit behind multiple firewalls in a DMZ zone.

You will often work hand in hand with Legal and Security departments when figuring out how data governance practices will be implemented.