What are the traits of a good data engineer?

As part of their daily workflow, data engineers must deal with enormous datasets. The proliferation of sensors, websites, point-of-sale, and other types of data collection has made data engineering a discipline with enormous demand. 

Data engineers prepare large, cumbersome datasets for data scientists to analyze. Additionally, they feed data into machine-learning models. They’re often the first point of contact when it comes to transforming data into something useful. 

Big Data Institute’s Managing Director, Jess Anderson, says the most-desired skills are determined by the company’s needs. According to Anderson, he looks for “a solid understanding of the frameworks they’re using”. It is also important to be able to develop complex systems for data.”

Dan Prince, founder, and CEO of Illumisoft looks for data engineers who are able to communicate complicated ideas clearly and efficiently. Data engineers must also possess “soft skills” such as empathy and communication. They value their ability to grasp a problem, understand its context, and ask the right questions.

Prince also expects they have some experience with self-initiated projects. “Many kids will go through college or a degree curriculum without ever trying to put any of their knowledge to work outside of academia. I’m looking for people that are bold enough to try, and if they have sold their services well while still a student, that’s even better.”

What are the mandatory skills for a data engineer?

“All data engineers should be able to code,” says Melissa Benua, VP of Engineering at mParticle, “though the language itself doesn’t matter. Candidates should also be familiar with distributed systems design principles. Likewise, they should have solid database experience. They should be skilled at writing SQL—especially high-performance and cost-efficient queries for processing large datasets—and basic database administration. Knowledge of AI/ML is a bonus but is generally not considered a requirement.”

“You must have experience with tools of the trade like Apache Hadoop and Spark, C++, Amazon Web Services, and Redshift,” Prince says. You also need to know a variety of database systems, both relational and non-relational. It is necessary to understand data warehousing solutions, ETL tools, machine learning, and data APIs. You should be familiar with Python, Java, and some scale-up programming languages. Other than communication skills, presentation skills, and self-initiative, a good understanding of distributed systems, algorithms, and data structures would be a plus.”

How about some sample questions for a data engineer interview?

• Did you ever transform unstructured data into structured data?
• How would you validate the migration of data from one database to another?
• What is Hadoop? How does it relate to Big Data? How does it work?
• What Python libraries would you use to process data efficiently?
• Are you more of a database or pipeline person?
• Tell us about a distributed system you’ve created. How did you design it?
• What are your conflict management strategies? Can you provide an example?
• How do you define *args and **kwargs?
• Create a video streaming service such as YouTube or Netflix.
• Develop a consumer-facing data storage service like Google Drive or Dropbox.
• Are you familiar with PostgreSQL or another RDBMS and with NoSQL databases in general? 
• Would it be possible to construct a pipeline that would periodically upload data to S3 based on data from a queue? If so, how would you scale it?
• Create a SaaS platform that competes with Google Analytics. Would it scale? Which parts of the problem should you solve first? What tradeoffs might you make?
• Using a dataset, write SQL to answer these relevant business questions?
• How does S3 differ from a NoSQL database?
• What makes a NoSQL database better than a relational database?
• How would you diagnose a performance issue in a Spark job?
• What is shuffle sorting?
• How is Spark different from S3?

How can you become a data engineer?

Experts generally agree that learning to code is the best way to learn data engineering, then familiarizing yourself with the platforms data engineers use. Since this discipline is code-centric, you’ll need to know SQL, Python, and Java to get started. 

Data engineers are by nature problem-solvers. “Tools such as Hackerrank can be useful for sharpening particular problem-solving skills,” Benua explains. “The best interviews are based on skills and experience acquired on the job, rather than memorized algorithms or ritualized coding challenges.” We also encourage candidates to take advantage of free trials or credits in cloud providers to experiment with setting up basic ETL pipelines or simple services.

https://www.suryasys.com/technology-and-data-can-improve-access-to-mental-health-resources/