Data Engineers Demystified – What do Data Engineers actually do?

What do Data Engineers do?

Businesses and organisations gather data all the time – from what you buy, to where you click on a website, to how and when you use an app. But just collecting information is only the start. If you want to use it to identify new business opportunities, it has to be stored in a way that allows it to be properly accessed and analysed – and that’s where a Data Engineer comes in. They’re the specialists in consolidating data so that it is fit for its purpose, or as Charlie puts it, “getting data from A to B and making sure it’s useful on the other side.”

A Data Engineer at Zühlke will join a team with people from a number of different fields, from Software Engineers to Business Analysts to work out how the data for a specific project needs to be collected and presented. They then put a system in place to extract the data from the client side and then package it so the team can use it. It requires them to understand what the data describes and how it will be used, so they work with subject matter specialists and other technical experts to problem solve and design the best solution for the job at hand.

What skills do you need to become a Data Engineer?

Data Engineers need to have a few different skills under their belts – and one of these is coding proficiency. You need to be comfortable working in a language like Python or Java, and understand SQL, so that you can manipulate the data itself. A basic knowledge of statistics is important too. You have to be able to step back and see the wood for the trees, as it were. “Data has no meaning when we get it,” Myriam Thursch, another member of the Zühlke team, explains. “It’s what we do with it that adds relevance for our colleagues and customers.”

Once you’ve got the technical aspects around coding and statistics mastered, the complementary abilities for a Data Engineer include a whole range of soft skills, like communication and collaboration. That’s because you rarely operate in isolation in this role. “There’s not as much sitting by yourself and programming as you’d think, actually,” explains Charlie. “It’s more about interacting with different people on your team to work things out together,” he says.

Adding to this emphasis on soft skills, Suleiman Deni emphasises the need for strengths in research and problem solving. “This field hasn’t been around for 30 years like software engineering, so we do a lot of figuring things out – you can’t just open a book and find the answer,” he says. This is something Myriam also builds on when she mentions the need for flexibility. Data Engineering is never static, because the needs of the customer and user keep changing. “The solutions I use for this project might not be right for the next one, and technology moves fast so what I’m doing now could even be obsolete in a few years. So, I’d say you have to stay up to date and curious.” she concludes.

Top tips for aspiring Data Engineers

The three team members conclude that it’s the kind of career you just need to get a feel for. Charlie suggests setting yourself a challenge like producing an analysis-ready data set using resources from somewhere like the Machine Learning and Data Science community Kaggle to do this. You ideally want to work with data that needs cleaning and validation, so you can practice those skills too. “If you can find what’s missing, or what would be easier along the way, then you’re wired right to become a Data Engineer,” he explains. And a big part of this is about becoming comfortable with working with data. “Lots of people freak out when they are faced with large volumes of data,” Myriam points out. “It looks scary but when you learn to work with it, you find a way to give it meaning, and that’s what I enjoy,” she concludes.

When asked the best way to work towards a career in Data Engineering, these three Zühlke team members emphasised the need to develop your ability to problem-solve and collaborate as much as your technical abilities. This is something Suleiman sums up, saying “there’s a difference between having a specific skill and being a Data Engineer.” His advice is to approach the field by getting to grips with the fundamentals of data first, rather than starting with learning how to use the tools. He mentions browsing the subreddits DataIsBeautiful and DataEngineering, or listening to resources like the Data Engineering Podcast or the Analytics Engineering Podcast, as a few places to start.

In conclusion

One common theme that comes up again and again when talking to Data Engineers is problem-solving. The role isn’t about specific tools or technical abilities, because these can all change over time. That’s one of the reasons we invest 10% of our turnover into the continuous learning development at Zühlke – we want people to continue to evolve their skills throughout their careers.

Ultimately, the day-to-day of a Data Engineer is about ensuring the right data is stored and accessed in the right place, turning rows and columns of numbers into something people can analyse and use in their projects and products. And, that’s how the teams at Zühlke are able to do amazing things for clients and end users.

If the kind of problem-solving our Data Engineers have described appeals to you, browse our open roles and find your fit with Zühlke.

So, Data Engineering in brief:

• Data Engineers work to organise data and make it accessible to a broader team, including Data Scientists and Business Analysts.
• The role involves both technical and analytical skills, requiring individuals to problem solve and come up with innovative data management solutions.
• The best way to see if a career in Data Engineering is right for you is to get your hands dirty (metaphorically speaking), trying out data projects yourself and connecting with Data Engineering communities online.