The Great Debate: Will AI Help or Hinder Data Engineering Roles?

Most people think a software engineer’s job is to develop software and, of course, it is. But what has always made a person an engineer is their drive to solve problems. The tools they use to solve those problems, on the other hand, have changed dramatically, abstracting away the complexity of carrying out the solutions engineers find.

At the dawn of computing, engineering typically involved manipulating raw binary instructions—a painstaking and tedious task. Today, engineers leverage generative AI tools to rapidly produce working code and automate repetitive tasks, freeing humans to solve more complex and impactful challenges.

At every new leap in innovation, every abstraction, skeptics have confidently predicted the newest technology would only impact certain niche scenarios and could never be trusted. Time and again, these skeptics were proven wrong.

Every time we’ve increased abstraction, the skeptics were there with fictitious, but historically accurate, uncertainties such as:

1. Assembly to Early High-Level (FORTRAN/COBOL): “Automation sounds clever, but who would trust a compiler over meticulous human assembly? Assembly code will always be necessary to ensure true performance and reliability.”
2. Structured Programming (C/Pascal): “Sure, structured programming is interesting academically—but professionals won’t put truly valuable software in the hands of these awkward languages.”
3. Object-Oriented Programming (C++/Java): “Object-oriented paradigms seem convoluted. Maybe they’re useful for GUI toys, but they’ll never be suitable for robust enterprise software projects!”
4. Dynamic/Scripting languages (Python, JavaScript): “Sure, scripting languages help in quick prototyping, but they’re too slow, too inefficient, and will never be trusted for large, real-world production systems.”
5. Frameworks and APIs (Django, Ruby on Rails, React): “Frameworks might suit hobbyist websites or small-scale apps, but good engineers wouldn’t trust someone else’s code to power significant apps.”
6. Generative AI-assisted development (GitHub Copilot, ChatGPT): “AI-generated code suggestions might be cute for simple snippets or boilerplate, but we’ll never trust it enough to handle critical programming tasks across the board!”

Yet all these predictions have proven categorically wrong. Once a new abstraction becomes sufficiently trustworthy, engineers swiftly, and almost entirely, stop worrying about what happens underneath.

Consider today’s scenario: no software engineer seriously checks if their compiled C code correctly translates to assembly language. Why bother when compilers reliably handle that flawlessly every time?

The Productivity Paradox and Why GenAI Is Already Winning at Certain Tasks 

At first glance, it seems intuitive to believe vastly greater productivity might reduce the need for engineers. However, history demonstrates precisely the opposite effect. At every major productivity increase, we’ve seen the global number of software engineers significantly increase—not decrease.

That’s for a few reasons. Increasing abstraction lowers the barrier to entry, letting more people learn to program effectively. As more people are working – and tools become more effective – productivity increases, which leads to software products becoming viable for new use cases or previously untapped markets. In other words, more markets mean more demand. And that means more engineering jobs.

So, while today’s generative AI solutions genuinely excel over human engineers in several areas, it’s not necessarily because they’re significantly smarter, but rather because they’re infinitely patient. Consider tedious tasks engineers traditionally dislike and how AI can help:

1. Generating comprehensive unit tests: Human engineers write fewer tests because they find it tedious. AI tools consistently generate reliable, comprehensive tests every single time.
2. Creating documentation: Human-written code documentation is often rushed, incomplete, or neglected. AI, on the other hand, instantly and consistently generates clear, comprehensive descriptions of codebases.
3. Commenting & reviewing merge requests: Humans often overlook details in code reviews or skip certain comments. AI generates thorough, patient reviews and comments every single time.
4. Detailed data flow diagrams: AI-generated documentation is comprehensive and patiently thorough, making diagrams humans frequently find tedious and time-consuming.

As we know, good software engineers aren’t fundamentally programmers tied to one language or toolset. They’re “problem solving engineers” first and foremost. They choose the best tools available, willingly and quickly changing programming languages and paradigms as easily as some change shoes when conditions or problems demand. Good engineers aren’t Python engineers or SQL specialists.” They’re problem solvers who currently prefer Python or SQL.

They’re also eager to adopt innovative technologies precisely because they’re eager to solve bigger, better, more meaningful problems. As a result, today’s Python, JavaScript, or Go engineers will become proficient in totally different tools tomorrow as necessary.

In other words, specific language technologies are temporary choices for good engineers. When something faster, simpler, more efficient, or more powerful arrives, good engineers joyfully adopt it and move forward.

 How to Build a Positive Engineering Future 

Generative AI lifts engineers from the drudgery of documentation, testing, reviewing mundane pull requests, and other repetitive tasks we historically dislike—but which AI handles patiently, accurately, and immediately. Moving forward, Generative AI’s new productivity layer will once again enlarge—not shrink—engineering job opportunities, roles, and impact, continuing a century-long historical pattern. 

History clearly shows that abstraction improvements don’t kill engineering jobs, they empower engineers to solve bigger and more complex problems. Generative AI isn’t replacing software engineers; it’s helping them spend time doing what they love most: solving problems. It’s an engineer’s paradise!