This article is written in my personal capacity. The views expressed are my own and do not represent Amazon or my employer. Examples are based on public information and synthetic scenarios; no confidential or proprietary material is used.

As Data Engineers, we learn to trust a certain set of tools. We learn SQL until joins become instinct. We learn to read an execution plan, trace a failed job, question a row count, and look twice when a number appears too clean. We build judgment slowly—one dataset, one incident, one hard lesson at a time.

Then GenAI arrives.

It can write SQL, draft code, explain an error, propose a model, prepare a design document, and summarize a meeting. Work that once began with an empty editor can now begin with a conversation.

But the difficult parts of Data Engineering often sit behind the code: data that changes without warning, errors that hide behind clean row counts, and business meaning that was never written down. GenAI can generate an artifact quickly; it cannot supply context it was never given.

So I want to bring the GenAI conversation down from the clouds and place it on the Data Engineer's desk. I have written separately about what this shift looks like in pipeline development. Here, I want to take one step back and look at Data Engineering as a whole: a high-level, non-exhaustive view of how GenAI is entering our day-to-day work.

The article is organized into three sections:

1. The changing layer: How GenAI is affecting common DE skills and tools, including prompting, specification-driven development, and agentic coding.

2. What must not change: The Data Engineering principles that every generated solution still has to preserve.

3. Where Data Engineering is going: How self-service and AI application development may broaden the role of the DE.

Part I: The changing layer—skills and tools

Understanding requirements and data

GenAI can help Data Engineers review business requirement documents, source documentation, data dictionaries, existing designs, and the data itself. It can summarize material, compare definitions, identify contradictions, draft profiling queries, and organize questions. Work that was traditionally time-consuming can become much faster once the relevant context is available.

That last condition matters. GenAI can only work with the information it can access. It is still the DE's responsibility to interview business users and source-data owners, understand consumers, collect missing requirements, and make the necessary metadata available. If the context is incomplete, the model may organize the wrong story very efficiently.

What changes: Once a DE has assembled reliable context, GenAI becomes a strong research and discovery assistant across documents, metadata, and data. Gathering and validating that context remains Data Engineering work.

SQL and data exploration

SQL is one of the clearest examples of change. A Data Engineer can describe the required result, provide schema and business context, and ask an LLM to generate the query. GenAI can also explain unfamiliar SQL, translate dialects, create profiling and reconciliation queries, produce mappings, and suggest optimizations.

The benefit is real: a DE may write much less SQL by hand. But a query can be syntactically correct, return plausible—or even correct—sample results, and still be too expensive at production scale. In my experience, LLMs do not consistently produce efficient SQL for complex data processing. The engineer still has to examine joins, grain, filters, data distribution, execution plans, scans, shuffles, and cost.

The responsibility for result validation also becomes heavier when more SQL can be produced in less time. A business decision made from generated SQL still rests on logic and data we chose to trust.

What changes: SQL generation becomes much faster. SQL expertise moves toward reviewing complex logic, validating result data, and ensuring that a correct answer is also produced efficiently.

Coding

I see strong benefits for coding. Data Engineers can now produce Python scripts, utilities, test cases, configuration, and automation quickly. Python remains a primary language for many DEs, but GenAI also lowers the barrier to working in languages and frameworks that were previously less accessible.

I find it useful to separate this into two kinds of coding.

Foundation-building code includes framework components, connectors, utilities, APIs, infrastructure code, and developer tooling. GenAI can accelerate both design and implementation here: generate scaffolding, explain libraries, propose interfaces, refactor code, and build tests. A DE can move from an idea to a working implementation with much less mechanical effort.

Data-processing code carries many of the same concerns as generated SQL. The code may run and still mishandle grain, duplicates, late data, nulls, ordering, memory, or scale. The model needs source context and explicit processing semantics, and the DE must validate the resulting data.

Prompting itself now ranges from a quick instruction to a full specification. Vibe coding can be useful for exploration. Production work needs more: interfaces, constraints, examples, acceptance criteria, tests, and the surrounding repository context. Agentic tools can perform more steps, but they are most useful when the DE defines the path they must stay within.

What changes: The distance from design to implementation shrinks, and DEs can work across a broader technical surface. Foundation-building code benefits greatly; data-processing code still demands deep data validation.

Data modeling and semantics

GenAI can automate much of the legwork around data modeling. It can inspect definitions, draft DDL, draw a first version of a model, compare designs, fetch related metadata, identify naming inconsistencies, and prepare questions for review.

Again, the quality of the result follows the quality of the context. A model cannot know whether “customer” means a person, account, household, or source-system record unless the organization has provided that meaning. The DE is often the person bringing together source behavior, consumer requirements, history, scale, and platform constraints.

What changes: GenAI makes model exploration and documentation faster. For high-impact or large-scale models, the DE remains the design owner because the DE holds the wider system context.

Infrastructure, platforms, and architecture

The Data Engineer still decides the architectural direction. GenAI can support that decision by researching options, comparing services, explaining configuration, and surfacing trade-offs. Once a direction is chosen, it can draft infrastructure code, generate framework scaffolding, and help investigate performance issues. This shortens the distance from research to implementation without handing architectural ownership to the model.

Architecture still depends on workload shape, existing systems, team capability, security requirements, cost, operational maturity, and failure tolerance. Those constraints rarely fit inside one prompt. GenAI can widen the option set; the DE still has to choose what belongs in the system.

What changes: Research and implementation accelerate. The DE continues to set the direction and make the architectural decisions.

Testing and data quality

Testing in Data Engineering often looks different from testing in a conventional software system. DEs may not always build large unit-test suites, but we do validate happy paths, bad paths, malformed input, boundaries, reconciliation results, and expected data behavior.

This is one of GenAI's clearest wins. LLMs can generate synthetic bad data quickly: nulls, duplicates, malformed records, schema drift, late data, unusual values, and partial files. They can build input variations and expected-result scaffolding that would be tedious to produce manually.

The DE still decides whether those cases are realistic and whether the expected result is correct. But the cost of creating a broad test set drops sharply.

What changes: Failure-oriented data generation and test preparation become much easier. DEs can test more paths without spending most of their time constructing test data.

Operations and troubleshooting

DE teams can build custom agents to inspect errors, read logs and runbooks, compare failed and successful runs, suggest solutions, and—within approved controls—restart jobs or perform recovery steps.

This has the potential to reduce on-call burden significantly. Common and well-understood failures can be diagnosed or resolved faster, while the engineer focuses on unfamiliar incidents and wider system impact. The important boundary is authority: an agent that can take production action needs limited permissions, clear approval rules, and a complete record of what it did.

What changes: Operational assistance can move from summarization toward controlled action. On-call work may shift from handling every known failure manually to supervising automation and solving the failures that do not fit an established pattern.

Documentation, communication, and knowledge transfer

Documentation may be one of the most immediate and personal GenAI benefits. This blog itself is an example. English is not my first language. My process is to sketch raw ideas, research them, give them structure, and then use GenAI to help with the writing. The ideas and judgment remain mine, while the tool helps me shape them for a reader.

The same approach applies to design documents, runbooks, data dictionaries, onboarding guides, change summaries, meeting notes, and stakeholder communication. A DE can begin with rough technical material and turn it into a clearer document for different audiences.

The limitation is the source material. GenAI can polish an incomplete explanation until it sounds finished. It may describe what code appears to do rather than why the system was designed that way. Decisions, trade-offs, owners, and business context still need to be supplied and reviewed by people.

Communication also remains more than writing. GenAI can prepare an agenda or draft an incident update, but it cannot own a commitment, resolve a disagreement, or build trust with a consumer.

What changes: GenAI lowers the language and drafting barrier and makes knowledge easier to shape and share. The DE still owns the meaning, accuracy, and commitments behind the words.

Part II: What must not change

Tools can evolve quickly. The properties we expect from a trustworthy data system cannot.

Data integrity, correctness, and fitness for use

Data must preserve its grain, keys, completeness, uniqueness, relationships, and business meaning. Generated SQL or code can execute successfully and still produce data that is unfit for its consumer.

Using GenAI responsibly now becomes part of the DE's work. Generated logic needs systems that verify it: evaluation harnesses, deterministic checks, data-quality controls, approval boundaries, and observable results. The faster we produce an implementation, the more important it becomes to prove that its output is correct and fit for use.

Explicit contracts, reliability, and recovery

Schemas, keys, freshness, ownership, quality expectations, and change behavior must remain explicit. Production systems still need idempotency, deterministic behavior where required, safe retries, replayability, backfills, fault handling, and controlled recovery.

At scale, deterministic processing should remain the foundation. LLMs should be introduced where interpretation or decision-making adds value—not inserted into every row-level operation merely because they are available. The deterministic system provides the rails; the model helps only where judgment is actually required.

Scalability, performance, cost, and observability

A generated solution may work on a sample and fail at production volume. DEs still need to validate scale, concurrency, latency, data movement, resource use, and cost. Systems also need actionable monitoring, lineage, diagnosable failures, visible downstream impact, and named owners. An AI assistant cannot explain what the system never measured.

Security, privacy, and shared responsibility

Least privilege, approved data handling, auditability, provenance, and controlled production access remain non-negotiable. GenAI adds new questions about what context can be shared, what generated artifacts can be trusted, and what an agent may execute.

Security and privacy cannot sit only with the platform team. Analysts, data owners, and other users who gain the ability to create data solutions must understand the boundaries as well. Broader access to creation requires broader awareness and shared responsibility.

These principles are not an argument against GenAI. They are the conditions under which we can use it responsibly.

Part III: Where Data Engineering is going

The next shift is bigger than Data Engineers getting better assistants. Agentic tools are beginning to change who can create data solutions—and what Data Engineers themselves can build.

More people will create data solutions—in partnership with DEs

Analysts and data owners may understand a particular business function or dataset more deeply than a DE whose responsibilities stretch across many domains and systems. As agentic interfaces improve, these users will increasingly be able to describe a source, transformation, destination, and schedule and participate directly in creating simpler pipelines.

This should happen in partnership with Data Engineers. It can remove unnecessary handoffs and bring domain knowledge closer to implementation, but ungoverned creation can also produce duplicated logic, uncontrolled cost, weak ownership, and systems that nobody is prepared to operate.

DEs are well positioned to lead this change by building the platform, approved components, contracts, permissions, automated validation, observability, and escalation paths that allow other users to create safely. The DE does not disappear from the process; the DE creates the environment in which more people can contribute.

Data Engineers will build data-focused AI applications

GenAI brings application development within closer reach of Data Engineers. That does not make it an overnight role change. DEs will still need to learn new tools, application patterns, deployment mechanisms, and operational practices before building full production applications.

I believe that investment is worth making because this is where part of the role is heading. DEs can build applications around data quality, metadata, operational investigation, evaluation, and infrastructure for AI data requirements. These applications need reliable data, clear semantics, controlled access, scale, and observability—the same areas where DEs already carry deep experience.

Complex data systems will remain Data Engineering work

Agentic tools make implementation easier; they do not remove complexity. High-volume, real-time, high-risk, cross-system, and operationally critical pipelines still require deliberate design and experienced ownership.

DEs will also use agents to build these systems. What is easy or difficult for GenAI will depend heavily on what is available for it to work with. A task becomes easier when the model has reliable context, metadata, examples, tools, and clear boundaries. A large part of what makes production Data Engineering difficult is precisely what is missing, unknown, inconsistent, or learned only through experience. That is where the DE's wider understanding remains essential.

Where does this leave Data Engineers?

Creating a pipeline or another data artifact may no longer belong only to Data Engineers. That does not make the role smaller. It gives the DE more ground to cover.

DEs will partner with analysts and data owners to create simpler solutions safely. They will lead the platforms and guardrails that make broader creation possible, continue to own complex and scaled production systems, and move further into data-focused AI applications and the infrastructure that supports AI data requirements.

As that scope expands, data quality, security, and privacy become even more important. The DE's value moves beyond producing an artifact toward designing the environment in which many artifacts—human-written and AI-generated—can be trusted.

The road ahead has no final map

I do not know exactly what happens next. None of us do. We may see increasingly autonomous Data Engineering agents or models built specifically for data work. I remain skeptical that production Data Engineering will become fully autonomous, because context, accountability, scale, and organizational meaning are not contained in code alone.

More people may be able to build with data. We will work beside them, build the systems that make that work safe, and step into AI application development ourselves. And when the context is missing, the scale is real, or the answer must be trusted, the Data Engineer will still be in the room.

The tools are changing hands. The responsibility is not.

This article is written in my personal capacity. The views expressed are my own and do not represent Amazon or my employer. Examples are based on public information and synthetic scenarios; no confidential or proprietary material is used.

Generative AI is already changing how software is designed and built. This article explores what that shift means for Data Engineers—and what it must not change. It is written for Data Engineers and software engineers who already understand production data systems. The goal is to examine where Generative AI can accelerate familiar work while preserving the properties that make data trustworthy.

The article uses pipeline development as one example. Data Engineering is much broader: it includes analytics, data platforms, storage and compute systems, streaming, governance, semantic layers, developer tooling, and data products. Pipelines give us a concrete responsibility through which we can compare traditional and GenAI-assisted workflows.

Data Engineer's Responsibilities

Before looking at acceleration, it helps to establish the engineering baseline. This is a refresher, not a lesson: experienced Data Engineers will already recognize these responsibilities and tenets.

Data Engineers design and operate the systems that make data available for products, analytics, and decisions. Their work can include data platforms, storage and compute, ingestion and transformation frameworks, streaming systems, data quality, lineage, governance, security, developer tooling, and data products.

GenAI capabilities for Data Engineers

GenAI can help across many parts of Data Engineering. For this article, the capabilities are intentionally stated at a high level before we apply them to one example pipeline workflow.

A traditional enterprise pipeline framework

Imagine you are a Data Engineer at a large retail company. Operational data arrives from Cassandra, SQL Server, and text files produced by external or legacy applications. The data must be extracted, checked, staged, and loaded into an enterprise data warehouse. Analytics teams, BI users, and other Data Engineering teams consume the resulting datasets.

At enterprise scale, this is not one pipeline. There may be hundreds or thousands of pipelines following the same broad lifecycle. Reimplementing source connectivity, staging, validation, loading, retries, alerting, and audit logic for every dataset would create unnecessary variation and support cost.

The usual answer is a reusable, configuration-driven ingestion framework. A pipeline configuration identifies the source type, extraction behavior, staging location, schema, target table, load method, schedule, and operational owner. A large extract might land as Parquet files in object storage, become visible through a staging table, pass structural validation, and then load into the warehouse using the appropriate strategy: append, merge, snapshot, or slowly changing dimension Type 2.

The framework owns common capabilities such as idempotent execution, audit columns, schema and data-type validation, logging, alerting, and run metadata. Each pipeline still owns use-case-specific meaning: expected freshness and volume, valid business values, reconciliation rules, acceptable null rates, and whether a failure should block downstream consumption. Some capabilities, such as orchestration and retries, may be provided by an orchestrator or scheduler.

The framework standardizes mechanics; the pipeline configuration and its owners define meaning.

Traditional onboarding workflow

Assume the company needs to onboard a daily inventory snapshot from SQL Server. The target grain is one row per store, product, and snapshot date.

The framework automates execution, but onboarding still requires considerable manual investigation, translation, and artifact creation. In the diagram below, the orange stages are owned by the Data Engineer; the blue stage is the existing production framework that executes the approved configuration.

The framework guarantees the repeatable mechanics: supported source connectors, staging conventions, schema and data-type checks, standard load behavior, retries, audit metadata, logging, and alerts. The Data Engineer remains responsible for the decisions that give the pipeline meaning: source interpretation, grain and keys, merge or history semantics, business rules, acceptable quality, performance expectations, ownership, and downstream impact.

The most time-consuming part is often source and business discovery. Before writing any configuration, the engineer may need to read business requirements documents, source documentation, existing mappings, database diagrams, and data dictionaries, and speak with source owners and consumers. The DE must then translate that context into technical decisions and implementation artifacts.

Applying GenAI to traditional pipelines

Assume that the enterprise pipeline framework already exists. The most immediate GenAI opportunity is therefore not to replace it, but to accelerate the manual development work around it.

LLM-assisted pipeline onboarding

With GenAI, a DE can design an agent which understands existing workflow for the existing framework. The Data Engineer provides a natural-language description of the pipeline requirement, together with relevant context about how the framework operates.

The assistant first structures and clarifies the discovery material. It then drafts the same atomic artifacts that the traditional workflow required the Data Engineer to create manually: source and staging decisions, target-table and load-strategy decisions, extraction SQL, DDL, mappings, and finally the framework configuration. It can also create failure-oriented test-data files for review.

The diagram below now shows how LLM assited development will look like.

The LLM performs much of the legwork that previously required the Data Engineer to start from an empty file: reading discovery material, finding a pattern, making a structured proposal, drafting SQL and DDL, creating mappings, assembling configuration, and producing failure-oriented test data.

Quick Comparison of Tradational and AI assisted development:

The human remains in the loop. The Data Engineer verifies the discovery context and generated outputs, confirms the grain and keys, checks the load semantics, validates that failure data is realistic, and decides whether the resulting pipeline is fit for use. A data owner may also confirm whether a quality rule should block publication or only raise a warning.

The first benefit is minimal-impact acceleration: GenAI reduces manual construction effort while the existing framework, engineering controls, and human accountability remain in place.

LLM assistance can continue after deployment. When the production framework raises a data alert, the alert owner can open a separate LLM assistant to inspect logs and run metadata, identify the likely error, and recommend a solution. This can shorten investigation time for issues reported by business users or detected by pipeline controls. The assistant remains outside the pipeline framework and does not autonomously modify or rerun the pipeline. It is LLM-assisted incident investigation, not yet an agentic pipeline.

Agentic pipelines

So far, GenAI has operated around the existing framework: it helps a human create artifacts or investigate an alert. Agentic pipelines bring governed GenAI capabilities into the framework itself.

The future of data pipelines is clearly moving toward agentic systems. Data Engineers are now figuring out how to build deterministic frameworks that can be developed and operated with agentic tools, and how to use GenAI capabilities to make pipelines smarter.

In a traditional configuration-driven framework, a configuration entry may directly call a particular Python function or operator. In an agentic framework, configuration can refer to an approved skill with a defined purpose, inputs, outputs, permissions, and validation behavior. A pipeline agent reads the configuration, selects and executes the requested skills, and records the outcome.

The pipeline agent sits at the center of the workflow, consumes a configuration file, and executes a controlled set of skills around it. Candidate skills include reading text files, reading from Cassandra, reading from SQL Server, writing local files, writing S3 files, running database SQL, aborting a pipeline, and raising an email or ticket alert.

The central idea is that the agent executes approved skills according to configuration. It does not receive unrestricted authority to invent arbitrary pipeline behavior. Skills should have clear contracts, bounded permissions, test suites, and observable outcomes.

Adding skills should be easier than adding bespoke pipeline code, while the framework retains governance and repeatability. Testing should be part of the agentic process: each skill should expose its expected inputs, outputs, failure behavior, and evaluation cases.

From reliable frameworks to agentic pipelines

GenAI adoption does not require replacing a pipeline framework that already works. A mature framework may carry technical debt, but it also encodes hard-won operational knowledge through its connectivity, validation, load behavior, retries, auditing, and alerting. The practical approach is to preserve those reliable mechanics, address technical debt where it limits change or supportability, and use GenAI to reduce the manual work around them.

The immediate opportunity is to onboard and modify conventional pipelines more efficiently. GenAI can translate requirements into configuration, SQL, DDL, test data, and documentation, while Data Engineers continue to review the decisions and own the production outcomes.

The longer-term direction is agentic. Capabilities can be exposed as governed skills that an agent selects and composes through configuration. Within defined permissions and validation boundaries, the agent can interpret pipeline intent, execute steps, evaluate results, and escalate uncertain decisions to a Data Engineer. This is something i would explore morei n future blogs.

The architecture may evolve, but the engineering standard does not. Whether a pipeline is developed manually, assisted by an LLM, or executed through an agent, it must remain correct, repeatable, observable, recoverable, secure, maintainable, and fit for use.

Personally, this is what makes the future exciting to me. GenAI lowers barriers that previously limited how quickly an idea could become a working system—including familiarity with every programming language, experience building application-level software, and the time required to explore unfamiliar architectures. It gives Data Engineers the ability to work across a broader technical surface, build beyond their traditional boundaries, and think at a scale that was difficult to reach before. The opportunity is not simply to write code faster, but to expand what we are able to imagine and build.