Digital Government Productivity & Transformation
Breaking the Bottleneck: How Human-AI Collaboration is Reshaping Data Engineering
22 July 2026
Learn how human-AI collaboration can help data teams deliver faster, more trusted data pipelines.
A data pipeline shouldn't take weeks to build, yet for many organisations, it does. Not because the work is complex, but because the same small group of data engineers is stuck writing the same boilerplate code. And this issue recurs for every single data pipeline in the queue.
The bottleneck isn't talent. It isn't ambition. It's a workflow that has yet to catch up with what AI can now do.
The challenge: A growing backlog and a team that can’t scale fast enough
Building a data pipeline today still involves a great deal of manual, repetitive work, and as demand grows, that work quietly becomes unmanageable for the data engineers.
Why it happens
Within most data teams, roles are split along a clear divide. Data users act as architects: they document business rules, define requirements, and understand the data deeply. Data engineers act as builders: they take those requirements and translate them, line by line, into working code by hand. The result is that data engineers spend the bulk of their time on repetitive, low-value tasks, writing boilerplate code and rebuilding similar validation checks from scratch, request after request, rather than on the higher-value engineering work they’re best placed to do.
Meanwhile, data users are confined to a narrow, passive role. Despite holding the deepest understanding of the data, they lack the production-level coding skills to act on it directly. Hence, they write documentation, gather requirements, and wait, while their expertise sits largely untapped.
Data user: A data user is someone who deeply understands a specific data domain but doesn’t have the technical skills to code or build data pipelines themselves.
Why it matters
Every new data initiative joins the same queue, competing for the same limited pool of data engineers. Backlogs grow, timelines stretch, and teams find themselves turning down or deprioritising work, not for lack of ambition, but for lack of bandwidth.
The deeper impact is structural. While the number of data initiatives an organisation can support is determined by headcount rather than demand or opportunity, scaling becomes impossible. Adding more data initiatives doesn’t accelerate progress, it just makes the bottleneck worse.
Why the time to change is now
Demand for data infrastructure keeps rising, but the pool of qualified data engineers remains limited and hiring more isn't a sustainable fix. The untapped resource is already in the room: data users who understand the data well but lack the tools to act on that knowledge directly. Giving them those tools, while freeing data engineers from repetitive manual work, is where real, scalable progress begins.
The fix: A data framework built for scale
The solution isn’t a bigger team, it’s a smarter way of working. A well-designed data framework removes the manual handoffs that create backlogs in the first place, by giving data users the tools to act on their own knowledge directly and automating the repetitive work that consumes engineering time.
What a good data framework should look like
Empowers data users
Rather than funneling every data requirement through an engineer, a good framework gives data users the tools to define data rules themselves, removing the friction of a middleman and putting expertise where it’s most useful.
Closes the gap between requirements and implement
Instead of data engineers manually interpreting and transcribing business logic into code the framework does the translation automatically, turning plain-text requirements into machine-readable codes instantly.
Automates repetitive tasks
Manual scripting gives way to generated code, freeing engineers to focus on strategic system architecture rather than boilerplate. This shifts their role from sole builders into quality assurance architects and human-in-the-loop reviewers, meaning a single engineer can support multiple data initiatives at once.
Builds compliance into the process
Compliance rules are baked directly into the initial templates, making governance the natural starting point of the data pipeline lifecycle rather than a hurdle at the finish line.
Changes how teams work together
It reshapes daily workflows and accountability, elevating data users from passive requirements-gatherers into active, hands-on co-creators, with real ownership over the data ecosystem they know best.
Inside GovTech’s AI for Data Engineering (AIDE) framework
AIDE is a human-AI collaboration framework that changes how roles are divided within a data team. Instead of writing requirements and waiting for a data engineer to build them, data users can now work directly with an AI assistant to generate the technical components needed to bring a data pipeline to life: the data contract that defines the data's structure and rules, a test suite that checks those rules are followed, and the data pipeline itself. The AI then runs the pipeline against the test suite, automatically adjusting it until it passes. Data engineers are no longer the sole builders, their role shifts to a final check, reviewing the AI-generated pipeline and test results before approving it for use.
How the AIDE Framework works
1. Generate the Data Contract
The data user provides the AI with a small sample of data (no more than 10 records) called a Golden Dataset, along with plain-language context describing what the data should look like. From this, the AI drafts a machine-readable data contract, a document that defines the data's structure and rules.
Compliance requirements, such as those under Singapore Government Data Management Framework (SG-DRM), are built into the contract from the outset, rather than checked or added later.
2. Generate the Test Suite
The AI converts the data contract into a set of test cases that check whether the data follows the rules defined. This process often surfaces gaps in the data contract that the data user hadn't considered.
The data user reviews the gaps and refine the data contract, and the AI updates the test cases accordingly. This cycle repeats until the underlying data contract is sound.
3. Build the Pipeline
Once the contract is approved, the AI automatically builds the data pipeline itself, without requiring manual coding. Before pipeline generation, users can specify coding standards and generate custom validation functions based on their business rules, ensuring the pipeline satisfies any additional data requirements. The pipeline is then checked against the test suite, and the AI continues adjusting it until all tests passes.
4. Validate when pipeline has gone live
Once the pipeline is live, VOWL, GovTech's Open-Source Data Quality validation engine enforces the rules defined in the contract. It continuously monitors the real data flowing through the data pipeline. Data Engineers remain involved at this stage as human-in-the-loop reviewers, stepping in when VOWL flags an issue to investigate and resolve it with the data user, ensuring data quality is maintained throughout the pipeline's active lifecycle.
From proof-of-concept to practice
AIDE began as a proof of concept, a way of testing whether human-AI collaboration could meaningfully change how data pipelines are built. The early results are promising. By leveraging AI to take over the mundane, repetitive parts of pipeline development, it frees data engineers to focus on higher-value work, while giving data users a direct way to act on their domain expertise without joining a queue. If the approach holds at scale, it could set a strong example for organisations on how modern data pipelines ecosystem is built: secure, compliant, and efficiently delivered from the start.
Hear directly from the teams who are already putting AIDE into practice
AIDE has been an eye-opening experience as it requires us to re-think some of our existing data management processes in order to allow for better streamlining of our works. More importantly, it is a step towards a larger effort to uplift and standardise Whole-of-Government’s data practices.— Tan Shao Xuan, Data Manager, URA
One of the biggest takeaways from our experience with AIDE is that it’s more than an AI tool, it’s a practical framework for data engineering. That makes it easier for different teams and organisations to adapt the framework to their own needs while accelerating delivery, improving consistency, and maintaining engineering quality.— Ng Hong Quan, Senior System Analyst, URA
How to scale your organisation’s data team
The principles behind GovTech's AIDE framework: empowering domain experts, removing manual handoffs, and automating repetitive work aren’t exclusive to government. Any organisation looking to break through the data engineering bottleneck can apply them.
6 tips to get started
1. Start with representative sample data
Rather than relying on long text documents to describe your data, begin with a small set of real, clean records. It's far easier to spot issues early when you're working with actual data rather than descriptions of it.
2. Bake compliance in from the start
Don't treat data policy compliance as an afterthought. Building the relevant rules directly into your data contract from the start means governance is enforced by design, not scrambled for at the finish line.
3. Define "good data" before building
Take time upfront to clearly define what correct, complete data looks like for your use case. Hidden problems are far cheaper to fix before a data pipeline goes live than after.
4. Let automation handle the repetitive work
Once your foundations are in place, use automation to take over manual scripting. This frees engineers to focus on reviewing outputs rather than writing code from scratch, allowing a smaller team to support more initiatives at once.
5. Keep humans in the loop
Automation handles the build, but people should own the decisions. Have someone review and approve outputs at each key stage, particularly when handling sensitive data.
6. Build on proven practices
Draw on proven frameworks and best practices developed from real-world data engineering experience, and adapt them to fit your own systems and context.
Looking ahead: Human AI-collaboration in data engineering
The engineering bottleneck that slows so many data teams down isn't inevitable, it's a result of how work has traditionally been divided. When data users are equipped to directly build and validate pipelines rather than simply describing what they need, and data engineers are freed to focus on higher-level review rather than manual construction, organisations can finally move at the pace that demand requires.
This is what that shift could look like in practice. Based on the early trials, AIDE show the potential to significantly reduce pipeline backlogs, as boilerplate coding and validation work move from data engineers to data users and automated workflows, allowing a smaller, more capable team to support many more data initiatives. Pipelines reaching production could also become easier to trust, with fewer failures and incidents, continuous validation from day one, and less need for manual corrections. None of this would require additional headcount, just the same data team, equipped with sharper tools and a more efficient way of working. That's the real shift human-AI collaboration makes possible: not a bigger team but one that can build faster, safer and at a scale that wasn’t possible before.