AI/ML ETL pipelines

The model is the easy part. The pipeline is the product.

Most AI projects do not stall because the model is incapable. They stall because the systems around the model are unreliable, unmeasured, or impossible to change safely.

A proof of concept works against a carefully prepared dataset. The demo gets a good reaction. The organization decides to proceed. Then the real data shows up, spread across databases, document stores, APIs, spreadsheets, and third parties. Records are duplicated. Schemas change without notice. Documents are out of date. Embeddings need to be regenerated. An upstream service fails quietly, and nobody notices the search index has been stale for three days. The model still answers with confidence. The data underneath is wrong.

I design and build the data infrastructure that makes AI and machine learning systems dependable: ingestion, transformation, enrichment, indexing, versioning, evaluation, and monitoring. That includes ETL and ELT, event-driven pipelines, retrieval-augmented generation, vector search, and the operational machinery between an interesting prototype and a system you can trust.

I want a repeatable way to get the right information in, measure the quality of what comes out, and know when something has changed.

I recognize the engagement when nobody can say whether quality moved after the last deploy, when retrieval returns documents that should have been deleted, or when the prototype only works while the engineer who built it is watching.

A model is only as useful as the information around it

Models, prompts, agents, and inference are part of the system. Before a model can retrieve anything useful, that information has to be found, extracted, validated, normalized, transformed, and indexed. Before a model can be trained with any reliability, the inputs need consistent definitions, reproducible transforms, and a real split between training and evaluation. Before anyone should trust the application in production, its behavior has to be measured against expectations that mean something. And someone has to know when the pipeline is failing.

This is where projects get difficult. Ingestion is a one-off script. The transform lives in a notebook nobody has opened in months. The vector index holds several generations of embeddings and no record of which model produced them. A change to chunking improves one set of questions and makes another set worse. Without versions and an evaluation, improvement and regression look the same.

I replace that uncertainty with data flows you can observe, transforms you can reproduce, and quality you can measure.

What I bring

Ingestion you can trust

These pipelines pull from sources with different formats, schedules, access rules, and failure modes: relational databases, object storage, document repositories, SaaS applications, external APIs, event streams, and unstructured files.

I start by naming the sources of truth and defining how information enters. I set ingestion contracts, who owns each source, how often it updates, what volume to expect, and what should happen when an upstream system is unavailable. Depending on the requirements that is a scheduled batch, incremental sync, change data capture, webhooks, or a stream.

I design the jobs to be observable, recoverable, and idempotent where that is practical. Running twice should not create duplicate records. A connection that dies halfway through should be resumable, or safe to repeat. A schema change at the source should fail visibly, not corrupt everything downstream. I also account for deletes, updates, and records that arrive late or out of order.

The ingestion layer should record what was received, when, where it came from, and whether it was processed.

Transforms you can run again

Raw data rarely arrives in the shape a model needs. Records need normalizing. Documents need parsing. Metadata has to be extracted. Inconsistent values have to be reconciled. Sensitive fields may need filtering or redaction. Retrieval systems need documents split into chunks, enriched, and prepared for embedding. Classical machine learning needs feature extraction, encoding, aggregation, and a check against the distributions you expect.

I design these as explicit, versioned stages, not a pile of scripts. Each stage has inputs, outputs, and a defined failure. Where it matters I keep source references, the transform version, processing timestamps, and the link between a derived record and its input.

That lineage is how you investigate. If a retrieval result is wrong, you should be able to tell whether the fault is the source document, the parser, the chunking, the metadata, the embedding, or the retrieval configuration. If a transform changes, you should know which downstream artifacts have to be regenerated. If quality gets worse, you should have a practical way back to the previous version. Processing should be repeatable, inspectable, and safe to modify.

Chunking, embeddings, and retrieval, chosen on purpose

Retrieval-augmented generation is often introduced as a short recipe: split documents, embed them, store the vectors, fetch the nearest matches. That can make a convincing prototype. It also fails in ways that are easy to miss. A chunk can separate a statement from the context that qualifies it. A document can cover several topics that one embedding represents badly. Semantic similarity can return text that is related to the question and does not answer it. Facts can sit in tables, headings, footnotes, or structured records that generic extraction handles poorly.

I design the retrieval pipeline around the structure of your information and the questions people actually ask. I look at parsing, chunk size, overlap, semantic segmentation, metadata, embedding models, and index configuration. Depending on the use case, hybrid search, keyword retrieval, metadata filters, reranking, query rewriting, or hierarchical retrieval may earn a place. I also look at how results are assembled into the context the model sees. More context is not automatically better. Extra or irrelevant context raises inference cost, introduces contradictions, and makes it harder for the model to find the evidence that matters.

I want the information required to answer the question, with its meaning intact, and enough provenance to judge whether the answer is supported. Every one of these choices is a hypothesis you can measure, not a setting chosen once during the first implementation.

Evaluation in the pipeline

AI systems are good at sounding plausible, which makes informal testing dangerous. A few successful demos do not establish retrieval quality, factual accuracy, or reliability across the questions users will ask.

I help build an evaluation set and a procedure you can run again, matched to what the application is for. For retrieval that can mean representative questions, the source documents you expect, relevance judgments, and queries the system should decline or handle differently. I measure recall, precision, ranking quality, and whether the supporting evidence comes back. For generated answers I look at factual correctness, faithfulness to the retrieved sources, completeness, citation accuracy, and whatever the application specifically requires. For predictive models I use metrics that fit the task, checks for leakage, calibration, subgroup performance, and a comparison to the baseline you already have.

Not every quality dimension collapses to one automated score. Where a person has to judge, I define the criteria and a repeatable way to apply them. Model-based evaluators can scale the assessment, and their judgments need validation too. The result is a harness you can run as the system changes, so a modification gets compared to a baseline and you can see which behaviors improved and which regressed.

Version the whole chain

An AI pipeline has more moving parts than a typical application deploy. A parser change alters extracted text. A chunking change moves document boundaries. A new embedding model changes the vectors. A revised retrieval strategy returns different evidence for the same question. Application code can stay still while behavior shifts.

I version the pieces that materially affect data quality and model behavior: source snapshots, transform code, chunking configuration, embedding models, index schemas, retrieval parameters, prompt templates, and evaluation sets. I say which artifacts have to stay compatible and which changes require reprocessing or reindexing. Where it helps I use parallel indexes, a staged deploy, a canary, or a controlled cutover so new behavior can be compared with what is running.

Rollback is more than reverting a commit. If a deploy has already rewritten derived data or replaced an index, getting the old behavior back may mean restoring compatible artifacts or rebuilding them from versioned inputs. I design that path before you need it. Experimentation should not cost you reproducibility.

A pipeline that fails where you can see it

Infrastructure monitoring can tell you a server is up. It may not tell you the AI system is working. An ingestion job can report success while processing fewer records than expected. A parser can keep running and extract empty text. A vector index can stay available and hold stale or incomplete data. The application looks healthy while the answers get worse.

I design observability for operational health and for data quality: ingestion success, processing latency, queue depth, retries, error categories, freshness, schema changes, document counts, and how many records were rejected or quarantined. I want visibility into each transform stage, into index updates, and into the relationship between source data and what was derived from it. Where it is useful I watch for shifts in distributions, unexpected changes in retrieval behavior, and quality regressions from a scheduled evaluation.

Alerts should match conditions that need a person. A brief upstream delay may only need a retry. A persistent failure to ingest a critical source needs a notification now. A measured drop in retrieval quality may be a reason to stop a new pipeline version from being promoted. Failures should be visible early enough to investigate before a customer sees them.

Schema drift and sources that change meaning

External systems evolve. API fields get renamed. Column types change. Document templates are revised. A required value becomes optional. Categories appear that were not in the original training data. Some of that fails loudly. The dangerous changes keep the pipeline running while the output gets wrong or incomplete.

I set validation around what enters: expected schemas, required fields, accepted ranges, referential relationships, and quality rules specific to the application. When a source violates them, the pipeline should respond on purpose. Depending on severity it rejects the record, quarantines it, continues with a documented warning, or stops the affected stage.

I also separate a structural change from a semantic one. A field can keep its name and type while meaning something else. A repository can start including drafts next to approved documents. A vendor can change how a value is calculated without changing the response shape. A schema check will miss them. They need someone who understands the domain. The meaning of the information has to survive as the upstream systems change.

Provenance, access, and what the system is allowed to use

Pipelines consolidate information from systems with different owners, confidentiality, and access rules. A retrieval index can mix public documentation with internal customer records. A sync can copy sensitive data into a system with broader access. A deleted document can remain available through a cache or an embedding that was never removed.

I trace how access restrictions and lifecycle rules propagate through the pipeline. I establish provenance, ownership metadata, retention, deletion behavior, and the relationship between a user’s permissions and what can be retrieved. Confidential or regulated information may need extra controls around encryption, isolation, audit, and the use of an external model provider.

Access control belongs in the application and the retrieval infrastructure. If a user is not allowed to see a document, that document should not be handed to the model on the hope that the model will refuse to reveal it. I also design how a correction or a deletion at the source reaches the derived artifacts. Data should stay traceable, restricted to the people allowed to use it, and manageable for its whole life.

Infrastructure matched to the workload

Orchestrators, brokers, streaming platforms, vector databases, feature stores, transform frameworks, and managed AI services all solve real problems. They also get adopted when a simpler thing would do.

I judge a tool against volume, update frequency, latency, failure tolerance, operating cost, and the team that has to run it. A daily document collection may only need a scheduled batch. Continuous updates from an operational database into search or analytics may justify change data capture. A broker makes sense when several consumers need a durable event stream they can process independently. A dedicated vector database is right for some workloads. PostgreSQL with a vector extension is enough for others.

Depending on the environment, the implementation might involve PostgreSQL, Kafka, Redpanda, Debezium, object storage, a workflow orchestrator, cloud data services, or vector search. I start from the workload, the requirements, and the people who will operate it, and I keep the infrastructure only as complicated as that demands.

Improvement as part of delivery

The first production release is not the end of the design. New sources arrive. Users ask questions the original evaluation set never contained. Models improve. Costs change. A retrieval strategy that worked on a small corpus behaves differently as the data grows.

I set up a loop from observed behavior back to a controlled change. Failures and low-quality outputs get collected, categorized, and added to future evaluation. A proposed change says what it is expected to improve and which metrics or cases will show that. It gets compared to the current baseline before it is promoted. Where it is practical, evaluation sits in the deployment workflow, so an unacceptable regression can be flagged or blocked and an improvement ships with evidence.

That record of experiments matters more than another round of prompt tweaks, retrieval tweaks, and embedding changes whose combined effect nobody can explain. You can see the tradeoff, and you can see whether quality moved. A release can still improve one metric and hurt another.

From a demo to something you can operate

Prototypes are supposed to be fast. Manually prepared data, a simplified ingestion script, hardcoded configuration, and a short list of test questions are reasonable when the question is whether the idea works. The mistake is treating those shortcuts as the production architecture.

Moving across means answering different questions. How does new data get in? How do you recover a failed job? What happens when a source document is deleted? Can the index be rebuilt from scratch? How are changes evaluated? Who gets told when the pipeline stops updating? What does it cost to process, store, index, and retrieve at the volume you expect?

I find the gaps between a working proof of concept and a system you can rely on. I say which pieces can stay, which need to be redesigned, and which production capabilities have to exist before the application is trusted. If the prototype is already the right shape, I leave it. Otherwise I add the operational discipline those responsibilities require.

What I will not recommend

I will not recommend a vector database when ordinary search or a structured query would be more reliable. I will not recommend real-time streaming for data that changes once a week. I will not introduce an orchestration framework when a scheduled job and clear failure handling would do. I will not recommend fine-tuning when the problem is poor retrieval, inconsistent source data, or an evaluation that does not exist. I will not present an automated quality score as proof of improvement without saying what the score measures.

Sometimes the useful change is better parsing, a more careful data model, removing irrelevant documents, fixing an ingestion bug, or writing a set of test questions you trust. Often the improvement that matters is making the information you already have more accurate, easier to reach, and possible to measure.

What you leave with

What I hand over depends on the application and the infrastructure you already have. It usually includes:

• An ingestion pipeline, scheduled or event-driven, with defined sources, validation, idempotent processing, failure handling, and monitoring.
• A reproducible transform workflow: versioned parsing, normalization, enrichment, chunking, embedding, or feature processing, with enough lineage to investigate an output and produce it again.
• A documented data architecture: sources, stages, storage, indexing, how data moves, and what it depends on.
• An evaluation harness: representative cases, how they are scored, and quality measures that fit the application.
• A baseline quality report: current performance, known weaknesses, the limits of the evaluation, and where an improvement is most likely to matter.
• Monitoring and failure visibility: operational metrics, data-quality checks, and alerts for ingestion failure, stale data, and serious processing problems.
• Versioning and recovery: how to rebuild derived data, how to handle a change that is not compatible, and how to revert a pipeline update that did not work.
• An engineering handoff: configuration, operating procedures, the architectural decisions, and enough context for your team to maintain and extend the system.

The result should not depend on the person who wrote the original script being around to explain it. Your engineers should be able to operate the pipeline, investigate a failure, reproduce a result, and evaluate a change on their own.

Who I do this with

Organizations that have shown an AI capability and are struggling to make it a product they can depend on. Startups building retrieval-augmented applications. Companies putting AI into internal workflows. Teams adding machine learning to a platform that already exists. Organizations whose data infrastructure was not built for this workload.

You may have useful information spread across operational databases, document repositories, third parties, and archives, and no reliable way to turn it into consistent model inputs. You may already have a working AI application and no evaluation or observability good enough to improve it with confidence. Sometimes the hard part is ingestion and data quality. Sometimes it is retrieval relevance, reproducibility, or the lack of a measurement that means anything. In each case the system’s behavior depends on a pipeline that is not well enough understood or controlled.

When the data infrastructure is dependable, engineers can experiment without losing track of what changed, product teams can compare a release to a baseline, and operators can see a failed ingestion job before a customer does. Leadership can tell a demonstrated improvement from an optimistic claim.

I build the path from ingestion and transformation through indexing, evaluation, and monitoring, so you can see why an answer was produced, judge whether it was correct, and change the system with an explanation you can stand behind. Correctness stays something you measure.