ENNPRO Get in touch
← All engineering roles

Data Engineer, AI Systems

Give model teams training and evaluation data they can trace back to its source.

Apply for this role

About the role

A client should be able to ask which records trained a model and get a precise answer. You’ll build the ingestion and processing jobs that make that possible, from the first source extract to the dataset used in a release.

Model engineers will depend on your datasets for training, evaluation and release decisions. You’ll work with the people who produce source records to understand what an identifier means, when a correction becomes visible and what a deleted record should do to downstream tables.

Working at ENNPRO

ENNPRO is an applied machine learning consultancy. We work alongside client engineers and the people who use their systems. Depending on the engagement, a client might need a service they can operate, a benchmark they can reproduce or enough evidence to decide whether to release a model. Their team should be able to use the result after handover.

You’ll spend time with the people who enter or export the records as well as with the engineers who consume them. A transaction ID might identify an order in one feed and a single line item in another. The person responsible for invoicing can often explain that difference more quickly than a week of profiling. Part of your job is to find the right person, verify the explanation against records and turn it into a durable data contract.

Your immediate partners are model engineers building training sets and analysts checking business totals. They may need different views of the same event. You’ll help them decide which transformation can be shared, where their definitions legitimately differ and how to publish both without losing the route back to the source.

Responsibilities

  • Design Python and SQL transformations that carry stable identifiers, event times and ingestion times from the source extract through the published dataset.
  • Make recurring loads and backfills safe to retry. Test late updates, repeated deliveries and partial writes against the same contract used in the scheduled pipeline.
  • Version evaluation snapshots so an engineer can reconstruct the inputs to a past result. Document the joins and exclusions that change which examples appear.
  • Give operators checks they can act on. Distinguish a late source feed from an empty feed and tell the model owner which training runs or reports are affected.
  • Design a correction policy with the source owner. Decide how a reversal, deletion or late amendment appears in history and what should happen to datasets that were published before it arrived.
  • Estimate the cost of a new data dependency. Measure the size and cadence of an extract, the cost of rebuilding a partition and the time a consumer can tolerate before a feed is considered stale.
  • Review changes with downstream users. Supply a sample of records that gained, lost or changed a label so the model owner can tell whether a new distribution reflects a business event or a processing fault.

Day to day

Most investigations begin with a few keys and a date. You’ll use SQL to follow those keys through joins and Python to exercise the transformation on a small fixture. Put an expectation around the rows that disappear as well as the ones that survive. A valid output schema tells the team very little if an inner join silently removed a whole supplier.

For production work, expect to discuss partition boundaries, schema changes, observation times and retry behavior with whoever operates the client’s platform. Leave a short explanation of which job to restart, which output is safe to replace and which consumers need to hear about a delay.

Projects and first 90 days

A representative assignment

A supplier reissues a month of records after changing the way it encodes cancellations. The model team needs the correction in its next training set and still needs to reproduce last month’s evaluation.

Trace a small sample across both versions, identify the business key, and rehearse the load twice. The handoff includes a reconciliation of changed rows, a preserved evaluation snapshot and an operator procedure for the next correction.

After the rehearsal, put the proposed correction in front of the analyst who reconciles the supplier account. Ask them to follow a cancellation across both versions. Their ability to explain the difference is a useful acceptance check before the revised dataset reaches model training.

How your first engagement could unfold

First month. Rebuild one published dataset with a teammate and reconcile its row count and totals against its source. Make a diagram of the joins and a list of the fields whose meanings need clarification. Agree with a downstream user on one test that would have caught a previous break.

Second month. Take responsibility for a recurring load. Add a fixture for a late correction, make the write safe to repeat and exercise recovery after a partial failure. Have a colleague run the recovery from your notes while you watch where the instructions need work.

Third month. Publish a versioned dataset for one training or evaluation run. Deliver a short change report, rehearse a backfill on a bounded partition, and show the model owner how to recover the exact records behind a disputed example.

Qualifications

Core experience

  • You’ve maintained a pipeline beyond its first successful run and can describe how you recovered from a failed or incomplete load.
  • You can write and review SQL involving deduplication, point-in-time joins and aggregation. You test the meaning of the output as well as the schema.
  • You can walk a model engineer through the provenance of an example and work with a source owner to resolve an ambiguous field.
  • You can reason about storage and compute tradeoffs when data stops fitting into memory, and can identify which operation dominates the cost of a pipeline.
  • You can read someone else’s transformation, propose a small reviewable change and supply fixtures covering normal records, corrections and missing fields.

Useful additional experience

  • Experience with warehouse or lakehouse systems, partitioned file formats or incremental processing on a scheduler.
  • Familiarity with point-in-time feature generation, training and validation splits, or systems that publish repeatable model datasets.
  • Experience negotiating a source contract across teams, especially when the source producer must keep supporting older consumers.

How to apply

Tell us about a data correction or backfill you owned. Include how you measured its effect on downstream users and checked that a retry was safe.

Describe the input, the intended output and the failure mode in ordinary language. A small table of invented rows is a good way to explain a difficult join or retry. Include the check that convinced the downstream user the repaired dataset was ready to use.

You can use an anonymized example for confidential work. The application form includes space for your account of the project, an optional résumé and an optional link to a work sample. It currently lets you prepare and review a draft in your browser.

Equal opportunity and accessibility

ENNPRO welcomes applicants of all backgrounds. We consider candidates on the skills and experience relevant to the role, including experience gained through different educational and career paths. If you need an accessible version of these materials or an adjustment to the application process, write to contact@ennpro.pro.

Start application