About the role
When two candidate models score within a fraction of a point, the average rarely tells an operator which to trust. You’ll investigate where they disagree and design experiments that help client teams make release decisions.
Your work starts with the decision a client needs to make about a model. You’ll establish what the comparison can answer, where the labels are trustworthy and which failures carry the greatest cost. Part of the job is knowing when an evaluation is too weak to support a release.
Working at ENNPRO
ENNPRO is an applied machine learning consultancy. We work alongside client engineers and the people who use their systems. Depending on the engagement, a client might need a service they can operate, a benchmark they can reproduce or enough evidence to decide whether to release a model. Their team should be able to use the result after handover.
A client may give you a table of scores and ask which model to release on Monday. Your first conversation will often be with the people who act on the predictions. Ask which mistakes they can correct, which ones reach a customer and how frequently the expensive cases occur. Those answers shape the evaluation population and the point at which you have enough evidence to make a recommendation.
You’ll own the experimental design and implementation alongside engineers who know the serving system and reviewers who understand the labels. As a senior member of the engagement, you’ll review proposed experiments from other engineers, help them narrow an ambiguous question and make it straightforward for someone else to challenge your interpretation.
Responsibilities
- Translate an operator’s failure report into a reproducible example. Record the input, model version and expected behavior so another engineer can repeat the observation.
- Choose a baseline and a sampling plan before comparing candidates. Keep related examples together when splitting data and reserve an untouched set for the final comparison.
- Inspect disagreements by class, source and operating condition. Quantify uncertainty where the samples support it and expose thin slices to the reviewer.
- Write up the release decision in operational terms. Show what improved, where a regression remains and which additional observation could change the recommendation.
- Set the unit of analysis before scoring a candidate. Repeated documents, multiple events from one customer and near-duplicate examples can make a large test set much less informative than its row count suggests.
- Design the annotation exercise with domain reviewers. Supply examples of borderline cases, measure where reviewers disagree and decide which disagreements require a clearer label definition before model comparison can continue.
- Keep an experiment record that ties each figure to its dataset, model version and evaluation code. Capture negative results and failed comparisons so a later team can understand why a seemingly promising idea was set aside.
Day to day
You’ll use notebooks for exploration and small, versioned programs for measurements that affect a release. Make a hand-checked fixture before running the full evaluation. A swapped class order or an unexpected rule for missing predictions can produce an entirely plausible chart. Review a few individual decisions alongside the aggregate result before circulating it.
Part of the work is editorial. A decision note needs to say which population you measured, what uncertainty remains and what action follows from the evidence. Explain how a reader can reproduce the result. Be prepared to defend a recommendation to collect fifty better examples before spending another week tuning the model.
Projects and first 90 days
A representative assignment
Two classifiers agree on most documents. Their aggregate scores are close, but the team suspects that one sends a costly document type to the wrong queue. Build a paired comparison on the same held-out examples and have a domain reviewer adjudicate the disputed labels.
The deliverable is a rerunnable experiment and a short decision note. A product owner should be able to see which cases drive the result and decide whether to release, restrict the rollout or collect more labels.
The team may also want a threshold at which a person takes over. Estimate how many reviews each proposed threshold creates on an ordinary day. Ask the operator whether that queue fits into a shift before treating an improvement in average model performance as a workable release plan.
How your first engagement could unfold
First month. Reproduce a result the team already relies on and inspect examples from both sides of a disputed decision. Write down the sampling unit, the source of the reference labels and the operating cost that makes this question worth answering. Have the domain owner correct your interpretation.
Second month. Own a paired baseline comparison from sample selection through error review. Run a small adjudication session, identify one meaningful slice and write a clear account of any data too sparse to support a conclusion. Put the experiment where another engineer can run it.
Third month. Lead a release discussion using the resulting evidence. Recommend a course of action, agree with the system owner on what to monitor and document a condition that would trigger a fresh comparison. Hand a colleague the evaluation and have them reproduce the key table independently.
Qualifications
Core experience
- You’ve implemented experiments in Python and can describe a leakage check or comparison that changed your reading of a result.
- You can explain how class imbalance, repeated observations and the choice of denominator affect an evaluation.
- You can review an experiment with engineers and communicate an inconclusive finding to the person accountable for the release.
- You have selected metrics for a real decision and can explain how a sampling or labeling decision changed what those metrics meant.
- You can work with versioned datasets and model outputs, write tests for evaluation code and review another researcher’s analysis closely enough to catch a misleading comparison.
Useful additional experience
- Experience with paired statistical comparisons, calibration, confidence intervals or resampling methods for dependent observations.
- Practical work with reference-label quality, adjudication, sampling under a review budget or evaluation of rare classes.
- Experience mentoring engineers who are moving from exploratory analysis to experiments that others will maintain.
How to apply
Share an evaluation where the headline metric missed something important. Explain how you discovered it and what the team decided to do.
Walk us through the original question, the comparison you ran and the consequence of the finding. An inconclusive experiment is a good choice if you can explain what the evidence ruled out and how you designed the next test. A private industry project and an independently reproduced paper are both suitable examples.
You can use an anonymized example for confidential work. The application form includes space for your account of the project, an optional résumé and an optional link to a work sample. It currently lets you prepare and review a draft in your browser.
Equal opportunity and accessibility
ENNPRO welcomes applicants of all backgrounds. We consider candidates on the skills and experience relevant to the role, including experience gained through different educational and career paths. If you need an accessible version of these materials or an adjustment to the application process, write to contact@ennpro.pro.
Start application