The model is the last thing you should worry about

Almost every ML project I'm brought into starts with someone asking which model the team plans to use — GPT-4 versus an open-source alternative, which embedding model, which fine-tuning approach. The question is always asked with genuine technical seriousness, and it's almost always the wrong first question. Not because the model doesn't matter — it does, eventually — but by the time a model decision is actually consequential, five or six earlier decisions have usually already determined whether the project can succeed at all. None of them involve picking a model.

The model conversation is appealing because models are comparable: benchmarks, published research, leaderboards. Choosing one feels like making a defensible decision — point to numbers, cite papers, reference deployments from companies with bigger engineering teams. The earlier work — problem framing, data quality, evaluation design, failure mode analysis — is murkier. There's no leaderboard for whether your problem statement is correct, no published benchmark for label quality. Those decisions are judgment calls rather than comparisons, which makes them harder to justify and easier to skip past on the way to the part that feels solvable.

Across systems I've designed — a local-first RAG for manufacturing, a federated query engine for enterprise sales, a model-merging architecture for financial analytics, an alignment pipeline for regulated output — the pattern determining outcome is consistent, and it sits upstream of model selection every time.

Problem framing. The most consequential decision in any ML project is whether the problem being solved is the right one. A model trained perfectly on the wrong problem doesn't produce partial value — it produces confident, useless outputs that erode trust faster than a system that didn't exist at all.

Data quality and fitness. Not a data engineering problem — an ML problem. A model trained on clean, well-labelled data representing the real deployment distribution will outperform a model trained on a larger, noisier dataset almost every time. The time spent on data quality isn't preparation for the real work. It is the real work.

Baseline definition. Before any model is trained, you need to know what you're trying to beat — not a published benchmark, the baseline for your specific problem, your specific environment, your specific data. Usually a simple rule, a heuristic, or human performance on the task. In several projects, a well-designed baseline outperformed the first three model iterations. That's not a failure. It's the system working correctly.

Evaluation design. The metric you choose is what the model will optimise for. Choose the wrong one and you'll build a system excellent at something that doesn't matter. The metric has to connect to the business outcome, be measurable in production, and be defined precisely enough that two people looking at the same output agree on whether it passed. Where those conditions can't be met, the problem definition isn't finished — regardless of how ready the team feels to start training.

Failure mode analysis. Every ML system fails in specific ways. The question is whether those failure modes are understood before deployment or discovered after. For a factory floor system, a confidently wrong answer is not a retrieval miss — it's a safety incident waiting to happen.

None of these five categories can be fixed by a better model, because none of them are model problems. The fix isn't better judgment applied at the model-selection stage. It's a different sequence — start with the things that determine whether the project is viable, and get to the model only when it's actually the thing that needs deciding.

Further reading: Sculley, D. et al. (2015). Hidden Technical Debt in Machine Learning Systems. NeurIPS. · Sambasivan, N. et al. (2021). "Everyone wants to do the model work, not the data work." ACM CHI. · Ribeiro, M.T. et al. (2020). Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. ACL. · Huyen, C. (2022). Designing Machine Learning Systems. O'Reilly.