A shortlist in 72 hours, contributing inside two weeks, and a 30-day guarantee if the fit is wrong.
72-hour shortlist · 30-day replacement guarantee · Month-to-month
An LLM prototype is deceptively easy to build. A weekend gets you something that answers questions about your documents impressively well, and everyone in the room is convinced.
Then it meets real usage. Retrieval returns the wrong chunk and the model confidently invents an answer. Users phrase things nobody anticipated. Latency is fine until three people use it at once. The API bill for the month is four figures higher than forecast. And nobody can say whether last week’s prompt change made things better or worse, because there’s no way to measure it.
The gap between prototype and product is where nearly every LLM project stalls, and it’s almost entirely about evaluation, retrieval quality, and cost discipline rather than model choice.
How do they know the system improved? Without measurement, prompt engineering is guesswork with confidence.
Chunking strategy, embedding choice, hybrid search, reranking. Most RAG failures are retrieval failures.
Caching, model routing, prompt compression, and hard budget limits. The clearest senior signal.
Streaming, parallel calls, and knowing which steps can be precomputed.
What the system does when the model returns nonsense, times out, or refuses. Production systems need answers.
Knowing when prompting is enough — which is most of the time — and when it genuinely isn't.
The distinctive risk with LLM systems is that the ground moves under them. Your code doesn’t change and the behaviour does.
A provider ships a new model version, or deprecates the one you pinned to with a few months’ notice. Output that reliably returned clean JSON starts wrapping itself in a markdown fence, and your parser throws on a fraction of requests. Or a prompt that worked by luck — leaning on a quirk of one model — stops working on its successor. Without a regression suite you find out from users, and you can’t tell whether the cause was your last deploy or theirs.
Agent workflows add a failure mode with a direct price. A loop where a tool call fails, the model retries, and the retry produces the same failure will happily run until it hits a limit — and each iteration is billed. One malformed input can cost more in an afternoon than the feature saves in a month.
The mitigations are unglamorous: pin model versions explicitly, keep a regression set you re-run on every prompt and model change, cap agent iterations hard, and set spend alerts. Engineers who have run this in production do all four without being asked.
The spend is also unusually hard to forecast. Token cost scales with how people actually use the feature rather than with your projections, and a retrieval step that quietly doubles context length doubles the bill without changing anything anyone can see.
Mid-level (2–4 years). Implements against a defined architecture. Comfortable with the major APIs and frameworks. Needs direction on evaluation and system design.
Senior (4–6 years). Designs the system end to end. Owns retrieval strategy, evaluation, and cost. Can explain trade-offs to non-technical stakeholders.
Lead (6+ years). Owns AI direction, makes build-versus-buy calls, and gives an honest assessment of what LLMs will and won’t solve.
Bands run shorter than in mature stacks — the field is young, and four years of relevant production experience is genuinely senior.
RAG over internal knowledge. Documents, tickets, wikis. The most common request and the most underestimated.
LLM features in an existing product. Summarisation, extraction, classification, drafting.
Agent workflows. Multi-step systems with tool use. Powerful and considerably harder to make reliable — AI agent developers is where that difficulty is discussed properly.
Prototype to production. A working demo that needs evaluation, monitoring, and cost control before real users touch it.
Evaluation infrastructure. Building the measurement layer a team skipped. Often the highest-value engagement. Related: AI integration and automation.
Evaluation harnesses. Unglamorous, and the thing that turns prompt work from opinion into measurement.
Week one. They build the evaluation set before touching prompts. Thirty to fifty real questions with known-good answers, drawn from your actual users if possible. Expect questions about where documents live, how often they change, whether anything is access-controlled, and what a wrong answer costs you — a wrong answer in an internal search tool and in a customer-facing assistant carry very different risk.
Weeks two to four. They report a baseline score and improve it deliberately, changing one variable at a time. Expect specifics: retrieval accuracy went from 61% to 78% by fixing chunking and adding a reranker, and here’s what still fails. They should also instrument cost per request early, before it matters.
The warning sign is prompt tinkering with no measurement. If week three is another round of prompt edits and the only evidence is a handful of examples that look better, the project has no way to know if it’s improving and no way to detect when a model update breaks it.
Ask for the failure cases in week one. A useful engineer collects the outputs that were wrong, groups them by cause, and comes back with a short list — retrieval missed it, the prompt was ambiguous, the source document was out of date. That list is the roadmap. Teams without one are left tuning prompts on instinct.
Choosing a framework before understanding the problem. LangChain and LlamaIndex are useful and they also hide the retrieval and prompting decisions that determine whether the system works. Teams end up debugging the abstraction rather than their own pipeline. For a first system, direct API calls make every decision visible; add a framework once you know what you’re standardising.
Measuring the model instead of the retrieval. When answers are wrong, the instinct is to try a bigger model. Most of the time the model was fine and it was handed the wrong context. Evaluate retrieval separately — did the right chunk come back at all — before touching generation. It’s cheaper and it’s usually where the problem is.
Building an agent when a workflow would do. Agents that decide their own steps are impressive and hard to make reliable, because the failure surface is combinatorial. If the sequence of steps is actually known in advance, write it as a pipeline with explicit LLM calls. It’s more predictable, cheaper, and far easier to debug at 2am.
Chasing the newest model instead of fixing retrieval. When answers are wrong the model is usually not the constraint — the context it received was incomplete, stale, or badly chunked. Swapping in a larger model produces fluent mistakes instead of clumsy ones, costs more per call, and delays the work that would actually help. Look at what was retrieved before looking at what generated the answer.
Rates depend on seniority, production experience, and engagement length. Tell us the role and we’ll give you a firm number.
Tell us the role and we will send a shortlist within 72 hours.
Tell Us the RoleTell us the role. Three to five profiles within 72 hours.