I previously wrote about our AI Engineering Skills Map, with the highest level skills being (i) Building and deploying AI applications, (ii) Software engineering fundamentals, (iii) Using coding agents, and (iv) Shaping the build. In this article, I will flesh out the first of them.
Being skilled at building and deploying AI applications means knowing:
- LLM foundations
- Grounding models with data
- Building agentic systems
- Evaluation-driven development
- Operating in production
- Machine learning foundations
This map of skills was formed by analyzing a large number of job postings, structured expert interviews, and survey responses.
The key difference between AI applications and non-AI software is that the former’s output is less predictable. You don’t know in advance what an LLM will output, or what predictions a supervised learning algorithm will make. Because of this uncertainty, building AI systems is a much more iterative process than building traditional software — it is harder to plan the process in advance. Skilled AI engineers repeatedly build a piece of software, examine it, and decide what to try next, taking a sequence of steps that are highly influenced by the intermediate results. Being able to skillfully decide what to do next allows you to create reliable software systems based on unreliable AI components. This requires knowing:
LLM foundations. Understanding how large language models tokenize input and generate output allows you to understand when to count on them and when they may fail. It also allows you to understand when to use a multimodal model, how to make tradeoffs on what to include in the context window, and reason about cache hits, knowledge cutoff, reasoning effort level, sampling parameters, and when to use special features such as tool calling. Understanding these foundations helps you choose the right model or mix of models and apply specialized techniques when needed, such as fine-tuning or self-hosting models.
Grounding models with data. LLMs require good input context to produce useful outputs. RAG using vector search was an early attempt to give LLMs relevant context, but the set of techniques for grounding models with data has grown significantly. For example, you will have to decide what to include in a prompt vs. what to let an LLM retrieve on demand using tools, and which representation fits the data and search queries: a vector index, a knowledge graph, or a semantic layer over structured data (such as customer records). You’ll also turn documents (text, PDFs, HTML, images) into LLM-ready inputs and engineer pipelines to keep data clean and fresh. When you understand the menu of techniques available to get data, you are better able to give your LLM relevant context.
Building agentic systems. Agentic systems range from workflows that execute a predefined sequence of LLM calls to ones based on an agent harness that lets an LLM repeatedly decide its own next step. You’ll have to choose the architecture — what steps to chain, what to parallelize, when to use code and when to use an LLM — and engineer the workflow or harness, with fallbacks. When designing the agent loop, you will also decide what tools the model can call (including MCP, CLI and sandbox execution environments), what memory architecture to use, how to manage context over long sessions, and when a task needs multi-agent orchestration instead of a single-agent architecture. You’ll also want to turn promising prototypes into reliable, safe and secure agents for production; this requires understanding guardrails, adversarial inputs, and identifying and working around key risks (such as data exfiltration), and governance.
Agentic workflows are evolving rapidly, and you will also benefit from understanding any cutting-edge techniques relevant to your application area, such as voice agents, computer-use agents, or generative UI.
Evaluation-driven development. In my experience, the most important trait that distinguishes someone great at building AI systems is whether you can drive a disciplined evals/error analysis loop to drive development. This allows you to repeatedly focus your effort on directions that are more likely to be fruitful. I’ve found this to be a tricky skill to master, because the right approach varies significantly by project and even according to the stage of the project.
Building good evals is a deep technical skill. You might look at a system’s traces and outputs, carry out exploratory data analysis, and combine that with product and business insight to decide what to measure. You should also understand the menu of options for evals, such as when to use deterministic (code-based) evaluations, when to use an LLM-as-a-judge, and when to have a human in the loop, and how to evaluate your evals so as to keep evolving them. These evaluations then feed into an iterative process that drives further development, and makes progress systematic rather than random.
Operating in production. Operating AI software is different from traditional software because of its unpredictability, cost, and latency. First, you should know how to build observability mechanisms to understand the system’s performance on real usage. You’ll track performance, detect drift, and respond quickly to model failures and security incidents such as adversarial prompt injections. Putting in place regression testing and CI/CD requires more statistical evaluations than traditional software, and the testing effort should be calibrated relative to the risk of a mistake. Additionally, it’s important to know how to select the right mix of techniques — such as model choice optimization, distillation and fine-tuning, and agentic workflow simplifications — to optimize for cost and latency, especially if your application reaches many users.
Machine learning foundations. Modern LLMs are built using machine learning techniques including supervised learning and reinforcement learning. Every engineer I know that’s good at building with LLMs also understands machine learning and deep learning at some depth. Additionally, many applications still require knowing how to use machine learning – either a model someone else trained or one you train yourself. This requires knowing the popular machine learning and deep learning models and tradeoffs in accuracy, training speed, inference speed, and so on, and understanding how to engineer the data needed to train and evaluate these models. The machine learning concepts of bias/variance, error analysis, and engineering your data — all of which are core mental frameworks for navigating how to work with systems with uncertain output — also remain key to making a wide range of decisions in AI system development.
There is a lot to learn to become good at building and deploying AI systems. This is a field with significant technical depth. But every bit you learn will help you become better at AI Engineering and build more exciting applications. A strong complement to these skills is software engineering. I will write about this in a future post.
