Applied AI engineering
AI reading list
A model response becomes useful through the application around it: the information retrieved, the structure of the result and the checks applied before and after a release. Follow a small extraction or retrieval example, define failures on your own data, then plan the logs and controls that make a running system understandable.
People to read
Open the first material, then continue with the author on X. People appear once, grouped by their main reading use and ordered by handle within each group.
2 of 7 people
No people match that search. Try another name, subject or reading task.
Prepare data and context
Turn source material into inspectable records, retrieve the right information and keep a reusable dataset for comparisons.
-
Build a reusable test set for a retrieval application. Liu and Andrei Fajardo describe a dataset with questions, reference answers and the underlying documents, so competing changes can be evaluated on the same task. Start with the dataset design before choosing a metric.
Start with this Web material · llamaindex.ai
Coauthored with Andrei Fajardo. Read the dataset, generation and evaluation workflow; keep the provenance of synthetic and human examples. The API snippets describe the 2023 LlamaIndex version.
This work in the author’s profile
Published . Material checked .
-
Find out whether the application retrieves the information a user needs. Liu starts with a retrieval baseline, then tests metadata and search methods before changing the final answer. His second essay shows how source details and facets can guide further exploration.
Start with this Web material · jxnl.co
Systematically Improving Your RAG
Read the retrieval baseline, metadata and feedback sections. Separate whether the right document was found from whether the final answer was useful. Client numbers are examples to interpret, not targets for your system.
This work in the author’s profile
Published . Material checked .
-
Turn unstructured content into records you can inspect. Willison shows a complete path from an extraction schema to saved responses, JSON and SQLite, using his LLM tool. It is a small, visible workflow for testing data extraction before building a larger application.
Start with this Web material · simonwillison.net
Structured data extraction from unstructured content using LLM schemas
Follow the people-extraction example, reusable templates and export to SQLite. The post explains output-format failures; also compare the extracted fields with the original source. Model support describes the 2025 release.
This work in the author’s profile
Published . Material checked .
Measure product quality
Use application traces and labeled examples to define failures, check automated judges and repeat experiments after a change.
-
Turn labeled examples into an experiment you can repeat after a prompt or model change. Yan separates evaluation criteria, holds back test data and joins the evaluation harness to configuration experiments. His AlignEval app makes the labeling and calibration steps concrete.
Start with this Web material · eugeneyan.com
Product Evals in Three Simple Steps
Read all three steps. Pay attention to one evaluator per criterion, held-out test examples and the uncertainty around a measured defect rate. Continue to AlignEval for an inspectable implementation.
This work in the author’s profile
Published . Material checked .
-
Find the failures that matter before adding a quality dashboard. Husain’s case study combines scoped assertions with readable application traces and human review. His follow-up shows how to calibrate a model judge against the domain expert’s decisions.
Start with this Web material · hamel.dev
Start with the Lucy case study, then the three evaluation layers. Notice which failures code can check, what a reviewer needs to see and why an automated judge needs its own evaluation.
This work in the author’s profile
Published . Material checked .
Operate and maintain the application
Plan latency, cost, failure handling and observability, then keep verification in the release and maintenance loop.
-
Keep verification in the development and maintenance loop. Osmani connects the final output and the path used to produce it with regression tests, versioned context and production observation. This route helps a team decide what must be checked before accepting an agent-built change.
Start with this Web material · addyosmani.com
Read Verification and How each phase actually changes. This is Osmani’s commentary on a whitepaper coauthored with Shubham Saboo and Sokratis Kartakis. Use its release-checking questions; cost diagrams are illustrative.
This work in the author’s profile
Published . Material checked .
-
Plan the components around a model API as the application grows. Huyen connects retrieval, guardrails, model routing and caching to latency, cost and failure handling. Her observability section shows what to record so a bad response can be traced to the step that produced it.
Start with this Web material · huyenchip.com
Building A Generative AI Platform
Follow the architecture diagrams, gateway and caching tradeoffs, then logs, metrics and traces. The components are a planning guide; the 2024 prices and sample API calls need current documentation before implementation.
This work in the author’s profile
Published . Material checked .
A few bearings for your reading
These guides explain the people, language and questions you may meet along the way.
How to evaluate an AI application on your own task
Define a useful success test, trace a published failure and compare fixes through Hamel Husain, Jason Liu, Eugene Yan and Chip Huyen.
An AI agent demo checklist: tools, state and success
Assess an AI agent demo through its tools, state, access and checked outcome. Read a published database-error recovery and plan the next useful test.
Before you download a local LLM
Separate model from runtime, budget memory beyond the file and plan a repeatable local-model test with original work from five useful authors.
How this list is selected
We selected people through their own engineering guides, project walkthroughs and coauthored work. Each first read answers a specific application question. Historical code and prices keep their original dates; product evaluations need the reader’s data and criteria. Recent X activity was not assessed for this selection.