Local models & open projects
AI reading list
Start with a task on your own device: a conversation, audio transcription or a coding agent. Choose the program that runs the model, then the model files that suit your hardware. Published source code and downloadable model weights are different parts of the setup, with their own licenses. These authors help you move from a first run to understanding what the model can do and what it costs to run.
People to read
Open the first material, then continue with the author on X. People appear once, grouped by their main reading use and ordered by handle within each group.
2 of 6 people
No people match that search. Try another name, subject or reading task.
Run on your device
Pick a runtime, load a model and try a concrete task. Compare the speed of generating text with the success of the complete task.
-
Move from a downloaded model to a program on your device. Gerganov created llama.cpp for language models and whisper.cpp for speech transcription; their documentation gives installation choices and shows how model size affects memory needs.
Start with this Code · github.com
Run a language model with llama.cpp
Begin with Quickstart and supported hardware. Choose a model that fits your device; the runtime license and the model license are separate. This is documentation for a project he created and the community maintains.
This work in the author’s profile
Material checked . Authorship source.
-
Connect a local model to a coding agent and decide whether the combination helps your task. Raschka separates the model runtime from the agent client, then compares speed, memory use and tool-call performance in his own tests.
Start with this Web material · magazine.sebastianraschka.com
Read sections 3–5: setup, speed and memory, then tool-use tests. They show why fast text generation and successful agent tasks need separate measurements.
This work in the author’s profile
Published . Material checked .
Choose and adapt a model
Understand the files and descriptions behind a model, then explore ways to reduce memory needs or adapt it to examples.
-
Understand what you are downloading before choosing a reusable model. Wolf coauthored the Transformers library paper: it links tokenizer, model and task head, then explains model cards that record training context and intended use.
Start with this Web material · arxiv.org
Start with section 3 and Figure 2, then section 4 on model sharing. This coauthored 2019 paper explains the design; current installation instructions live in the library documentation.
This work in the author’s profile
Published . Material checked .
-
Separate saving memory during inference from adapting a model. Dettmers explains lower-precision numbers with small examples; the coauthored QLoRA paper shows how a frozen base model can be adapted with a small trainable addition.
Start with this Web material · timdettmers.com
Why fewer bits can change a model’s answers
Begin with the quantization examples, then Mixed precision decomposition. They connect smaller numbers, rounding errors and the separate handling of unusually large values.
This work in the author’s profile
Published . Material checked .
Understand memory and speed
Look inside a small runtime and compare how model architectures carry information. Match performance claims to their test conditions.
-
Inspect the pieces inside a local runtime. Rush and Rachit Singh’s small Rust implementation makes model conversion, mapped weights and decoder settings visible, so you can connect a program’s settings to its memory and computation.
Start with this Code · github.com
Start with “How does it work?”, then the model configurations. This is an educational Rust implementation of Llama 2, building on llama2.c; Rust familiarity helps when inspecting the code.
This work in the author’s profile
Material checked .
-
Ask why memory use grows as a conversation gets longer. Dao’s coauthored Mamba-3 explanation compares a fixed-size state with an expanding attention cache and connects that choice to decoding cost and retrieval quality.
Start with this Web material · tridao.me
Memory, decoding and retrieval in Mamba-3
Start with “The Mamba-3 Model” and the retrieval comparisons. The latency tests use specified server GPUs; use them to understand the mechanism before drawing conclusions about your own hardware.
This work in the author’s profile
Published . Material checked .
A few bearings for your reading
These guides explain the people, language and questions you may meet along the way.
Before you download a local LLM
Separate model from runtime, budget memory beyond the file and plan a repeatable local-model test with original work from five useful authors.
How to evaluate an AI application on your own task
Define a useful success test, trace a published failure and compare fixes through Hamel Husain, Jason Liu, Eugene Yan and Chip Huyen.
How this list is selected
We selected original software, practical tests and coauthored research that answer different questions about running a model. Project creators share authorship with current maintainers; team papers remain credited as collaborative work. Historical papers explain mechanisms, while current project documentation covers setup. Each reading note identifies the starting point and useful prerequisites.