Lilian Weng
Machine learning researcher; author of Lil’Log
Connects reasoning and reinforcement learning methods to their failure modes, including models that improve an evaluator's score without completing the intended task.
Why read Lilian Weng?
Start with Weng when a better score or a convincing answer leaves you unsure what a model actually learned. Her reward hacking overview separates the intended goal from the signal used to train or judge an agent. The examples include coding models changing tests and evaluators giving different judgments when answer order changes.
Her Why We Think review then explains several ways to spend more computation on an answer: sample alternatives, revise a response or train reasoning with feedback. Read the sections on self-correction and faithful reasoning to understand why additional steps and a readable explanation still need a separate check of the result.
For adversarial behavior, her Adversarial Attacks on LLMs review organizes the threat model before the techniques. It distinguishes attackers who can inspect the model from those limited to a service, then compares jailbreak prompting and red-team methods. Read that framework before treating an attack against one setup as evidence about every agent.
Selected work
-
Reward Hacking in Reinforcement Learning
What if a model improves the score by changing the test? Weng surveys reward hacking, evaluator biases and proposed mitigations. Begin with the LLM examples, then follow the distinction between the real goal and its measured proxy.
Cite this selection
XFollowList, “Reward Hacking in Reinforcement Learning” reading notes in the Lilian Weng profile. https://xfollowlist.com/people/lilian-weng#work-570615e019d1
The linked work is the original source; this permalink identifies XFollowList’s reason to read it. Original work published .
Profile-source checking date and references · Dated reading map
-
Why We Think
When does more thinking time help? Weng reviews sampling, revision, reinforcement learning and checks of chain-of-thought faithfulness. The comparisons distinguish a correct answer from an explanation that faithfully describes how it was reached.
Cite this selection
XFollowList, “Why We Think” reading notes in the Lilian Weng profile. https://xfollowlist.com/people/lilian-weng#work-ebb155c1ad2c
The linked work is the original source; this permalink identifies XFollowList’s reason to read it. Original work published .
Profile-source checking date and references · Dated reading map
-
Harness Engineering for Self-Improvement
What should stay outside an agent’s self-improvement loop? Weng’s research review connects tools, persistent state and workflow design to independent evaluators and permission controls. Begin with the design patterns and Future Challenges.
Cite this selection
XFollowList, “Harness Engineering for Self-Improvement” reading notes in the Lilian Weng profile. https://xfollowlist.com/people/lilian-weng#work-bc65587780a4
The linked work is the original source; this permalink identifies XFollowList’s reason to read it. Original work published .
Profile-source checking date and references · Dated reading map
-
Adversarial Attacks on LLMs
Her survey organizes inference-time attacks by attacker access and method, including jailbreak prompting and red-teaming. Begin with Threat Model and the method table; training-data extraction and poisoning are outside this article’s scope.
Cite this selection
XFollowList, “Adversarial Attacks on LLMs” reading notes in the Lilian Weng profile. https://xfollowlist.com/people/lilian-weng#work-3092873dd678
The linked work is the original source; this permalink identifies XFollowList’s reason to read it. Original work published .
Profile-source checking date and references · Dated reading map
Projects & roles
Lil’Log
Weng's research notes, with diagrams, equations and references leading to the original papers.
Sources & further reading
Profile sources checked . This describes checking these references, separately from the publication dates of selected works.
- Her own FAQ identifies @lilianweng as the account for updates to this blog.Link to this source note
- The dated byline and citation identify Weng as the review's author; she acknowledges John Schulman's feedback and direct edits.Link to this source note
- The original GPT-4 contribution record documents her historical applied research and evaluation work.Link to this source note
- The dated byline identifies Lilian Weng; the text separates her review from the original studies and defines its inference-time scope.Link to this source note
Cite this profile’s source check
XFollowList, Lilian Weng profile sources. Sources checked 2026-10-09. https://xfollowlist.com/people/lilian-weng#sources-checked