Evaluating Long-Context Question & Answer Systems
Evaluation metrics, how to build eval datasets, eval methodology, and a review of several benchmarks.
Tech Media 164 stories
AI Engineer 2025 – Improving RecSys & Search with LLM techniques
Recsys & search are converging with LLMs via semantic IDs, data augmentation, and unified foundation models.
Exceptional Leadership: Some Qualities, Behaviors, and Styles
What makes a good leader? What do good leaders do? And commando, soldier, and police leadership.
Building News Agents for Daily News Recaps with MCP, Q, and tmux
Learning to automate simple agentic workflows with Amazon Q CLI, Anthropic MCP, and tmux.
Why We Think
Special thanks to John Schulman for a lot of super valuable feedback and direct edits on this post. Test time compute (Graves et al. 2016, Ling,…
An LLM-as-Judge Won't Save The Product—Fixing Your Process Will
Applying the scientific method, building via eval-driven development, and monitoring AI output.
Frequently Asked Questions about My Writing Process
How I started, why I write, who I write for, how I write, and more.
Moving To Substack
I’m freezing this blog and starting to post on my Substack instead. The authoring experience is much more convenient for me there. Please follow me there,…
NVIDIA GTC 2025 – Building LLM-Powered Applications
Chip Huyen and I share what we've learned, best practices, and insights at NVIDIA GTC 2025.
Improving Recommendation Systems & Search in the Age of LLMs
Model architectures, data generation, training paradigms, and unified frameworks inspired by LLMs.
Common pitfalls when building generative AI applications
As we’re still in the early days of building applications with foundation models, it’s normal to make mistakes. This is a quick note with examples of…
Building AI Reading Club: Features & Behind the Scenes
Exploring how an AI-powered reading experience could look like.
Agents
Intelligent agents are considered by many to be the ultimate goal of AI. The classic book by Stuart Russell and Peter Norvig, Artificial Intelligence: A Modern…
2024 Year in Review
A peaceful year of steady progress on my craft and health.
nbsanity – Share Notebooks as Polished Web Pages in Seconds
nbsanity - Share Notebooks as Polished Web Pages in Seconds
Seemingly Paradoxical Rules of Writing
With regard to writing, there are many rules and also no rules at all.
Building an Audience Through Technical Writing: Strategies and Mistakes
Building an Audience Through Technical Writing: Strategies and Mistakes
Reward Hacking in Reinforcement Learning
Reward hacking occurs when a reinforcement learning (RL) agent exploits flaws or ambiguities in the reward function to achieve high rewards, without genuinely learning or completing…
How to Run a Weekly Paper Club (and Build a Learning Community)
Benefits of running a weekly paper club, how to start one, and how to read and facilitate papers.
My Minimal MacBook Pro Setup Guide
Setting up my new MacBook Pro from scratch
39 Lessons on Building ML Systems, Scaling, Execution, and More
ML systems, production & scaling, execution & collaboration, building for users, conference etiquette.
Using LLM-as-a-Judge For Evaluation: A Complete Guide
Using LLM-as-a-Judge For Evaluation: A Complete Guide
AlignEval: Building an App to Make Evals Easy, Fun, and Automated
Look at and label your data, build and evaluate your LLM-evaluator, and optimize it against your labels.
Weights & Biases LLM-Evaluator Hackathon – Hackathon Judge
Being a human judge at the Weights & Biases LLM-as-a-Judge Hackathon