The Bitter Lesson

In machine learning and AI, people often refer to a famous essay: The Bitter Lesson by Richard Sutton. Link

TLDR: One-sentence summary: As data and compute grow, general-purpose learning and search methods consistently outpace systems built on domain-specific human knowledge.

---

The point of this essay: mainly due to Moore’s Law, scaling compute power almost always beats out human-centric approaches to creating machine learning algorithms. You can attempt to hand-craft algorithms that try to capture the complexity of human cognition, but these attempts are:

1. Usually irrelevant in a matter of years, as Moore’s law will make the amount of compute needed to “brute force” the problem they target an infinitessimally cheap amount of compute. Basically compute becomes so cheap that brute-forcing is just as fast/easy/good of a solution.
2. Usually inferior to the eventual ML-driven or brute-force-driven compute-backend solutions that ultimately beat out the handcrafted algorithmic solution. The ML-driven systems backed by scaled compute capture the meta-solution, they capture the method with which we use to think instead of trying to implement how we think.

In otherwords, using learning + search backend by big compute power always eventually beats algorithmic handcrafted machine learning systems, sooner or later. The two modes: learning and search. Learning of course referring: increase amounts of data and compute power creates a better AI system. Search referring to using heavier compute power on a larger scale, traversing larger and deeper data structures in ways previously thought imperformant or wasteful.

Why it’s called the bitter lesson: the methods that defeated the world chess champion, Kasparov, in 1997, were based on massive, deep search. All of the previous art in computer chess leveraged human understanding of the special structure of chess, when a simpler compute + brute-force based approach eventually won, the human-knowledge-based chess researcherrs were very bitter and poor losers.