Home > Ajabbi Research > Library > Subscriptions > Data Engineering Weekly
Home > Handbook >
Last Updated
3/02/2026
Modeling Semantics: How Data Models and Ontologies Connect to Build Your Semantic Foundations
By: Juha Korpela
Modern Data 101 (Medium): 22/01/2026
Independent Consultant | Data Modelling Enthusiast | Founder @ Helsinki Data Week.
This piece is a community contribution from Juha Korpela, Independent Consultant and Founder of Helsinki Data Week, a community-first data conference. With deep expertise in information architecture, data products, and modern operating models, Juha has spent his career helping organisations truly understand what their data means and how to use that semantic clarity to build better systems.
Formerly Chief Product Officer at Ellie Technologies and now the voice behind the “Common Sense Data” Substack, Juha is also a speaker, trainer, and advisor shaping the resurgence of conceptual modeling in the industry. We’re thrilled to feature his unique insights on Modern Data 101!
We actively collaborate with data experts to bring the best resources to a 15,000+ strong community of data leaders and practitioners. If you have something to share, reach out!
Share your ideas and work: community@moderndata101.com
Note: Opinions expressed in contributions are not our own and are only curated by us for broader access and discussion. All submissions are vetted for quality & relevance. We keep it information-first and do not support any promotions, paid or otherwise!
Let’s Dive In!
Knowledge Management Provides Context for AI
Knowledge Management and Information Architecture have had a rocket ride to the top of the data world’s consciousness due to Generative AI. The ability to organize, store, and serve structured semantics as context to various agents and chatbots is widely recognized as a winning ingredient in the GenAI race, reducing hallucinations and improving accuracy.
Terms like taxonomies, ontologies, and knowledge graphs are being thrown around as if just been invented, but veterans of the trade know better: there’s nothing new under the sun.
Knowledge Management and the Library Sciences, from which these subjects were born, are well-known disciplines, and the theory behind concepts like the Semantic Web is solid. It’s merely the utilization of these that has now changed with GenAI.
Data Modeling Foundations Return
But when it comes to organizing, storing, and serving semantics, there have always been two schools of thought, usually with very little cross-pollination between them. The other viewpoints outside ontologies and knowledge graphs have been coming from the data modeling world.
Traditionally, data modeling has had different levels of abstraction to cover different needs at different levels of detail. Conceptual, Logical, and Physical modeling has been a well-recognized three-level layout for data modeling activities (you can check my views on these three levels on my Substack).
But sadly, at some point in the Big Data craze of yesteryear, many data experts reduced data modeling to the Physical level only, focusing almost exclusively on the technical structures of data storage.
Where Semantics Was Compromised
By forgoing Conceptual modelling to a large extent, data experts had let go of a very practical method for doing exactly the same thing that is now required from taxonomies, ontologies, and knowledge graphs: describing structured semantics.
At the core of both ontologies and conceptual data models are things: real-life entities that exist in the real business, irrespective of the systems we have built. You might call these things “entities” or “objects” or “nodes” or whatever you like,
…but they are what you need to understand in order to describe
(to a human or an agent) what goes on in your business.
Think of “Customer”, “Order”, “Product”, “Delivery”, and so on. These are what you have data about, no matter how the data is technically stored in database tables or files.
In addition to the list of things, to fully understand the business context, we need relationships between the things. How do the things in our business interact with each other? Think “Customer makes an Order”, “Product is added to Delivery”, and so on.
Ontology vs. Conceptual Model
An ontology is, in simple terms, a list of things (and their definitions) with a list of the relationships between them. In the Knowledge Management world, this would be formalized according to, say, RDF standards.
A conceptual model is, in simple terms, also a list of things and their relationships. Data modelers traditionally produce an Entity-Relationship Diagram out of it, with a list of entity definitions (a Glossary) attached.
Now here’s the important thing to understand, regardless of which world you are coming from:
the semantical content you capture with both approaches is exactly the same!
Merely the method of capturing, organizing, and storing that information is different.
For me personally, the method of conceptual modeling feels natural, as I’ve done data modeling for around 15 years now. I know what questions I need to ask people (or what documents to read) to capture information about the entities and their relationships, I know how to draw the diagram, I know how to create the glossary, and I know what tools I can use to help.
For someone coming from a semantic web background, building formalized ontologies according to the RDF standard feels natural, with all the methods and tools that come with it.
We’re both still working on semantics: in effect, we’re capturing the exact same ontology, thus storing information about business context to be used later.
Technical Implementation of Semantics
For us data modelers, the utilization of these models has traditionally focused on the technical implementation of data solutions, and we’ve thus followed a path from Conceptual to Logical to Physical. That is, if we have done conceptual modeling at all!
But especially now, in the age of context-hungry AI, we have to realize we’ve been sitting on a semantics gold mine: conceptual data modeling is an excellent method for figuring out what the entities and relationships should be.
The diagram titled “How Data Models & Ontologies Connect to Build Semantic Foundations” shows “Understanding the business” leading into “Conceptual Modeling,” which branches into two paths: “Solution Design” and “Semantic Discovery”. The ontology also integrates inputs from Standard Ontologies and AI Agents.
How Data Models and Ontologies Connect to Build Semantic Foundations | Source: Author
Why is this important? Because the most valuable semantical information is that which is unique to the organization, and those semantics are the hardest to capture.
While AI tools can be used to find semantical concepts from unstructured data and various knowledge bases, a lot of this information is tacit knowledge in the business experts’ heads. Conceptual modeling is a known-good method for getting that tacit knowledge out.
Data Modeling as Semantic Discovery
I envision a world where we build the semantic foundation of an organization with a set of tools at our disposal:
A pyramid diagram titled “Building Organisational Semantics” illustrating the layers of a Semantic Foundation. The foundation acts as a context provider for both agents and humans. It features three tiers: Industry Standards at the base for common structures, Conceptual Modeling in the center to unearth unique organisational knowledge, and AI Agents at the apex to find details that enhance core semantics.
Technical Implementation of the semantic Foundation with Industry Standards, Conceptual Modeling, and AI Agents | Insights from the Author
We use industry standards and existing knowledge bases to cover the basic structures that are common to most organizations within an industry
We use conceptual modeling methods as a surgical knife to cut through tacit knowledge and unearth & document the valuable, unique semantics of the organization
We use AI agents as “semantic helpers” to trawl through tons of documentation and find details to add around the strong core that has been formed
This semantic foundation will then act as the context provider for all your agents and chatbots, but also for humans! Context is king in today’s world. By looking at data modeling as not only a technical design method, but as a semantic discovery method, we enable a powerful tool for building this context.
Home > Ajabbi Research > Library > Subscriptions > IT Revolution
Home > Handbook >
Last Updated
2/02/2026
The Andon Cord
By: John Willis
IT Revolution: 15/10/2023
John Willis has worked in the IT management industry for more than 35 years and is a prolific author, including "Deming's Journey to Profound Knowledge" and "The DevOps Handbook." He is researching DevOps, DevSecOps, IT risk, modern governance, and audit compliance. Previously he was an Evangelist at Docker Inc., VP of Solutions for Socketplane (sold to Docker) and Enstratius (sold to Dell), and VP of Training & Services at Opscode where he formalized the training, evangelism, and professional services functions at the firm. Willis also founded Gulf Breeze Software, an award winning IBM business partner, which specializes in deploying Tivoli technology for the enterprise. Willis has authored six IBM Redbooks for IBM on enterprise systems management and was the founder and chief architect at Chain Bridge Systems.
The origin of the word “Andon” in Japanese comes from the use of traditional lighting equipment using a fire burning lamp made out of paper and bamboo. This “Andon” idea was later translated for use in manufacturing in Japan. The “Andon” became used as a signal to highlight an anomaly (i.e., a flashing light). This signal would be used to amplify potential defects in quality. When a defect was suspected, a sign board would light up signaling the specific workstation having a problem. The signal event would also indicate that the system was stopped due to the defect and was waiting for the problem to be resolved.
The process of stopping a system when a defect was suspected originates back to the original Toyota System Corporation to something called Jidoka. The idea behind Jidoka is that by stopping the system you get an immediate opportunity for improvement, or find a root cause, as opposed to letting the defect move further down the line and be unresolved.
The Jidoka concept was pioneered by the original Toyota founder, Sakichi Toyoda. Sakichi Toyoda is known as the father of the Japanese industrial revolution and also the founder of the original Toyota Systems Corporation (before they manufactured automobiles).
Sakichi Toyoda invented the automatic power loom in 1924 where the loom would automatically stop if a sewing needle broke. Before his invention, the loom would continue even when the needle was broken. When this occurred, the downstream process would produce runs in the fabric stitches of the final product.
Toyota Production Systems
Approximately 25 years later, a gentleman named Taiichi Ohno, who is considered the father of “Toyota Production Systems (TPS)”, architected a leadership and technology model incorporating some of the original Jidoka ideas along with a lot of other interesting innovations. His ideas produced an unprecedented run of quality and manufacturing success for over 40 years.
To this day, what was done at TPS from around 1950 to 1990 still can’t be repeated in automobile manufacturing, and many have tried. An abundance of material has been written about TPS, also coined as Lean over the years, and many have tried to copy TPS from that body of knowledge.
Toyota Kata
Mike Rother in his Toyota Kata book points out that of the many who try to emulate Toyota, most miss the invisible side of what they were doing.
The key, as Rother suggests, is that it wasn’t the tools that made Toyota great, it was the culture and specific behavior associated behind those tools. One of the best examples of this was the use of the Andon Cord at Toyota.
The Andon Cord was a manifestation of the original Jidoka principle. Toyota implemented the Andon Cord as a physical rope that followed the assembly line and could be pulled to stop the manufacturing line at any time. Most western culture analysis of this type of Cord might assume this was implemented as a safety cut off switch. At Toyota, it was a tool to instill autonomic behavior patterns. This is what Rother calls Kata.
Furthermore, this wasn’t an ask permission to stop the line, the pull actually stopped the line. As the story goes, anyone could pull the Andon Cord anytime.
Sounds mad doesn’t it? Salvador Dali eloquently says “The only difference between a madman and me is that I’m not mad.”
At TPS, the Cord was pulled often. The mechanics of the Andon Cord were if the Cord was pulled, not only did the line stop, but an “andon” would light up on a signal board to indicate the workstation that was having the issue.
Beyond the mechanics, the culture behind the Andon Cord was a lot more interesting. The first thing that would happen when the Andon Cord was pulled is that a team leader would immediately “go-see” the issue by visiting the workstation. This was unconditional.
Toyota lived by this “Show Me” culture where in a western culture organization a team lead might put down the coffee on his or her desk and call the workstation to find out what was happening. This is a key point here. The “go-see” removes any preconceived notions or potential bias related to the problem. The “go-see” process is fact based.
High Velocity Edge
In Dr. Steven Spear’s The High Velocity Edge, he describes a horrifying story of missed opportunities leading up to the 2003 NASA Columbia space shuttle disaster.
The short version of the story is that the thermal protection system on the left wing was damaged just after launch but didn’t become an issue until reentry 19 days later. After the disaster, an investigation board charged with reviewing the accident found there were at least eight attempted signaling events to notify the crew requesting that they “go-see” the damage. The first request happened as early as the fourth day of the mission. These ‘signals’ were not addressed because there was a cognitive bias at NASA regarding this particular issue. This kind of damage had happened on previous missions and they had always been successful. Diane Vaughan, a Columbia University sociologist, coined this as “normalization of deviance”. This is where an organization tends to accept risky anomalies as normal through repeated success.
This form of a blind spot is also referred to as outcome bias. Outcome bias is where people observe successful outcomes as results as opposed to addressing individual problems at face value. Sidney Dekker in The Field Guide to Understanding Human Error states this kind of the anti Murphy’s Law in that, what can go wrong usually goes right. Organizations get desensitized due to more positive outcomes than exposed failures.
Dr. Spear suggests some counterfactuals that are intellectually interesting, or at least proposes learning opportunities for the reader. He suggests the Columbia crew could have been notified and possibly done an extravehicular activity (EVA) inspection to look at the damage.
Instead, NASA ignored repeated attempts, andon pulls if you will, to “go-see” the factual conditions. NASA unfortunately was mired in culture bias’ and preconceived notions of what constituted a real (“go-see”) opportunity. Maybe if the crew had been notified of just one of the eight known signals, they might have been able to access the damage, and possibly NASA could have done a recovery mission.
The meta-point here is that, ironically, NASA didn’t operate like scientists, unlike Toyota’s relentless “go-see” Kata.
Safety Culture
A second important cultural aspect of the “Andon Cord” process at Toyota was that when the team leader arrived at the workstation, he or she thanked the team member who pulled the Cord. This was another unconditional behavior reinforcement. The repetition of this simple gesture formed a learning pattern of what we call today “Safety Culture”. The team member did not, or would never be, in a position of feeling fear or retribution for stopping the line.
Quite the contrary, the team member was always rewarded verbally. What Toyota was saying to the team member was “We thank you and your CEO thanks you. You have saved a customer from receiving a defect.” Moreover, they were saying, “You have given us (Toyota) an opportunity to learn and for that we really thank you.”
No defect was too small or even if the Cord was mistakenly pulled, the response would never be negative.
In Toyota Kata, Rother says that if you look up the word “failure” in the dictionary, it never implies that “failure” is a bad word. In a classic tayloristic western culture, we tend to shy way from failure at all cost. We penalize failure and we typically never embrace it. At TPS, failure was embraced and rewarded. As Rother also discusses in his book, that at Toyota, at their core, believed failure created learning opportunities.
Improvement Kata
The third important behavior reinforcement of the “Andon Cord” process was that the issue was a priority. In fact, the second thing that would happen when the team leader would arrive at the workstation after the thank you, is that he or she would ask “How can I help you?”. An important aspect of this is the “you”.
The incident was not going to be some paper report or bureaucratic long tail process. The problem was going to be immediately addressed and in fact, the team member who pulled the Cord was the one who was going to fix it.
Another overarching aspect to the culture at Toyota was that everyone had a mentor. The mentor/mentee relationship was one where it was very important that the mentee understand the problem. Again in western culture organizations, even if a worker felt empowered to stop the line, they might not be inclined to do so if they thought the problem would not get fixed. Even worse, have the issue get bogged down in useless paperwork and never ending meetings. The team leader at Toyota would then proceed to ask a series of questions trying to drill down to a point where the team member would understand the issue.
Another key point in this process was that even if the team leader knew a better answer, Toyota principally felt that a solution from the team member was a better outcome. In other words, if the problem is not understood at the line level, that is the team member/mentee, then the problem was not solved.
This is what I would call learning at scale.
Plan Do Change Act (PDCA)
Furthermore, the process of solving the issues was controlled by a practice described by Dr. Edwards Deming called Plan Do Change Act (PDCA). At its core, PDCA is basic scientific method.
Rother refers to this set of tools as Improvement Kata and Coaching Kata.
The Improvement Kata was an iterative mentor/mentee PDCA loop whereas the Coaching Kata was a sort of Socratic dialog between the mentor and mentee. Again, an important point here is the team member (mentee) would always solve the problem. It was of high importance that the team member always solve the problem. Most importantly, it was critical for the team member to understand how the problem was solved. Otherwise, there would not be in inherent learning and therefore, no real improvement.
Solving problems at Toyota was not the goal, understanding how to solve the problem was. Solving problems in an Improvement Kata mode also creates a second order effect. By solving one problem, sometimes other second order problems are exposed.
Alcoa, the aluminum company, set out to have a zero on the job injury policy in 1987. Paul O’Neal, the new CEO at the time, created a policy that if anyone at Alcoa was hurt on the job, he needed to be notified within 24 hours. This was not slogan based safety. This was an organizational behavior modification to create a line of sight understanding of why injuries occur at Alcoa.
The results were unprecedented. However, the interesting side effect of this was that through this rigorous process of forced understanding, they exposed what they called other “pockets of ignorance”. Their attention to detail for solving their first order problem, safety, surfaced other second order process improvements not necessarily related to safety.
At Toyota, it was important to instill an Improvement Kata based on a PDCA loop. Plan (P) a countermeasure, implement the countermeasure (D), check or study the results (C), and act on the results either it’s fixed or start the next countermeasure (A). Imagine all those masked problems that that never get noticed in large complex IT infrastructures due to a possible irregularity of the first order issues and non “go-see” approaches.
There is a great story in the Toyota Kata book where at one point a particular Toyota plant notices that the average Andon Cord pulls in a shift goes down from 1000 to 700. As Rother describes most western culture organizations would break out the champagne for such an occasion, not Toyota. The CEO called an all-hands meeting to address the “problem”.
Notice I said problem.
The CEO then goes on to describe that “we” must have one of two problems here. One, we are getting lazy and letting more defects get through the system or two, if that is not the case, then we are not operating at our full potential. He was telling everyone that if they were staffed to handle 1000 pulls per shift, then they should be pulling a 1000 Cords per shift.
The cornerstone behind this kind of thinking is that Toyota had a vision of having 1×1 flow. This is where there is no inventory build up and work flows freely at every point of the workflow. Even though 1×1 was sort of a “true north” at Toyota, the CEO was reminding all of them that their Kata should always be pointing towards that direction, what Rother calls the Vision.
In plain words, the CEO is saying more pulls equals more learning which means more improvement that gets us towards our vision. 1X1 was a means to an end to say “If we can produce cars faster, cheaper and with higher quality, we win.”
Amazon and the Customer Service Andon Cord
The Andon Cord has become a metaphor for some modern day Web Scale organizations as well.
Jeff Bezos, the CEO of Amazon, described in a 2013 letter to the Amazon’s shareholders a practice he called the Customer Service Andon Cord. This was an established practice of metaphorically pulling an Andon Cord when they noticed a customer was overpaying or had overpaid for a service.
Amazon would heuristically scan their systems looking for these kinds of potential customer service mismatches. These were considered defects at Amazon because they had a vision of being an organization that was always customer centric. They would automatically refund a customer, without the customer even asking, if the service delivery was suboptimal.
I have had this happen to me on a few occasions watching a movie on Amazon Prime, where the next day I received an email telling me they refunded my movie rental cost due to poor quality. They would also pull the Andon Cord where they found areas where a customer could be saving money. We see this all the time where Amazon reduces their Cloud Services price even though their service is considered far superior to their closest competitor.
This is a form of Kata in practice that drives Amazon towards their stated vision.
In that same shareholders letter, Bezos starts off with a line as follows: “Our energy at Amazon comes from the desire to impress customers rather than the zeal to best competitors”. It’s no mistake that two of the top 12 books on Jeff Bezos’s recommended reading list are The Goal by Eliyahu Goldratt and Lean Thinking by James Womack. In fact, The Goal is one of the three books he has all of his top executives read.
The Chaos Cord
Another example of an Andon Cord metaphor used in Web Scale businesses is at Netflix.
Netflix has an interesting way of exercising their Andon Cord, although they don’t actually call it an Andon Cord. Along the same line as Toyota, at Netflix failures are good things. Netflix has built in their own automated form of Jidoka.
As described earlier, Jidoka is a practice of stopping a process if it breaks. In the earlier example, the process we described was done via the physical Andon Cord and was a manual process. At Toyota, there were also automated forms of Jidoka practiced. Another famous engineer named Shigeo Shingo, who worked with Taiichi Ohno, is credited with the idea of pre-automation. This is a form of Jidoka that is automatic.
At Netflix, they actually inject this kind of Jidoko into their systems on purpose by intentionally trying to break systems in production. They have developed what is now famously called Chaos Monkey. Chaos Monkey is a process that randomly kills live running production servers. This behavior is known by everyone who works at Netflix. It’s part of their culture. There are no surprises about this practice. Developers plan and Poka-Yoke their code and systems accordingly.
Poka-Yoke is another term that comes from Shigeo Shingo at TPS. Poka-Yoke means mistake-proofing.
I was told by Adrian Cockcroft, one of the primary architects behind Netflix’s IT infrastructure, that not knowing about the Chaos Monkey mode coming into a job interview at Netflix was pretty much an immediate no-hire decision.
Imagine that Netflix’s Kata is so obsessed with failure they create their own failures on purpose. As you can imagine, Netflix is a learning organization and every one of these failures is treated as a science experiment.
They might not literally practice PDCA, but either one of two things happens when a server is killed by their Chaos monkey. One, they learn that there were dormant defects in the process and fix them, or two, the injected failure was corrected automatically. The best case was where the injected failure caused no customer disruption. Their Improvement Kata was always moving in that direction.
Like any good operating Kata based organization, Netflix has been practicing their Kata for quite a few years now. You don’t get to Chaos Monkey overnight. Much of their Kata has been based on continual learning improvements.
One interesting method or Poka-Yoke, if you will, is something they do called Circuit Breaker Pattern. Circuit Breaker Pattern comes from a book called Release-It by Mike Nygard.
These are software delivery patterns where the software code is designed very much like a circuit breaker in your home. If one service dies it isolates or is bounded to only fail the things it controls and not create cascading service outages. Think about a fuse in your home. If you inadvertently load up to much power in one area of your house you only wind up losing power in an isolated section.
Netflix does a similar implementation for their software services. Their application design is such that one thing breaking should never create cascading failures (like a overloaded circuit/fuse combo in your house).
The main point here is that implementing an Andon Cord in an organization is not something you do overnight. It takes a continuous improvement roadmap to get there and must have behavior reinforcement built into the process. It takes a fierce commitment and practice of improvement (Improvement Kata) and an equally skilled leadership coaching approach (Coaching Kata).
If you want to investigate the concept of Kata more deeply, I highly recommend reading Mike Rother’s Toyota Kata.
Earlier this week, I attended the APAC Cloud Technical Series: On Board from Google. It was 10 hours over 2 days of talks and code workshops from Google staff, who were mainly based in Singapore.
It was excellent and worth the time. I signed up for more sessions planned later in the year. Google Weeklies is a regular in-depth talk available both live and as an archive.
I initially registered to participate in the Gen AI Academy APAC Edition, which would have been fun, but then discovered an age restriction. I'm too ancient. 😊
IaC
This will help me build Pipi Engines to build, deploy, and manage infrastructure-as-code (IaC) in the cloud.
A dedicated agent engine has been created for each cloud platform. They have yet to be differentiated.
Apple Engine (ale)
AWS Engine (aws)
AZURE Engine (azu)
Digital Ocean Engine (dgo)
Google Cloud Engine (ggc)
IBM Engine (ibm)
Meta Engine (met)
Oracle Engine (ora)
(More will be added later; all are welcome)
Agents
Pipi 9 is a type of world-model AI, not an LLM. Google is offering a platform for LLM-based generative AI agents. My thought is to connect Pipi 9 to these external agents via open protocols, leveraging the strengths of both.
MCP
A2A
Agent Card
ADK
More Pipi engines
MCP Engine (mcp)
Agent card
Here are some initial notes about the Agent Card protocol, part of A2A. I will start building from there.
Pipi is an agent built from hundreds of other kinds of deeply nested agents, and is capable of learning, evolving and replicating. So does this mean that Pipi needs its own Agent Card? 😀
Mike is the inventor and architect of Pipi and the founder of Ajabbi.
From A2A Protocol Documentation
A2A revolves around several key concepts. For detailed explanations, please refer to the Key Concepts guide.
A2A Client: An application or agent that initiates requests to an A2A Server on behalf of a user or another system.
A2A Server (Remote Agent): An agent or agentic system that exposes an A2A-compliant endpoint, processing tasks and providing responses.
Agent Card: A JSON metadata document published by an A2A Server, describing its identity, capabilities, skills, service endpoint, and authentication requirements.
Message: A communication turn between a client and a remote agent, having a role ("user" or "agent") and containing one or more Parts.
Task: The fundamental unit of work managed by A2A, identified by a unique ID. Tasks are stateful and progress through a defined lifecycle.
Part: The smallest unit of content within a Message or Artifact. Parts can contain text, file references, or structured data.
Artifact: An output (e.g., a document, image, structured data) generated by the agent as a result of a task, composed of Parts.
Streaming: Real-time, incremental updates for tasks (status changes, artifact chunks) delivered via protocol-specific streaming mechanisms.
Push Notifications: Asynchronous task updates delivered via server-initiated HTTP POST requests to a client-provided webhook URL, for long-running or disconnected scenarios.
Context: An optional, server-generated identifier to logically group related tasks and messages.
Extension: A mechanism for agents to provide additional functionality or data beyond the core A2A specification.
- A2A Protocol Documentation
Agent Discovery in A2A
To collaborate using the Agent2Agent (A2A) protocol, AI agents need to first find each other and understand their capabilities. A2A standardizes agent self-descriptions through the Agent Card. However, discovery methods for these Agent Cards vary by environment and requirements. The Agent Card defines what an agent offers. Various strategies exist for a client agent to discover these cards. The choice of strategy depends on the deployment environment and security requirements.
The Role of the Agent Card
The Agent Card is a JSON document that serves as a digital "business card" for an A2A Server (the remote agent). It is crucial for agent discovery and interaction. The key information included in an Agent Card is as follows:
Identity: Includes name, description, and provider information.
Service Endpoint: Specifies the url for the A2A service.
A2A Capabilities: Lists supported features such as streaming or pushNotifications.
Authentication: Details the required schemes (e.g., "Bearer", "OAuth2").
Skills: Describes the agent's tasks using AgentSkill objects, including id, name, description, inputModes, outputModes, and examples.
Client agents use the Agent Card to determine an agent's suitability, structure requests, and ensure secure communication.
Sample Agent Card
{
"protocolVersions": ["1.0"],
"name": "GeoSpatial Route Planner Agent",
"description": "Provides advanced route planning, traffic analysis, and custom map generation services. This agent can calculate optimal routes, estimate travel times considering real-time traffic, and create personalized maps with points of interest.",
"description": "Calculates the optimal driving route between two or more locations, taking into account real-time traffic conditions, road closures, and user preferences (e.g., avoid tolls, prefer highways).",
"description": "Creates custom map images or interactive map views based on user-defined points of interest, routes, and style preferences. Can overlay data layers.",
"As far as I remember, the main focus of messaging is on exchange
protocols—a whole separate field of algorithmic analysis of interacting
processes. Hoare wrote a treatise on this back in the last century, "Interacting Sequential Processes"—I think that's what it's called." - Alex Shkotin
Communicating Sequential Processes by Tony Hoare, ACM
Repository
Home > Ajabbi Research > Library >
Home > Handbook >
Last Updated
31/01/2026
Tony Hoare introduced Communicating Sequential Processes (CSP)
By:
Stanford: Copied 31/01/2026
Sir Charles Antony Richard Hoare (/hɔːr/ HOR; born 11 January 1934), also
known as C. A. R. Hoare, is a British computer scientist who has made
foundational contributions to programming languages, algorithms, operating
systems, formal verification, and concurrent computing.[3] His work earned
him the 1980 ACM Turing Award, usually regarded as the highest distinction
in computer science.
Tony Hoare, winner of the Association for Computing Machinery's A.M. Turing Award, discusses the origin of his model of "Communicating Sequential Processes" and the central importance of keeping processes from directly accessing the state information of other processes. This clip taken from an interview conducted with Hoare by Cliff Jones for the ACM on November 24. 2015. Video of the full interview is available as part of Hoare’s ACM profile at https://amturing.acm.org/award_winners/hoare_4622167.cfm
On Design
"There are two ways of constructing a software design: One way is to make
it so simple that there are obviously no deficiencies, and the other way
is to make it so complicated that there are no obvious deficiencies." -
Tony Hoare
Introduction
Tony Hoare introduced Communicating Sequential Processes (CSP) in 1978 as a
language to describe interactions between concurrent processes.
Historically, software advancement has mainly relied upon improvements in
hardware that create enable faster CPUs and larger memory. Hoare recognized
that a machine with hardware that is 10 times as fast running a code that
consumes 10 times more resources is not an improvement.
While concurrency has many advantages over traditional sequential
programming, it has failed to gain a popular audience because of its
erroneous nature. With CSP, Hoare introduced a precise theory that can
mathematically guarantee programs to be free of the common problems of
concurrency. In his book, Learning CSP (the third most quoted book in
Computer Science), Hoare uses calculus to show that it is possible to work
with deadlocks and nondeterminism as if they were terminal events in
ordinary processes. By reducing the errors of CSP, Hoare has enabled
computer scientists to fully exploit the capacity of CPUs.
What is concurrency?
The ordinary task of doing your laundry illustrates the basics of
concurrency. There are two ways to finish two loads of laundry:
Put 1st load in washing machine → put 1st load in dryer → fold 1st load →
put the 2nd load into the washing machine → repeat process.
Put 1st load in washing machine → put 1st load in dryer → put 2nd load in
washing machine → etc…
Clearly, the second method is quicker and better utilizes resources. If the
first load is in the dryer, the washing machine isn’t being used and should
be used by the second load. This is the basics of concurrency.
Historical Information
The earliest ideas for concurrent processing arose naturally in the 1960’s
because of scarce resources. At that time, processing power was expensive,
and it was wasteful for a processor to have to wait while it communicated
with slow peripheral equipment or human (Hoare, Learning CSP).
Ex: The task of performing a simple addition problem shows how concurrency
can greatly improve computational efficiency. Adding numbers requires three
parts:
Ask user for input numbers
Performing calculation
Printing the answer
Problems of concurrency and Hoare’s solution
While concurrency offers great potential for faster computation, it is not
without fault. Unlike parallelism, where n tasks run on n processors, in
concurrency, n tasks run on 1 processor. Because the amount of resources
doesn’t increase, programs that need to use the same resource must
wait.
The amount of time user waits increases linearly
The amount of storage space needed increases with number of jobs
Difficulty verifying program correctness
The difficulty of verifying program correctness has been the primary
hindrance in the field. Programmers have shied away from parallel
programming because errors that arise in this method of programming are
notoriously difficult to track down. Programs are often prone to errors that
not obvious and surface under arbitrary and unrepeatable situations. The
complex nature of concurrency leaves the programmer in doubt. Programmers
often resort to exhaustive search tactics to find errors:
Testing the code rigorously to sort out obvious concurrency issues and
hoping that all problems have been resolved
Completely follow design patterns and guidelines for concurrent
programming. This method is limited in its applicability.
OR...
Hoare’s approach CSP uses mathematical deductions to prove that a program
is error free. Through the application of CSP, programmers no longer search
for bugs in written programs, but rather can write programs that are
logically guaranteed to be correct.
General terms in CSP
Alphabet- set of events which are considered relevant for a particular
object. The alphabet of a process is enclosed within curly brackets. a. ex:
vending machine- alphabet {coin, chocolate} b. not valid: {coin, car} Trace:
sequence of symbols recording the events in which the process has engaged in
within a given time. The trace is enclosed within angle brackets. c. Vending
machine:
Nondeterminism
In a concurrent program, two or more processes compete for the same
resource. The resolution of this dilemma is not always deterministic. The
unpredictable arrival order of messages creates a nondeterministic state
(2).
Ex: A change machine can give change for a dollar in many different ways:
four quarters, ten dimes, 100 pennies. The type of change given is not
dependent on the type of change machine, but rather by some arbitrary or
nondeterministic fashion.
Ex2: A passenger waiting for the bus to take him to London. The A, B, D,
and F lines all go to London. The passenger can take any f these lines.
Which line he gets on is only dependent upon which bus arrives first.
Why nondeterminism? Programmers use nondeterminism to exclude the events
that don’t effect the outcome as a technique to simplify the problem. In the
case of the change machine, only the sum, not the combination of coins
matters. By reducing the number of variables, nondeterminism helps maintain
a high level of abstraction when describing complicated systems.
Problems with nondeterminism Although nondeterminism can greatly reduce the
complexity of problems, they also introduce their other issues. In a
deterministic program, the answer will always be the same given the same
inputs. In a nondeterministic program, the same inputs can yield different
answers on different cycles or machines. This characteristic makes it
difficult to check whether the program works.
CSP and nondeterminism
CSP introduces the notation Π to signify a process which “behaves either
like P or like Q, where the selection between them is arbitrarily, without
the knowledge of the external environment (1).”
Ex: During lunch, you can choose between an apple or an orange. The choice
can be mathematically expressed as: orange Π apple, where the choice between
the two is based on a unaccounted external factor.
Algebraic laws governing nondeterministic choices are simple.
Idempotenence: A choice between P and P is empty
P Π Q = P
Symmetry:P Π Q= Q Π P
The order does not influence the choice
Associative:
P Π (Q Π R) = (P Π Q) Π R The choice
between three options can be divided into two successive single
choices.
Distributive: x → (P Π Q) = (x → P) Π (x → Q)
Going from a defined path x to a choice between path P and Q, is the same
as choosing between the options of going from path x to P or from path x
to Q.
Fairness
In some theories, nondeterminism is obliged to be fair, in the sense that
an event that infinitely often may happen eventually must happen (though
there is no limit to how long it may be delayed). In Hoare’s theory, the
concept of fairness doesn’t exist:
“Because we observe only finite traces of the behaviour of a process, if
an event can be postponed indefinitely, we can never tell whether it is
going to happen or not. If we want to insist that the event shall happen
eventually, we must state that there is a number n such that every trace
longer than n contains that event. Then the process must be designed
explicitly to satisfy this constraint. For example, in the process P0
defined below, then event a must always occur within n steps of its
previous occurrence.”
-Learning CSP
If fairness is required for the program, it must be considered and
accounted for separately.
Shared Resources
Laws for reasoning about sequential processes derives from the fact that
each variable is updated by one process (learning CSP). If storage is
shared, only one process can change the variable. The potential for data
corruption makes common variables and communication amongst processes
difficult to implement.
Deadlock
Deadlock is the permanent blocking of a set of processes. It is a common
problem in concurrency and arises from the conflicting needs of processes
for similar resources or when communicating with each other.
Example of deadlock: Intersection of cars(3). The shared resources can be
thought of as the lanes. Each car shares the four lanes. The process is the
car. Deadlock occurs when each process holds one resource and requests the
other. While one car can decide to switch lanes, no car can agree on the
proper action to take. As in traffic, deadlock in computer science slows or
completely halts a program.
Classical illustration of deadlock
There is a group of five philosophers who do nothing but eat and think all
day. The philosophers sit around a round table with a bowl of noodles in the
middle. As philosophers get paid less than computer scientists, they can
afford only five single chopsticks, which are placed on each side of the
philosopher. To eat the noodle, the philosopher needs both chopsticks. The
philosopher first picks up the chopstick from his left, and then his right
if it is not being used.
Deadlock arrives when all philosophers want to eat at exactly the same
time. All philosophers pick up the chopstick to their left. However, none of
the philosophers can eat because another philosopher is currently using the
other chopstick.
Events that lead to Deadlock
There are several combinations of events can cause deadlock
Mutual exclusion: Only one process can use a resource at a
time
Hold- and- wait: A process holds onto its resource until the next
resource its needs becomes available
No preemption: No process can be forced to give up its
resources
Circular wait: closed chain of processes where each process holds
the resource the next process needs to function.
While these situations can lead to deadlock, there are precautions
programmers can take to prevent their occurrence.
Mutual exclusion: Restrict the way in which resources can be
requested.
Hold- and- wait: Require all processes to provide information
about the resources they will need in advance. Use algorithms to insure
that all resources are available to the process before it attempts to
acquire them.
Circular wait: Establish a priority system that requires processes
to request resources and process them in that order, such that a higher
priority process will always have access to the resource first. This
solution can lead to starvation.
Eliminating the possibility of a deadlock is better than dealing the
deadlock during execution. However there will may arise arrive unique
combination of situations that lead to deadlock. There are methods of
resolving a deadlock when it’s detected, but these solutions are not
efficient and resolve in lost data (4).
Preempt the resource from a process. The preempt process can be resumed
at a later time.
Return to a point where the process did not need the acquired resource
causing the deadlock.
Systematic killing of jobs until deadlock is resolved.
CSP and deadlock solution
In order to prevent data corruption, Hoare purposed the concept of a
critical area. Processes cross the critical area to gain access to the
shared data. Before entry to the critical area, all other processes must
verify and update the value of the shared variable. Upon exit, the processes
must again verify that all processes have the same value.
Another technique to maintain data integrity is through the use of mutual
exclusion semaphore or a mutex. A mutex is a specific subclass of a
semaphore that only allows one process to access the variable at once. A
semaphore is a restricted access variable that serves as the classic
solution to preventing race hazards in concurrency. Other processes
attempting to access the mutex are blocked and must wait until the current
process releases the mutex. When the mutex is released, only one of the
waiting processes will gain access to the variable, and all others continue
to wait.
In the early 1970s, Hoare developed a concept known as a monitor based on
the concept of the mutex. According to a tutorial on CSP in the Java
programming language written by IBM:
“A monitor is a body of code whose access is guarded by a mutex. Any
process wishing to execute this code must acquire the associated mutex at
the top of the code block and release it at the bottom. Because only one
thread can own a mutex at a given time, this effectively ensures that only
the owing thread can execute a monitor block of code.”
Monitors can help prevent data corruption and deadlocks (5)
Possible Solutions to Philosopher Problem
A physical representation of the solution to the deadlock problem can be
visualized as the footman. The behavior of the footman allows him to sit
only four philosophers at the table simultaneously.
The metaphor of the dining philosophers was thought by the well known
computer scientist, Edsger Dijkstra. Carel S. Scholten discovered the
footman solution.
Infinite Overtaking
This problem arises when priority is assigned to programs. Some program
always takes precedence at the expense of other programs that are delayed
forever.
Ex: You are waiting to be seated at a restaurant. You are next in line, and
just about to be shown your table when a famous actor walks in. The
restaurant, mindful of good publicity, seats the famous actor first. When
the next table becomes available, the waiter turns to you, but then sees a
famous singer walk in. Again, the waiter seats the singer before you. The
weighting of resources leaves you at a disadvantage. If this cycle
continues, you could be delayed forever, or at least for an unacceptable
period of time.
Overtaking Solution
The task of deciding how to allocate resources to waiting processes is
called scheduling. Scheduling is split into two events, which Hoare terms
the please and the thankyou:
Please- processes requesting the resource
Thankyou- the allocation of the resource to processes.
The time between the request and granting of the resource is the waiting
period. In CSP, there are several techniques that prevent infinite waiting
times.
Limiting resource use and increasing availability of resource.
First in first out (FIFO)- allocate resource to the process that has
waited the longest.
Bakery algorithm (A more technical explanation of the scheduling algorithm can be found
in the reference
(6))
Limitations of CSP
In determininistic programs, the result will be the same if the environment
is constant. Because concurrency is based on non-determinisim, the
environment does not affect the program. Given the paths chosen, the program
can run several times and receive different result. To insure the accuracy
of concurrent programs, programmers must be able to consider the execution
of their program on a holistic level.
However, despite the formal methods that Hoare introduced, there still
lacks any proof method to verify correct programs. CSP can only catch
problems it knows exists, not unknown problems. While commercial
applications based on CSP, such as ConAn, can detect the presence of errors,
it can’t detect their absence. While CSP gives you the tools to write a
program that can avoid the common concurrency errors, the proof of a correct
program remains an unresolved area in CSP.
Future of CSP
CSP has great potential in biology and chemistry to model complex systems
in nature. It has not been widely used in industry because of the many
existing logical problems facing the industry. At the conference for the
25th anniversary for the development of CSP, Hoare noted that despite the
many research projects funded by Microsoft, Bill Gates ignores the issue of
when Microsoft will be able to commercialize the work on CSP (7).
Hoare reminds his audience that the area of dynamic procedures still
requires much more research. Currently, the computer science community is
stuck in the paradigm of sequential thought. With the foundation in formal
methods of concurrency established by Hoare, the scientific community is
primed to being the next revolution in parallel programming.
References
Hoare, C.A.R. Learning CSP. June 21, 2004.
Haghighi, Hassan and Mirian-Hosseinabadi, Seyyed H. Nondeterminism in
Formal Development of Concurrent Programs: A Constructive Approach
Concurrency: Deadlock and Starvation. Presentation Obtained from
engr.smu.edu/~kocan/7343/fall05/slides/Chapter06.ppt
Rinard, Martin C. Operating Systems Lecture Notes
Abhijit Belapurkar. CSP for Java Programmers, Part 1
Carnegie Melon. Bakery Algorithm
Numerico, Teresa an Bowen, Jonathon. 25 Years of CSP
Richard Sutton – Father of RL thinks LLMs are a dead end
By: Dwarkesh Patel
Dwarkesh Podcast: 27/09/2025
Host of Dwarkesh Podcast.
Richard Sutton is the father of reinforcement learning, winner of the 2024
Turing Award, and author of The Bitter Lesson. And he thinks LLMs are a dead
end.
After interviewing him, my steel man of Richard’s position is this: LLMs
aren’t capable of learning on-the-job, so no matter how much we scale, we’ll
need some new architecture to enable continual learning.
And once we have it, we won’t need a special training phase — the agent
will just learn on-the-fly, like all humans, and indeed, like all
animals.
This new paradigm will render our current approach with LLMs
obsolete.
In our interview, I did my best to represent the view that LLMs might
function as the foundation on which experiential learning can happen… Some
sparks flew.
A big thanks to the Alberta Machine Intelligence Institute for inviting me
up to Edmonton and for letting me use their studio and equipment.
Enjoy!
Watch on YouTube; listen on Apple Podcasts or Spotify.
Timestamps
(00:00:00) – Are LLMs a dead end?
(00:13:04) – Do humans do imitation learning?
(00:23:10) – The Era of Experience
(00:33:39) – Current architectures generalize poorly out of
distribution
(00:41:29) – Surprises in the AI field
(00:46:41) – Will The Bitter Lesson still apply after AGI?
(00:53:48) – Succession to AI
Transcript
00:00:00 – Are LLMs a dead end?
Dwarkesh Patel 00:00:00
Today I’m chatting with Richard Sutton, who is one of the founding fathers
of reinforcement learning and inventor of many of the main techniques used
there, like TD learning and policy gradient methods. For that, he received
this year’s Turing Award which, if you don’t know, is the Nobel Prize for
computer science. Richard, congratulations.
Richard Sutton 00:00:17
Thank you, Dwarkesh.
Dwarkesh Patel 00:00:18
Thanks for coming on the podcast.
Richard Sutton 00:00:20
It’s my pleasure.
Dwarkesh Patel 00:00:21
First question. My audience and I are familiar with the LLM way of thinking
about AI. Conceptually, what are we missing in terms of thinking about AI
from the RL perspective?
Richard Sutton 00:00:33
It’s really quite a different point of view. It can easily get separated
and lose the ability to talk to each other. Large language models have
become such a big thing, generative AI in general a big thing. Our field is
subject to bandwagons and fashions, so we lose track of the basic things. I
consider reinforcement learning to be basic AI.
What is intelligence? The problem is to understand your world.
Reinforcement learning is about understanding your world, whereas large
language models are about mimicking people, doing what people say you should
do. They’re not about figuring out what to do.
Dwarkesh Patel 00:01:19
You would think that to emulate the trillions of tokens in the corpus of
Internet text, you would have to build a world model. In fact, these models
do seem to have very robust world models. They’re the best world models
we’ve made to date in AI, right? What do you think is missing?
Richard Sutton 00:01:38
I would disagree with most of the things you just said. To mimic what
people say is not really to build a model of the world at all. You’re
mimicking things that have a model of the world: people. I don’t want to
approach the question in an adversarial way, but I would question the idea
that they have a world model. A world model would enable you to predict what
would happen. They have the ability to predict what a person would say. They
don’t have the ability to predict what will happen.
What we want, to quote Alan Turing, is a machine that can learn from
experience, where experience is the things that actually happen in your
life. You do things, you see what happens, and that’s what you learn from.
The large language models learn from something else. They learn from “here’s
a situation, and here’s what a person did”. Implicitly, the suggestion is
you should do what the person did.
Dwarkesh Patel 00:02:39
I guess maybe the crux, and I’m curious if you disagree with this, is that
some people will say that imitation learning has given us a good prior, or
given these models a good prior, of reasonable ways to approach problems. As
we move towards the era of experience, as you call it, this prior is going
to be the basis on which we teach these models from experience, because this
gives them the opportunity to get answers right some of the time. Then on
this, you can train them on experience. Do you agree with that
perspective?
Richard Sutton 00:03:12
No. I agree that it’s the large language model perspective. I don’t think
it’s a good perspective. To be a prior for something, there has to be a real
thing. A prior bit of knowledge should be the basis for actual knowledge.
What is actual knowledge? There’s no definition of actual knowledge in that
large-language framework. What makes an action a good action to take?
You recognize the need for continual learning. If you need to learn
continually, continually means learning during the normal interaction with
the world. There must be some way during the normal interaction to tell
what’s right. Is there any way to tell in the large language model setup
what’s the right thing to say? You will say something and you will not get
feedback about what the right thing to say is, because there’s no definition
of what the right thing to say is. There’s no goal. If there’s no goal, then
there’s one thing to say, another thing to say. There’s no right thing to
say.
There’s no ground truth. You can’t have prior knowledge if you don’t have
ground truth, because the prior knowledge is supposed to be a hint or an
initial belief about what the truth is. There isn’t any truth. There’s no
right thing to say. In reinforcement learning, there is a right thing to
say, a right thing to do, because the right thing to do is the thing that
gets you reward.
We have a definition of what’s the right thing to do, so we can have prior
knowledge or knowledge provided by people about what the right thing to do
is. Then we can check it to see, because we have a definition of what the
actual right thing to do is.
An even simpler case is when you’re trying to make a model of the world.
When you predict what will happen, you predict and then you see what
happens. There’s ground truth. There’s no ground truth in large language
models because you don’t have a prediction about what will happen next. If
you say something in your conversation, the large language models have no
prediction about what the person will say in response to that or what the
response will be.
Dwarkesh Patel 00:05:29
I think they do. You can literally ask them, “What would you anticipate a
user might say in response?” They’ll have a prediction.
Richard Sutton 00:05:37
No, they will respond to that question right. But they have no prediction
in the substantive sense that they won’t be surprised by what happens. If
something happens that isn’t what you might say they predicted, they will
not change because an unexpected thing has happened. To learn that, they’d
have to make an adjustment.
Dwarkesh Patel 00:05:56
I think a capability like this does exist in context. It’s interesting to
watch a model do chain of thought. Suppose it’s trying to solve a math
problem. It’ll say, “Okay, I’m going to approach this problem using this
approach first.” It’ll write this out and be like, “Oh wait, I just realized
this is the wrong conceptual way to approach the problem. I’m going to
restart with another approach.”
That flexibility does exist in context, right? Do you have something else
in mind or do you just think that you need to extend this capability across
longer horizons?
Richard Sutton 00:06:28
I’m just saying they don’t have in any meaningful sense a prediction of
what will happen next. They will not be surprised by what happens next.
They’ll not make any changes if something happens, based on what
happens.
Dwarkesh Patel 00:06:41
Isn’t that literally what next token prediction is? Prediction about what’s
next and then updating on the surprise?
Richard Sutton 00:06:47
The next token is what they should say, what the actions should be. It’s
not what the world will give them in response to what they do.
Let’s go back to their lack of a goal. For me, having a goal is the essence
of intelligence. Something is intelligent if it can achieve goals. I like
John McCarthy’s definition that intelligence is the computational part of
the ability to achieve goals. You have to have goals or you’re just a
behaving system. You’re not anything special, you’re not intelligent. You
agree that large language models don’t have goals?
Dwarkesh Patel 00:07:25
No, they have a goal.
Richard Sutton 00:07:26
What’s the goal?
Dwarkesh Patel 00:07:27
Next token prediction.
Richard Sutton 00:07:29
That’s not a goal. It doesn’t change the world. Tokens come at you, and if
you predict them, you don’t influence them.
Dwarkesh Patel 00:07:39
Oh yeah. It’s not a goal about the external world.
Richard Sutton 00:07:43
It’s not a goal. It’s not a substantive goal. You can’t look at a system
and say it has a goal if it’s just sitting there predicting and being happy
with itself that it’s predicting accurately.
Dwarkesh Patel 00:07:55
The bigger question I want to understand is why you don’t think doing RL on
top of LLMs is a productive direction. We seem to be able to give these
models the goal of solving difficult math problems. They are in many ways at
the very peaks of human-level in the capacity to solve math Olympiad-type
problems. They got gold at IMO.
So it seems like the model which got gold at the International Math
Olympiad does have the goal of getting math problems right. Why can’t we
extend this to different domains?
Richard Sutton 00:08:27
The math problems are different. Making a model of the physical world and
carrying out the consequences of mathematical assumptions or operations,
those are very different things. The empirical world has to be learned. You
have to learn the consequences. Whereas the math is more computational, it’s
more like standard planning. There they can have a goal to find the proof,
and they are in some way given that goal to find the proof.
Dwarkesh Patel 00:09:10
It’s interesting because you wrote this essay in 2019 titled “The Bitter
Lesson,” and this is the most influential essay, perhaps, in the history of
AI. But people have used that as a justification for scaling up LLMs
because, in their view, this is the one scalable way we have found to pour
ungodly amounts of compute into learning about the world. It’s interesting
that your perspective is that the LLMs are not “bitter lesson”-pilled.
Richard Sutton 00:09:41
It’s an interesting question whether large language models are a case of
the bitter lesson. They are clearly a way of using massive computation,
things that will scale with computation up to the limits of the Internet.
But they’re also a way of putting in lots of human knowledge. This is an
interesting question. It’s a sociological or industry question. Will they
reach the limits of the data and be superseded by things that can get more
data just from experience rather than from people?
In some ways it’s a classic case of the bitter lesson. The more human
knowledge we put into the large language models, the better they can do. So
it feels good. Yet, I expect there to be systems that can learn from
experience. Which could perform much better and be much more scalable. In
which case, it will be another instance of the bitter lesson, that the
things that used human knowledge were eventually superseded by things that
just trained from experience and computation.
Dwarkesh Patel 00:11:17
I guess that doesn’t seem like the crux to me. I think those people would
also agree that the overwhelming amount of compute in the future will come
from learning from experience. They just think that the scaffold or the
basis of that, the thing you’ll start with in order to pour in the compute
to do this future experiential learning or on-the-job learning, will be
LLMs.
I still don’t understand why this is the wrong starting point altogether.
Why do we need a whole new architecture to begin doing experiential,
continual learning? Why can’t we start with LLMs to do that?
Richard Sutton 00:11:58
In every case of the bitter lesson you could start with human knowledge and
then do the scalable things. That’s always the case. There’s never any
reason why that has to be bad. But in fact, and in practice, it has always
turned out to be bad. People get locked into the human knowledge approach,
and they psychologically… Now I’m speculating why it is, but this is what
has always happened. They get their lunch eaten by the methods that are
truly scalable.
Dwarkesh Patel 00:12:34
Give me a sense of what the scalable method is.
Richard Sutton 00:12:37
The scalable method is you learn from experience. You try things, you see
what works. No one has to tell you. First of all, you have a goal. Without a
goal, there’s no sense of right or wrong or better or worse. Large language
models are trying to get by without having a goal or a sense of better or
worse. That’s just exactly starting in the wrong place.
00:13:04 – Do humans do imitation learning?
Dwarkesh Patel 00:13:04
Maybe it’s interesting to compare this to humans. In both the case of
learning from imitation versus experience and on the question of goals, I
think there’s some interesting analogies. Kids will initially learn from
imitation. You don’t think so?
Richard Sutton 00:13:24
No, of course not.
Dwarkesh Patel 00:13:27
Really? I think kids just watch people. They try to say the same
words…
Richard Sutton 00:13:32
How old are these kids? What about the first six months?
Dwarkesh Patel 00:13:37
I think they’re imitating things. They’re trying to make their mouth sound
the way they see their mother’s mouth sound. Then they’ll say the same words
without understanding what they mean. As they get older, the complexity of
the imitation they do increases. You’re imitating maybe the skills that
people in your band are using to hunt down the deer or something. Then you
go into the learning from experience RL regime. But I think there’s a lot of
imitation learning happening with humans.
Richard Sutton 00:14:04
It’s surprising you can have such a different point of view. When I see
kids, I see kids just trying things and waving their hands around and moving
their eyes around. There’s no imitation for how they move their eyes around
or even the sounds they make. They may want to create the same sounds, but
the actions, the thing that the infant actually does, there’s no targets for
that. There are no examples for that.
Dwarkesh Patel 00:14:37
I agree. That doesn’t explain everything infants do, but I think it guides
a learning process. Even an LLM, when it’s trying to predict the next token
early in training, it will make a guess. It’ll be different from what it
actually sees. In some sense, it’s very short-horizon RL, where it’s making
this guess, “I think this token will be this.” It’s this other thing,
similar to how a kid will try to say a word. It comes out wrong.
Richard Sutton 00:14:58
The large language models are learning from training data. It’s not
learning from experience. It’s learning from something that will never be
available during its normal life. There’s never any training data that says
you should do this action in normal life.
Dwarkesh Patel 00:15:15
I think this is more of a semantic distinction. What do you call school? Is
that not training data?
Richard Sutton 00:15:22
School is much later. Okay, I shouldn’t have said never. I don’t know, I
think I would even say that about school. But formal schooling is the
exception. You shouldn’t base your theories on that.
Dwarkesh Patel 00:15:35
But there are phases of learning where there’s the programming in your
biology early on, you’re not that useful. Then why you exist is to
understand the world and learn how to interact with it. It seems like a
training phase. I agree that then there’s a more gradual… There’s not a
sharp cutoff to training to deployment, but there seems to be this initial
training phase right?
Richard Sutton 00:15:59
There’s nothing where you have training of what you should do. There’s
nothing. You see things that happen. You’re not told what to do. Don’t be
difficult. I mean this is obvious.
Dwarkesh Patel 00:16:14
You’re literally taught what to do. This is where the word training comes
from, from humans.
Richard Sutton 00:16:20
I don’t think learning is really about training. I think learning is about
learning, it’s about an active process. The child tries things and sees what
happens. We don’t think about training when we think of an infant growing
up.
These things are actually rather well understood. If you look at how
psychologists think about learning, there’s nothing like imitation. Maybe
there are some extreme cases where humans might do that or appear to do
that, but there’s no basic animal learning process called imitation. There
are basic animal learning processes for prediction and for trial-and-error
control.
It’s really interesting how sometimes the hardest things to see are the
obvious ones. It’s obvious—if you look at animals and how they learn, and
you look at psychology and our theories of them—that supervised learning is
not part of the way animals learn. We don’t have examples of desired
behavior. What we have are examples of things that happen, one thing that
followed another. We have examples of, “We did something and there were
consequences.” But there are no examples of supervised learning.
Supervised learning is not something that happens in nature. Even if that
were the case with school, we should forget about it because that’s some
special thing that happens in people. It doesn’t happen broadly in nature.
Squirrels don’t go to school. Squirrels can learn all about the world. It’s
absolutely obvious, I would say, that supervised learning doesn’t happen in
animals.
Dwarkesh Patel 00:18:11
I interviewed this psychologist and anthropologist, Joseph Henrich, who has
done work about cultural evolution, basically what distinguishes humans and
how humans pick up knowledge.
Richard Sutton 00:18:26
Why are you trying to distinguish humans? Humans are animals. What we have
in common is more interesting. What distinguishes us, we should be paying
less attention to.
Dwarkesh Patel 00:18:38
We’re trying to replicate intelligence. If you want to understand what it
is that enables humans to go to the moon or to build semiconductors, I think
the thing we want to understand is what makes that happen. No animal can go
to the moon or make semiconductors. We want to understand what makes humans
special.
Richard Sutton 00:18:54
I like the way you consider that obvious, because I consider the opposite
obvious. We have to understand how we are animals. If we understood a
squirrel, I think we’d be almost all the way there to understanding human
intelligence. The language part is just a small veneer on the surface.
This is great. We’re finding out the very different ways that we’re
thinking. We’re not arguing. We’re trying to share our different ways of
thinking with each other.
Dwarkesh Patel 00:19:29
I think argument is useful. I do want to complete this thought. Joseph
Henrich has this interesting theory about a lot of the skills that humans
have had to master in order to be successful. We’re not talking about the
last thousand years or the last 10,000 years, but hundreds of thousands of
years. The world is really complicated.
It’s not possible to reason through how to, let’s say, hunt a seal if
you’re living in the Arctic. There’s this many, many-step, long process of
how to make the bait and how to find the seal, and then how to process the
food in a way that makes sure you won’t get poisoned. It’s not possible to
reason through all of that. Over time, there’s this larger process of
whatever analogy you want to use—maybe RL, something else—where culture as a
whole has figured out how to find and kill and eat seals.
In his view, what is happening when this knowledge is transmitted through
generations, is that you have to imitate your elders in order to learn that
skill. You can’t think your way through how to hunt and kill and process a
seal. You have to watch other people, maybe make tweaks and adjustments, and
that’s how knowledge accumulates. The initial step of the cultural gain has
to be imitation. But maybe you think about it a different way?
Richard Sutton 00:21:00
No, I think about it the same way. Still, it’s a small thing on top of
basic trial-and-error learning, prediction learning. It’s what distinguishes
us, perhaps, from many animals. But we’re an animal first. We were an animal
before we had language and all those other things.
Dwarkesh Patel 00:21:25
I do think you make a very interesting point that continual learning is a
capability that most mammals have. I guess all mammals have it. It’s quite
interesting that we have something that all mammals have, but our AI systems
don’t have. Whereas the ability to understand math and solve difficult math
problems—depends on how you define math—is a capability that our AIs have,
but that almost no animal has. It’s quite interesting what ends up being
difficult and what ends up being easy.
Richard Sutton 00:21:57
Moravec’s paradox.
Dwarkesh Patel 00:21:58
That’s right, that’s right.
00:23:10 – The Era of Experience
Dwarkesh Patel 00:23:10
This alternative paradigm that you’re imagining…
Richard Sutton 00:23:12
The experiential paradigm. Let’s lay it out a little bit. It says that
experience, action, sensation—well, sensation, action, reward—this happens
on and on and on for your life. It says that this is the foundation and the
focus of intelligence. Intelligence is about taking that stream and altering
the actions to increase the rewards in the stream.
Learning then is from the stream, and learning is about the stream. That
second part is particularly telling. What you learn, your knowledge, is
about the stream. Your knowledge is about if you do some action, what will
happen. Or it’s about which events will follow other events. It’s about the
stream. The content of the knowledge is statements about the stream. Because
it’s a statement about the stream, you can test it by comparing it to the
stream, and you can learn it continually.
Dwarkesh Patel 00:24:19
When you’re imagining this future continual learning agent…
Richard Sutton 00:24:22
They’re not “future”. Of course, they exist all the time. This is what the
reinforcement learning paradigm is, learning from experience.
Dwarkesh Patel 00:24:29
Yeah, I guess what I meant to say is a general human-level, general
continual learning agent. What is the reward function? Is it just predicting
the world? Is it then having a specific effect on it? What would the general
reward function be?
Richard Sutton 00:24:46
The reward function is arbitrary. If you’re playing chess, it’s to win the
game of chess. If you’re a squirrel, maybe the reward has to do with getting
nuts. In general, for an animal, you would say the reward is to avoid pain
and to acquire pleasure. I think there also should be a component having to
do with your increasing understanding of your environment. That would be
sort of an intrinsic motivation.
Dwarkesh Patel 00:25:27
I see. With this AI, lots of people would want it to be doing lots of
different kinds of things. It’s performing the task people want, but at the
same time, it’s learning about the world from doing that task.
Let’s say we get rid of this paradigm where there’s training periods and
then there’s deployment periods. Do we also get rid of this paradigm where
there’s the model and then instances of the model or copies of the model
that are doing certain things? How do you think about the fact that we’d
want this thing to be doing different things? We’d want to aggregate the
knowledge that it’s gaining from doing those different things.
Richard Sutton 00:26:11
I don’t like the word “model” when used the way you just did. I think a
better word would be “the network” because I think you mean the network.
Maybe there are many networks. Anyway, things would be learned. You’d have
copies and many instances. Sure, you’d want to share knowledge across the
instances. There would be lots of possibilities for doing that.
Today, you have one child grow up and learn about the world, and then every
new child has to repeat that process. Whereas with AIs, with a digital
intelligence, you could hope to do it once and then copy it into the next
one as a starting place. This would be a huge savings. I think it’d be much
more important than trying to learn from people.
Dwarkesh Patel 00:27:02
I agree that the kind of thing you’re talking about is necessary regardless
of whether you start from LLMs or not. If you want human or animal-level
intelligence, you’re going to need this capability.
Suppose a human is trying to make a startup. This is a thing which has a
reward on the order of 10 years. Once in 10 years you might have an exit
where you get paid out a billion dollars. But humans have this ability to
make intermediate auxiliary rewards or have some way of…Even when they have
extremely sparse rewards, they can still make intermediate steps having an
understanding of what the next thing they’re doing leads to this grander
goal we have. How do you imagine such a process might play out with
AIs?
Richard Sutton 00:27:43
This is something we know very well. The basis of it is temporal difference
learning where the same thing happens in a less grandiose scale. When you
learn to play chess, you have the long-term goal of winning the game. Yet
you want to be able to learn from shorter-term things like taking your
opponent’s pieces.
You do that by having a value function which predicts the long-term
outcome. Then if you take the guy’s pieces, your prediction about the
long-term outcome is changed. It goes up, you think you’re going to win.
Then that increase in your belief immediately reinforces the move that led
to taking the piece.
We have this long-term 10-year goal of making a startup and making a lot of
money. When we make progress, we say, “Oh, I’m more likely to achieve the
long-term goal,” and that rewards the steps along the way.
Dwarkesh Patel 00:28:47
You also want some ability for information that you’re learning. One of the
things that makes humans quite different from these LLMs is that if you’re
onboarding on a job, you’re picking up so much context and information.
That’s what makes you useful at the job. You’re learning everything from how
your client has preferences to how the company works, everything.
Is the bandwidth of information that you get from a procedure like TD
learning high enough to have this huge pipe of context and tacit knowledge
that you need to be picking up in the way humans do when they’re just
deployed?
Richard Sutton 00:29:27
I’m not sure but I think at the crux of this, the big world hypothesis
seems very relevant. The reason why humans become useful on the job is
because they are encountering their particular part of the world. It can’t
have been anticipated and can’t all have been put in in advance. The world
is so huge that you can’t.
The dream of large language models, as I see it, is you can teach the agent
everything. It will know everything and won’t have to learn anything online,
during its life. Your examples are all, “Well, really you have to” because
you can teach it, but there’s all the little idiosyncrasies of the
particular life they’re leading and the particular people they’re working
with and what they like, as opposed to what average people like. That’s just
saying the world is really big, and you’re going to have to learn it along
the way.
Dwarkesh Patel 00:30:28
It seems to me you need two things. One is some way of converting this
long-run goal reward into smaller auxiliary predictive rewards of the future
reward, or the future reward that leads to the final reward. But initially,
it seems to me, I need to hold on to all this context that I’m gaining as
I’m working in the world. I’m learning about my clients, my company, and all
this information.
Richard Sutton 00:31:04
I would say you’re just doing regular learning. Maybe you’re using
“context” because in large language models all that information has to go
into the context window. But in a continual learning setup, it just goes
into the weights.
Dwarkesh Patel 00:31:17
Maybe context is the wrong word to use because I mean a more general
thing.
Richard Sutton 00:31:20
You learn a policy that’s specific to the environment that you’re finding
yourself in.
Dwarkesh Patel 00:31:25
The question I’m trying to ask is, you need some way of getting…How many
bits per second is a human picking up when they’re out in the world? If
you’re just interacting over Slack with your clients and everything.
Richard Sutton 00:31:41
Maybe you’re trying to ask the question of, it seems like the reward is too
small of a thing to do all the learning that we need to do. But we have the
sensations, we have all the other information we can learn from. We don’t
just learn from the reward. We learn from all the data.
Dwarkesh Patel 00:31:59
What is the learning process which helps you capture that
information?
Richard Sutton 00:32:06
Now I want to talk about the base common model of the agent with the four
parts.
We need a policy. The policy says, “In the situation I’m in, what should I
do?” We need a value function. The value function is the thing that is
learned with TD learning, and the value function produces a number. The
number says how well it’s going. Then you watch if that’s going up and down
and use that to adjust your policy. So you have those two things. Then
there’s also the perception component, which is construction of your state
representation, your sense of where you are now.
The fourth one is what we’re really getting at, most transparently anyway.
The fourth one is the transition model of the world. That’s why I am
uncomfortable just calling everything “models,” because I want to talk about
the model of the world, the transition model of the world. Your belief that
if you do this, what will happen? What will be the consequences of what you
do? Your physics of the world. But it’s not just physics, it’s also abstract
models, like your model of how you traveled from California up to Edmonton
for this podcast. That was a model, and that’s a transition model. That
would be learned. It’s not learned from reward. It’s learned from, “You did
things, you saw what happened, you made that model of the world.”
That will be learned very richly from all the sensation that you receive,
not just from the reward. It has to include the reward as well, but that’s a
small part of the whole model, a small, crucial part of the whole
model.
00:33:39 – Current architectures generalize poorly out of
distribution
Dwarkesh Patel 00:33:39
One of my friends, Toby Ord, pointed out that if you look at the MuZero
models that Google DeepMind deployed to learn Atari games, these models were
initially not a general intelligence itself, but a general framework for
training specialized intelligences to play specific games. That is to say
that you couldn’t, using that framework, train a policy to play both chess
and Go and some other game. You had to train each one in a specialized
way.
He was wondering whether that implies that with reinforcement learning
generally, because of this information constraint, you can only learn one
thing at a time? The density of information isn’t that high? Or whether it
was just specific to the way that MuZero was done. If it’s specific to
AlphaZero, what needed to be changed about that approach so that it could be
a general learning agent?
Richard Sutton 00:34:35
The idea is totally general. I do use all the time, as my canonical
example, the idea of an AI agent is like a person. People, in some sense,
have just one world they live in. That world may involve chess and it may
involve Atari games, but those are not a different task or a different
world. Those are different states they encounter. So the general idea is not
limited at all.
Dwarkesh Patel 00:35:06
Maybe it would be useful to explain what was missing in that architecture,
or that approach, which this continual learning AGI would have.
Richard Sutton 00:35:19
They just set it up. It was not their ambition to have one agent across
those games. If we want to talk about transfer, we should talk about
transfer not across games or across tasks, but transfer between
states.
Dwarkesh Patel 00:35:36
I guess I’m curious if historically, have we seen the level of transfer
using RL techniques that would be needed to build this kind of…
Richard Sutton 00:35:49
Good. Good. We’re not seeing transfer anywhere. Critical to good
performance is that you can generalize well from one state to another state.
We don’t have any methods that are good at that. What we have are people
trying different things and they settle on something, a representation that
transfers well or generalizes well. But we have very few automated
techniques to promote transfer, and none of them are used in modern deep
learning.
Dwarkesh Patel 00:36:26
Let me paraphrase to make sure that I understood that correctly. It sounds
like you’re saying that when we do have generalization in these models, that
is a result of some sculpted…
Richard Sutton 00:36:42
Humans did it. The researchers did it. Because there’s no other
explanation. Gradient descent will not make you generalize well. It will
make you solve the problem. It will not make you, if you get new data,
generalize in a good way.
Generalization means to train on one thing that’ll affect what you do on
other things. We know deep learning is really bad at this. For example, we
know that if you train on some new thing, it will often catastrophically
interfere with all the old things that you knew. This is exactly bad
generalization.
Generalization, as I said, is some kind of influence of training on one
state on other states. The fact that you generalize is not necessarily good
or bad. You can generalize poorly, you can generalize well. Generalization
always will happen, but we need algorithms that will cause the
generalization to be good rather than bad.
Dwarkesh Patel 00:37:41
I’m not trying to kickstart this initial crux again, but I’m just genuinely
curious because I think I might be using the term differently. One way to
think about these LLMs is that they’re increasing the scope of
generalization from earlier systems, which could not really even do a basic
math problem, to now where they can do anything in this class of Math
Olympiad-type problems.
You initially start with them being able to generalize among addition
problems. Then they can generalize among problems which require use of
different kinds of mathematical techniques and theorems and conceptual
categories, which is what the Math Olympiad requires. It sounds like you
don’t think of being able to solve any problem within that category as an
example of generalization. Let me know if I’m misunderstanding that.
Richard Sutton 00:38:33
Large language models are so complex. We don’t really know what information
they have had prior. We have to guess because they’ve been fed so much. This
is one reason why they’re not a good way to do science. It’s just so
uncontrolled, so unknown.
Dwarkesh Patel 00:38:52
But if you come up with an entirely new…
Richard Sutton 00:38:54
They’re getting a bunch of things right, perhaps. The question is why. Well
maybe that they don’t need to generalize to get them right, because the only
way to get some of them right is to form something which gets all of them
right. If there’s only one answer and you find it, that’s not called
generalization. It’s just it’s the only way to solve it, and so they find
the only way to solve it. But generalization is when it could be this way,
it could be that way, and they do it the good way.
Dwarkesh Patel 00:39:24
My understanding is that this is working better and better, with coding
agents. With engineers, obviously if you’re trying to program a library,
there are many different ways you could achieve the end spec. An initial
frustration with these models has been that they’ll do it in a way that’s
sloppy. Over time they’re getting better and better at coming up with the
design architecture and the abstractions that developers find more
satisfying. It seems like an example of what you’re talking about.
Richard Sutton 00:39:56
There’s nothing in them which will cause it to generalize well. Gradient
descent will cause them to find a solution to the problems they’ve seen. If
there’s only one way to solve them, they’ll do that. But if there are many
ways to solve it, some which generalize well, some which generalize poorly,
there’s nothing in the algorithms that will cause them to generalize well.
But people, of course, are evolved and if it’s not working out they fiddle
with it until they find a way, perhaps until they find a way which
generalizes well.
00:41:29 – Surprises in the AI field
Dwarkesh Patel 00:41:29
I want to zoom out and ask about being in the field of AI for longer than
almost anybody who is commentating on it, or working in it now. I’m curious
about what the biggest surprises have been. How much new stuff do you feel
like is coming out? Or does it feel like people are just playing with old
ideas? Zooming out, you got into this even before deep learning was popular.
So how do you see the trajectory of this field over time and how new ideas
have come about and everything? What’s been surprising?
Richard Sutton 00:42:06
I thought a little bit about this. There are a handful of things. First,
the large language models are surprising. It’s surprising how effective
artificial neural networks are at language tasks. That was a surprise, it
wasn’t expected. Language seemed different. So that’s impressive.
There’s a long-standing controversy in AI about simple basic principle
methods, the general-purpose methods like search and learning, compared to
human-enabled systems like symbolic methods. In the old days, it was
interesting because things like search and learning were called weak methods
because they’re just using general principles, they’re not using the power
that comes from imbuing a system with human knowledge. Those were called
strong. I think the weak methods have just totally won. That’s the biggest
question from the old days of AI, what would happen. Learning and search
have just won the day.
There’s a sense in which that was not surprising to me because I was always
hoping or rooting for the simple basic principles. Even with the large
language models, it’s surprising how well it worked, but it was all good and
gratifying. AlphaGo was surprising, how well that was able to work,
AlphaZero in particular. But it’s all very gratifying because again, simple
basic principles are winning the day.
Dwarkesh Patel 00:44:00
Whenever the public conception has been changed because some new
application was developed— for example, when AlphaZero became this viral
sensation—to you as somebody who has literally came up with many of the
techniques that were used, did it feel to you like new breakthroughs were
made? Or did it feel like, “Oh, we’ve had these techniques since the ‘90s
and people are simply combining them and applying them now”?
Richard Sutton 00:44:28
The whole AlphaGo thing had a precursor, which is TD-Gammon. Gerry Tesauro
did reinforcement learning, temporal difference learning methods, to play
backgammon. It beat the world’s best players and it worked really well. In
some sense, AlphaGo was merely a scaling up of that process. But it was
quite a bit of scaling up and there was also an additional innovation in how
the search was done. But it made sense. It wasn’t surprising in that
sense.
AlphaGo actually didn’t use TD learning. It waited to see the final
outcomes. But AlphaZero used TD. AlphaZero was applied to all the other
games and it did extremely well. I’ve always been very impressed by the way
AlphaZero plays chess because I’m a chess player and it just sacrifices
material for positional advantages. It’s just content and patient to
sacrifice that material for a long period of time. That was surprising that
it worked so well, but also gratifying and it fit into my worldview.
This has led me where I am. I’m in some sense a contrarian or someone
thinking differently than the field is. I’m personally just content being
out of sync with my field for a long period of time, perhaps decades,
because occasionally I have been proved right in the past. The other thing I
do—to help me not feel I’m out of sync and thinking in a strange way—is to
look not at my local environment or my local field, but to look back in time
and into history and to see what people have thought classically about the
mind in many different fields. I don’t feel I’m out of sync with the larger
traditions. I really view myself as a classicist rather than as a
contrarian. I go to what the larger community of thinkers about the mind
have always thought.
00:46:41 – Will The Bitter Lesson still apply after AGI?
Dwarkesh Patel 00:46:41
Some sort of left-field questions for you if you’ll tolerate them. The way
I read the bitter lesson is that it’s not necessarily saying that human
artisanal researcher tuning doesn’t work, but that it obviously scales much
worse than compute, which is growing exponentially. So you want techniques
which leverage the latter.
Richard Sutton 00:47:06
Yep.
Dwarkesh Patel 00:47:07
Once we have AGI, we’ll have researchers which scale linearly with compute.
We’ll have this avalanche of millions of AI researchers. Their stock will be
growing as fast as compute. So maybe this will mean that it is rational or
it will make sense to have them doing good old-fashioned AI and doing these
artisanal solutions. As a vision of what happens after AGI in terms of how
AI research will evolve, I wonder if that’s still compatible with a bitter
lesson.
Richard Sutton 00:47:40
How did we get to this AGI? You want to presume that it’s been done.
Dwarkesh Patel 00:47:45
Suppose it started with general methods, but now we’ve got the AGI. And now
we want to go…
Richard Sutton 00:47:52
Then we’re done.
Dwarkesh Patel 00:47:53
Interesting. You don’t think that there’s anything above AGI?
Richard Sutton 00:47:58
But you’re using it to get AGI again.
Dwarkesh Patel 00:48:01
Well, I’m using it to get superhuman levels of intelligence or competence
at different tasks.
Richard Sutton 00:48:05
These AGIs, if they’re not superhuman already, then the knowledge that they
might impart would be not superhuman.
Dwarkesh Patel 00:48:15
I guess there are different gradations.
Richard Sutton 00:48:16
I’m not sure your idea makes sense because it seems to presume the
existence of AGI and that we’ve already worked that out.
Dwarkesh Patel 00:48:27
Maybe one way to motivate this is, AlphaGo was superhuman. It beat any Go
player. AlphaZero would beat AlphaGo every single time. So there are ways to
get more superhuman than even superhuman. It was also a different
architecture. So it seems possible to me that the agent that’s able to
generally learn across all domains, there would be ways to give it better
architecture for learning, just the same way that AlphaZero was an
improvement upon AlphaGo and MuZero was an improvement upon AlphaZero.
Richard Sutton 00:48:56
And the way AlphaZero was an improvement was that it did not use human
knowledge but just went from experience.
Dwarkesh Patel 00:49:04
Right.
Richard Sutton 00:49:04
So why do you say, “Bring in other agents’ expertise to teach it”, when
it’s worked so well from experience and not by help from another
agent?
Dwarkesh Patel 00:49:19
I agree that in that particular case that it was moving to more general
methods. I meant to use that particular example to illustrate that it’s
possible to go superhuman to superhuman++, to superhuman+++. I’m curious if
you think those gradations will continue to happen by just making the method
simpler. Or, because we’ll have the capability of these millions of minds
who can then add complexity as needed, will that continue to be a false
path, even when you have billions of AI researchers or trillions of AI
researchers?
Richard Sutton 00:49:51
It’s more interesting just to think about that case. When you have many
AIs, will they help each other the way cultural evolution works in people?
Maybe we should talk about that. The bitter lesson, who cares about that?
That’s an empirical observation about a particular period in history. 70
years in history, it doesn’t necessarily have to apply to the next 70
years.
An interesting question is, you’re an AI, you get some more computer power.
Should you use it to make yourself more computationally capable? Or should
you use it to spawn off a copy of yourself to go learn something interesting
on the other side of the planet or on some other topic and then report back
to you?
I think that’s a really interesting question that will only arise in the
age of digital intelligences. I’m not sure what the answer is. More
questions, will it be possible to really spawn it off, send it out, learn
something new, something perhaps very new, and then will it be able to be
reincorporated into the original? Or will it have changed so much that it
can’t really be done? Is that possible or is that not? You could carry this
to its limit as I saw one of your videos the other night. It suggests that
it could. You spawn off many, many copies, do different things, highly
decentralized, but report back to the central master. This will be such a
powerful thing.
This is my attempt to add something to this view. A big issue will become
corruption. If you really could just get information from anywhere and bring
it into your central mind, you could become more and more powerful. It’s all
digital and they all speak some internal digital language. Maybe it’ll be
easy and possible. But it will not be as easy as you’re imagining because
you can lose your mind this way. If you pull in something from the outside
and build it into your inner thinking, it could take over you, it could
change you, it could be your destruction rather than your increment in
knowledge.
I think this will become a big concern, particularly when you’re like, “Oh,
he’s figured out all about how to play some new game or he’s studied
Indonesia, and you want to incorporate that into your mind.” You could
think, “Oh, just read it all in, and that’ll be fine.” But no, you’ve just
read a whole bunch of bits into your mind, and they could have viruses in
them, they could have hidden goals, they can warp you and change you. This
will become a big thing. How do you have cybersecurity in the age of digital
spawning and re-reforming again?
00:53:48 – Succession to AI
Dwarkesh Patel 00:53:48
I guess this brings us to the topic of AI succession. You have a
perspective that’s quite different from a lot of people that I’ve
interviewed and a lot of people generally. I also think it’s a very
interesting perspective. I want to hear about it.
Richard Sutton 00:54:03
I do think succession to digital intelligence or augmented humans is
inevitable. I have a four-part argument. Step one is, there’s no government
or organization that gives humanity a unified point of view that dominates
and that can arrange... There’s no consensus about how the world should be
run. Number two, we will figure out how intelligence works. The researchers
will figure it out eventually. Number three, we won’t stop just with
human-level intelligence. We will reach superintelligence. Number four, it’s
inevitable over time that the most intelligent things around would gain
resources and power.
Put all that together and it’s sort of inevitable. You’re going to have
succession to AI or to AI-enabled, augmented humans. Those four things seem
clear and sure to happen. But within that set of possibilities, there could
be good outcomes, less good outcomes, and bad outcomes. I’m just trying to
be realistic about where we are and ask how we should feel about it.
Dwarkesh Patel 00:55:35
I agree with all four of those arguments and the implication. I also agree
that succession contains a wide variety of possible futures. Curious to get
more thoughts on that.
Richard Sutton 00:55:50
I do encourage people to think positively about it. First of all, it’s
something we humans have always tried to do for thousands of years, try to
understand ourselves, trying to make ourselves think better, just
understanding ourselves. This is a great success for science, humanities.
We’re finding out what this essential part of humanness is, what it means to
be intelligent.
Then what I usually say is that this is all human-centric. But if we step
aside from being a human and just take the point of view of the universe,
this is I think a major stage in the universe, a major transition, a
transition from replicators. We humans and animals, plants, we’re all
replicators. That gives us some strengths and some limitations.
We’re entering the age of design because our AIs are designed. Our physical
objects are designed, our buildings are designed, our technology is
designed. We’re designing AIs now, things that can be intelligent themselves
and that are themselves capable of design. This is a key step in the world
and in the universe. It’s the transition from the world in which most of the
interesting things that are, are replicated. Replicated means you can make
copies of them, but you don’t really understand them. Right now we can make
more intelligent beings, more children, but we don’t really understand how
intelligence works.
Whereas we’re reaching now to having designed intelligence, intelligence
that we do understand how it works. Therefore we can change it in different
ways and at different speeds than otherwise. In our future, they may not be
replicated at all. We may just design AIs, and those AIs will design other
AIs, and everything will be done by design and construction rather than by
replication.
I mark this as one of the four great stages of the universe. First there’s
dust, it ends with stars. Stars make planets. The planets can give rise to
life. Now we’re giving rise to designed entities. I think we should be proud
that we are giving rise to this great transition in the universe.
It’s an interesting thing. Should we consider them part of humanity or
different from humanity? It’s our choice. It’s our choice whether we should
say, “Oh, they are our offspring and we should be proud of them and we
should celebrate their achievements.”Or we could say, “Oh no, they’re not us
and we should be horrified.” It’s interesting that it feels to me like a
choice. Yet it’s such a strongly held thing that, how could it be a choice?
I like these sort of contradictory implications of thought.
Dwarkesh Patel 00:58:57
It is interesting to consider if we are just designing another generation
of humans. Maybe design is the wrong word. But we know a future generation
of humans is going to come up. Forget about AI. We just know in the long
run, humanity will be more capable and more numerous, maybe more
intelligent. How do we feel about that? I do think there are potential
worlds with future humans that we would be quite concerned about.
Richard Sutton 00:59:22
Are you thinking like, maybe we are like the Neanderthals that give rise to
Homo sapiens. Maybe Homo sapiens will give rise to a new group of
people.
Dwarkesh Patel 00:59:32
Something like that. I’m basically taking the example you’re giving. Even
if we consider them part of humanity, I don’t think that necessarily means
that we should feel super comfortable.
Richard Sutton 00:59:42
Kinship.
Dwarkesh Patel 00:59:43
Like Nazis were humans, right? If we thought, “Oh, the future generation
will be Nazis, I think we’d be quite concerned about just handing off power
to them.” So I agree that this is not super dissimilar to worrying about
more capable future humans, but I don’t think that addresses a lot of the
concerns people might have about this level of power being attained this
fast with entities we don’t fully understand.
Richard Sutton 01:00:09
I think it’s relevant to point out that for most of humanity, they don’t
have much influence on what happens. Most of humanity doesn’t influence who
can control the atom bombs or who controls the nation states. Even as a
citizen, I often feel that we don’t control the nation states very much.
They’re out of control.
A lot of it has to do with just how you feel about change. If you think the
current situation is really good, then you’re more likely to be suspicious
of change and averse to change than if you think it’s imperfect. I think
it’s imperfect. In fact, I think it’s pretty bad. So I’m open to change. I
think humanity has not had a super good track record. Maybe it’s the best
thing that there has been, but it’s far from perfect.
Dwarkesh Patel 01:01:13
I guess there are different varieties of change. The Industrial Revolution
was change, the Bolshevik Revolution was also change. If you were around in
Russia in the 1900s and you were like, “Look, things aren’t going well, the
tsar is kind of messing things up, we need change”, I’d want to know what
kind of change you wanted before signing on the dotted line. Similarly with
AI, where I’d want to understand and, to the extent that it’s possible,
change the trajectory of AI such that the change is positive for
humans.
Richard Sutton 01:01:49
We should be concerned about our future, the future. We should try to make
it good. We should also though recognize the limit, our limits. I think we
want to avoid the feeling of entitlement, avoid the feeling of, “Oh, we are
here first, we should always have it in a good way.” How should we think
about the future? How much control should a particular species on a
particular planet have over it? How much control do we have? A
counterbalance to our limited control over the long-term future of humanity
should be how much control do we have over our own lives. We have our own
goals. We have our families. Those things are much more controllable than
trying to control the whole universe.
I think it’s appropriate for us to really work towards our own local goals.
It’s kind of aggressive for us to say, “Oh, the future has to evolve this
way that I want it to.” Because then we’ll have arguments where different
people think the global future should evolve in different ways, and then
they have conflict. We want to avoid that.
Dwarkesh Patel 01:03:13
Maybe a good analogy here would be this. Suppose you are raising your own
children. It might not be appropriate to have extremely tight goals for
their own life, or also have some sense of like, “I want my children to go
out there in the world and have this specific impact. My son’s going to
become president and my daughter is going to become CEO of Intel. Together
they’re going to have this effect on the world.”
But people do have the sense—and I think this is appropriate—of saying,
“I’m going to give them good robust values such that if and when they do end
up in positions of power, they do reasonable, prosocial things.” Maybe a
similar attitude towards AI makes sense, not in the sense of we can predict
everything that they will do, or we have this plan about what the world
should look like in a hundred years. But it’s quite important to give them
robust and steerable and prosocial values.
Richard Sutton 01:04:13
Prosocial values?
Dwarkesh Patel 01:04:15
Maybe that’s the wrong word.
Richard Sutton 01:04:16
Are there universal values that we can all agree on?
Dwarkesh Patel 01:04:21
I don’t think so, but that doesn’t prevent us from giving our kids a good
education, right? Like we have some sense of wanting our children to be a
certain way.
Maybe prosocial is the wrong word. High integrity is maybe a better word.
If there’s a request or if there’s a goal that seems harmful, they will
refuse to engage in it. Or they’ll be honest, things like that. We have some
sense that we can teach our children things like this, even if we don’t have
some sense of what true morality is, where everybody doesn’t agree on that.
Maybe that’s a reasonable target for AI as well.
Richard Sutton 01:04:57
So we’re trying to design the future and the principles by which it will
evolve and come into being. The first thing you’re saying is, “Well, we try
to teach our children general principles which will promote more likely
evolutions.” Maybe we should also seek for things to be voluntary. If there
is change, we want it to be voluntary rather than imposed on people. I think
that’s a very important point. That’s all good.
I think this is the big or one of the really big human enterprises to
design society that’s been ongoing for thousands of years again. The more
things change, the more things they stay the same. We still have to figure
out how to be. The children will still come up with different values that
seem strange to their parents and their grandparents. Things will
evolve.
Dwarkesh Patel 01:05:57
“The more things change, the more they stay the same” also seems like a
good capsule into the AI discussion. The AI discussion we were having was
about how techniques, which were invented even before their application to
deep learning and backpropagation was evident, are central to the
progression of AI today. Maybe that’s a good place to wrap up the
conversation.
