Building Autonomous AI
Kareem Ayoub (Marshall Scholar 2012) — Vice President, AI Technical Strategy, Google DeepMind
Timothy Wu — Professor, Columbia Law School
2026 Marshall Forum on AI
October 19, 2026 · Stanford University
CONVERSATION | Lightly edited transcript
Timothy Wu
All right. Hi everybody. Let's get started. My name is Tim Wu. I am a professor at Columbia Law School. I was briefly a visiting professor at Stanford Law School, so I somewhat know my way around the campus. Today we are joined by Kareem Ayoub, who is a VP of AI Technical Strategy at Google DeepMind. Thank you so much for joining us. And we're going to talk about AI, what else, and some of the nuts and bolts sides of this topic. So Kareem, why don't we start with you? You graduated from Rice University, if I'm not mistaken, took the Marshall, went to Oxford, and you ended up joining DeepMind in 2014, which makes you in AI terms like ancient, even though you're not an old person.
So why don't you tell us a little bit about those early days and how you came to join DeepMind in England and how that went down?
Kareem Ayoub (Marshall Scholar 2012)
Sure. First of all, thank you all for having me. Thank you to Stanford Law School and the Association of Marshall Scholars for hosting this, Tim, for moderating this, appreciate it, and everyone here for joining us. I think of the US chapter of my life as the pursuit of understanding of the organic brain and the UK chapter of my life as the artificial brain. And what I realized on the US side of my academics and my work was just how little I knew about the brain after having studied it for so long and so intensely. And at the same time, I was making this observation that the size and the complexity of the problems that I was trying to tackle were increasing more rapidly than my unaided ability to make progress against them.
And so what drew me to AI at the time, and this is at a time when it was not the most widely used term, especially saying something like artificial general intelligence was pretty uncommon.
Timothy Wu
You haven't switched to super intelligence.
Kareem Ayoub
Or super intelligence. This was at a time when that was very early, but the thing that drew me to it was I really came to think of AI as a force multiplier for human ingenuity, and I loved that because I wanted it to be a tool that allowed us to accelerate our problem solving capabilities. And that's what drew me to, at the time, what was a little-known company called DeepMind. And we were able to lean into this idea of AI as a force multiplier for human ingenuity by really exploring how AI could identify patterns in very complex environments that allowed us to make extraordinary progress in those environments.
Timothy Wu
I think a lot of people would like the idea of going to graduate school in England and happening to happen into, frankly, one of the most exciting startups. How did it actually happen? How did it actually go from here I am a graduate student to finding this company?
Kareem Ayoub
Sure. again, because it was so early, this was at a time when I could be speaking on a panel alongside one of the co-founders of DeepMind, and one thing leads to another, and I'm able to partner with some incredible founders of the company and now incredible colleagues over at Google to do pretty amazing work. This was very early, so this was before things like AlphaGo, for example, or AlphaFold and things like that.
Timothy Wu
Can you remind everyone just what AlphaGo and AlphaFold are just in case?
Kareem Ayoub
Sure. So it goes back to this idea of AI as a force multiplier. So AlphaGo was an AI system that we built to play the board game Go. And for those who don't know, Go is this incredibly old, over 2,000 years, but incredibly complex board game. So there are more possible moves in the game of Go than there are atoms in the known universe. And the reason that that is important is because you can't brute force the best move every single turn. And so we built AlphaGo, and in 2016, it played at the time, the world number one, Lee Sedol, and won across a series of matches.
But the thing that was more exciting than that was AlphaGo was able to play moves like the now famous Move 37, which went against 2,000 years of human convention of how to play the game. It was so unconventional that the commentators thought it was a mistake when it first was played, but it turned out to be a very brilliant move at the time.
AlphaFold, fast-forward a few years, was another AI system that we'd built that looked at the grand challenge of protein folding. And the reason proteins are so important is because they are the building blocks of life. If you understand how a protein goes from a one-dimensional sequence to how it folds, you can understand how it interacts with other parts of the body and also certain things like disease pathways. We built AlphaFold and then released the 200 million protein structures known to science for free to the world. And now something like over three million scientists across 190 plus different countries are using AlphaFold to accelerate everything from drug therapeutics to our understanding of neglected diseases.
Timothy Wu
If I'm not mistaken, there was Nobel Prize awarded for some of that work for your boss or your… Yeah. Did I get that right? Yeah.
Kareem Ayoub
Yes, precisely. So 2024, yes, exactly. Demis was awarded the Nobel Prize alongside a couple others for the work on AlphaFold because of the consequential nature of it. And since AlphaFold, it was then extended into AlphaGenome, which was released at the end of last year, but the AlphaGenome Atlas was actually just released this year. And the reason that the AlphaGenome complements AlphaFold is AlphaFold looks at the 2% of the genome that encodes for proteins directly. AlphaGenome looks at the 98% of the rest of the genome that encodes for how the proteins are built, where they're built, why they're built, to what volumes they're built.
And so with the two of those, you actually get a really incredible source of information that allows scientists and doctors to shift their focus from how do I crystallize and understand the structure to what hypothesis do I have on how to make progress against this disease or understanding that disease pathway?
Timothy Wu
Now, what you've just been talking about is, in my mind, what I associate DeepMind mainly with, which is frontier research, heroic kind of discoveries and similar works. I take it as part of your job is to try to translate some of that ability into, I guess, it's not to say this stuff isn't practical, but more day-to-day kind of stuff. And one of the conversations ongoing I think this year or one of the developments is the switch or the move to agentic versions of AI. Why don't you talk a little bit I take this as part of your job description, the challenge of moving from the more frontier style, heroic kind of AI research and discoveries and putting into a more practical setting for corporations or individuals.
Kareem Ayoub
Over the last year, if you take AI's development in phases, before 2026, I'd say the primary mechanism of working with AI was in this chatbot style, you submit a question or a request, you get a response, question, response, question, response. And we've shifted now from that model being the product to more the agent and the system being the product. And part of that shift is driven by this kind of observation that if you take a model and you give that model tools, maybe the ability to write code and execute code, you give it a CPU, so access to some persistent state, maybe some memory, maybe access to the internet, and you take that whole tool belt and you wrap it around the model, that's known as a harness, and it imbues that model within that system with the capability to work over longer time horizons, to take on more complicated tasks.
And that becomes incredibly important because the interaction mode shifts from quick question-and-response to here's a more ambitious or less defined goal, and please go handle some of this messy middle while in pursuit of that goal. And so that interaction pattern shifts fundamentally when you go from this model to the agent, which is what then we look at within businesses, okay, how do I take that particularly within more verifiable domains like software engineering to extract some value?
Timothy Wu
Can you boil that down? If you were going to define most cleanly the difference between, I guess, chat AI, chatbot AI and agentic AI, knowing that they're often through the same interface, and many people in this room may be actually using agentic AI without realizing it's one of those things that can happen. You ask a question, then if you were forced to precisely define the line between the two, is that possible or what would you say?
Kareem Ayoub
Sure. there are probably different definitions that folks will focus on, but the technical way to look at this is are you, for example, doing a single forward pass of the model where you submitted some sort of prompt and the model is doing its next token generation to produce an output or is the model calling itself? Is it writing code? Is it writing to memory? Is it operating over a longer time horizon? You see differences in the behaviors of those two. So with an agent, you'll have additional tokens per turn, you'll have additional model calls per turn, you'll have additional sessions per user.
And if I tell it through the story of or through the lens of, for example, software engineering, I'd say maybe around November 2025, December 2025, there was a shift where before that most software engineers were using AI to auto complete lines of code.
After that shift due to a combination of the model capability maybe and potentially the overall harness that the agent sat within, software engineers shifted to not really using their integrated development environments too often and primarily interacting with code through the lens of interacting with an agent. And so you get a fundamental shift in the way of working as well when you go from the model to the agent.
Timothy Wu
Why don't we do a poll of our audience here? Who here, I'm going to assume people in this room use… Well, maybe I'll ask first, who here uses AI on a daily basis? I'm going to guess. Who here finds themselves using agentic AI? Who here doesn't know if they're using agentic AI or not? No one maybe wants to admit to that. Yeah, so it seems like a lot of this room is shifting to the more agentic. Well, there was this amusing tweet last week written by Sonya, I looked this up after our conversation, Sonia Baschez, who's I guess a podcaster, and she said something I thought was kind of entertaining.
She said, Silicon Valley, I'm paraphrasing, keeps building husbands. By that she means these agentic agents or AIs that'll do some kind of heroic task. But what I really need in my life is more like a wife.
I need someone who's constantly doing all these intermediate work. And I just thought you had a comment on that. Is that a challenge? Is that the direction you're trying to take things DeepMind?
Kareem Ayoub
I don't know that particular tweet. What I will say is a lot of the anxiety around perhaps what's being conveyed with that is that on the one hand you've got these incredible model capabilities and these flashes of brilliance. And then on the other, sometimes it falls a little bit short of expectations. So on the one hand, we've got AI that is helping us to restore and translate ancient texts that have been lost to history to predict adverse weather events, to save large swaths of communities. We're literally pointing AI into the cosmos and listening to the faint whispers of gravitational waves created by black holes colliding billions of light years away.
And at the same time, within businesses, very understandably, we're asking very basic questions around where's the value in this technology for me and my work every single day? And it is that contrast that is driving a lot of, okay, I know it can do these incredible things, but what does that actually mean for me as I deploy it, as I use it to summarize and write emails or draft a strategy report or do some financial analysis?
And that's probably the contrast that people are grappling with on a day-to-day basis.
Timothy Wu
How much do you think the challenge sometimes is also for typical logistical tasks? I was thinking about, I want AI to help me do my… Maybe other people have got this figured out, but it has not helped me with my receipts and just the typical, the most annoying part of every job is I need to file all my receipts and stuff. Maybe if someone here's got that figured out, and I was thinking a lot of it comes down to there's so many autonomous systems that you need access to and all these petty things, not like can we read the waves of the early universe and how they're talking to us, but more like I need access to this system to read this and then everybody's sort of blocked out.
How much in the work you do you find that less of the higher physics problems and more of the for this business, they don't have access to this data or that system or this kind of thing?
Kareem Ayoub
I've been fortunate in my role to be able to engage with lots of companies over the last few years, several hundred, and I'm always learning from them to be clear. And what I've noticed is that the best companies and the best people at those companies do a few things really well that have been durable over, I'd say the last two or three years in particular. They really focus on getting their sandboxes and infrastructure right. So all of the connectors that you're talking about, the ability for their people to move really fast within predefined guardrails becomes extraordinarily important.
The second thing they do really well is they bet with the improving capabilities of models and agents. The best companies will ask themselves, "Is the thing that I'm building going to become 10 times better in six months when the underlying model or agent improves 10 times or will it regress or get threatened?" The third thing the best companies seem to do is they really nail their evaluations, the actual metrics that they care about.
And that's particularly hard because when you have this technology which is accelerating one slice of the work that you do, that bottleneck shifts to other parts of the company and what you used to measure becomes less useful or less directly sampling what you're trying to actually make progress against.
Timothy Wu
Do you have an example of that shifting? It's a really good point about shifting bottleneck or shifting metric. Do you have an example or is it too hard to describe an example where you thought that this was the right measure of progress, but you were wrong because a different bottleneck showed up?
Kareem Ayoub
Oh, sure. And by the way, a lot of this comes from… MIT in the mid-1990s had some really great work from the environmental sciences on how you intervene in complex systems. And the summary of that research, and then I'll come onto your question, was humans are not, this was their research to be clear, humans were not great at identifying where to have the highest impact intervention within sufficiently complex systems. We often gravitate towards the lowest hanging fruits, and if we identify the high leverage things to do, we often push them in the wrong direction. For the work that we see where that bottleneck can shift, if you look at software engineering as one example, okay, you can now write code much faster, but has the review of that code accelerated as well?
Has the ability for you to deploy that into production accelerated at the same rate?
And that answer is different for different companies depending on your risk profile, how you've set up your infrastructure. Same thing if you're a marketing leader within a company. If you can now produce marketing content and images and videos infinitely faster, okay, well, you still have to review all the proposals that are coming across your desk and make sure that it makes sense and that you want it to go out.
Timothy Wu
Yeah, in the end, it always ends up that we're the bottlenecks, the human bandwidth ends in time. frankly, if we could just expand the day beyond 24 hours, I don't know if you're working on that, but it's a great point. Let me bring up something related now that we're talking about deployment. I was on a panel in a conversation with Sebastian Mallaby, he wrote a book about DeepMind, and one of the things that he mentioned that I thought was very interesting was that companies that have highly proprietary data are actually leaning against American companies and towards, this sort of surprised me, Chinese open source models, mainly because they don't fully trust that the American companies will not end up getting their data one way or another.
So they want an open weight system, something completely open source where they can build it themselves.
He was saying it, have you found that a challenge in your efforts to deploy or do you have some version, I guess, of some of the technology to deploy that doesn't end up one way or another getting at sensitive data?
Kareem Ayoub
There are a couple things there. So first is when you think about the layers of the technology as it's deployed, you're moving beyond just the model now. So we talked about agents and what it means to build that harness around the model. The amazing thing is that we're seeing a lot of companies and individuals build into their own harnesses across multiple different models in such a way that it benefits their work. And the reason for that is in a domain like software engineering where you have a verifiable goal, the verifiable goal could be as simple as does the code compile, yes or no?
It's easier to have an agent work towards a goal of, okay, make the code compile. But if you're a marketing leader or a lawyer or some other field where the work that you do is less objectively verifiable in the same way that you might describe software engineering, you have to somehow encode that in your own way, and you can do that via systems like the harness.
And so that becomes an extraordinary tool for people to build that value. The other thing I would say is one of the big things that businesses are focusing on as they think about models and technologies and agents is moving away from purely thinking about just token production. So a lot of the way in which we use this technology is token production for knowledge in some way, shape or form. But businesses are having to turn onto how do I balance that with other things I also care about like cost or am I getting this task done and am I getting this task done within a certain amount of time?
Timothy Wu
But what about his point about data? he said what he was surprised to find, say your average hedge fund that has this, what it regards as very sensitive data, and he said to his surprise, he said a lot of companies want to air gap, have no access and have a totally open weight, open source kind of model. I guess have you found that to be a challenge or are you planning to offer something similar like that? I'm just curious if that is something you guys have found to be a challenge.
Kareem Ayoub
Sure. first of all, Google of course is developing both proprietary models as well as open models. There's our generative media models, video, audio. So there's a whole host of models specifically for the reason that you outlined, which is that companies and businesses and governments are optimizing for different things at different times. And you can think of it as a Pareto frontier where you can optimize for cost, latency, performance, and control. And depending on where you sit across those axes, you will prioritize those four dimensions in different ways and your level of risk appetite, what you're willing to pay, the level of performance that's needed, all of it is contextual within the work that you do on a day-to-day basis.
And so of course, the industry is seeing different people sit on different points on that Pareto curve, totally natural, and I think it depends on what you're optimizing for.
Timothy Wu
Let's talk, I think, necessarily about AI safety. It strikes me that this conversation for a long time was a very abstract conversation, pitting narratives of curing every disease possible overnight versus imminent human extinction, robot takeover, terminator-like scenarios. But at least this summer I've started to feel, and I'm particularly referring to Hugging Face, when that became public and some of the other OpenAI incidents became public, suddenly I feel it's taken on a more practical bent. And no, maybe we're not talking about immediate extinction and robot takeover, but we did see in the Hugging Face incident I'm referring to, OpenAI models break out of their, I guess, container or wherever they were supposed to be and break into Hugging Face trying to solve a problem, hack their way into Hugging Face, another site.
So in other talks, I've heard you talk about some of the ways you think this should be approached, and I just want to give you an opportunity to talk about that.
Kareem Ayoub
Sure. There's a lot of conversation about this in the industry at the moment and in the public zeitgeist, and it seems like there is a tension between the exponential increase in capabilities of the technology and the linear governance keeping pace. And if you decompose that tension along a few axes, you end up thinking about the current conversation in maybe three buckets. The first is there's an active conversation on what is the role that AI has in the research and development lifecycle itself? The second is how do we ensure the right level of explainability and transparency into how these systems themselves work?
The third axis is control, and specifically one of what is the control plane for people, and in what ways does that intersect with these technologies? A lot of the tension decomposes into those three, and it underscores the importance of having technical mitigations to help us really understand the actual systems that we're building.
So for example, and to be clear, everyone has different approaches here, Google has the frontier safety framework with critical capability levels on different capabilities that one would care about that we track and measure progress towards that we can mitigate in a technical way, any potential misuse or progress towards those thresholds in a harmful way, for example. So it's really about understanding within these very complex systems, how do you ensure and underscore the correct technical mitigations associated with that system?
Timothy Wu
Now, one of the things that people sometimes say, and I think this makes a lot of intuitive sense, it's like, how can we start deploying technologies when we don't fully understand how they work? You know what I mean? We were talking about education earlier. Someone asked whether we could really deploy things if there's no evidence, they improve learning. That's a slightly different point. But overall, if we don't understand exactly, and it's a black box, is that a safety concern or do you see it differently? How do you answer that critique?
Kareem Ayoub
The way I'd like to answer that critique is that you really lean into governance as a superpower. I'll give you a couple examples. Take a historical example first. We didn't know how aspirin worked for 70, 75 years before we finally understood the mechanism for how it worked. It was awarded a Nobel Prize for that understanding. But the key observation is that you've got a probabilistic system. These generative AI technologies are probabilistic inherently, and Rice's theorem, Turing's halting problem, give us some insight into what to expect from that probabilistic system, but very importantly, the way in which you can deploy that into infrastructure that becomes extraordinarily sensitive.
I think it's also helpful to look at historical analogs where other industries have had to also tackle very similar challenges. If you look at the late '80s, early '90s with the financial services sector, you had financial services companies that were deploying black box or probabilistic, whatever technologies at the time.
And what became very important was to be able to provide a deterministic perimeter around that probabilistic system that allowed you to govern the way in which that system interacted with other systems, and that's being deployed into very sensitive financial infrastructure. The lesson there is alongside all of the work on the system itself, which we just talked about, so the continued investment into really understanding how to mitigate from a technical perspective and make progress on monitoring and safety and explainability, you also have to think about what is that deterministic governance perimeter, and to use that as a superpower as you think about deployments for your use cases.
Timothy Wu
Now, you brought up the computer trading systems as a possible model. If I'm not mistaken, they underwent some pretty significant failures at certain times, particularly in the first financial crisis, sorry, the 2008 financial crisis, other situations. Are there lessons to be learned from that? It seems to me that if I remember correctly about the systems, the problem is that there was a lot of risk accumulating outside of what was being measured. I could see why there could be a model, but can we learn from their failure and how do you prevent the same failures happening with any kind of perimeter system you set up?
Kareem Ayoub
The lesson there is that it's hard, but that it's important and it's doable. I think the tech industry overall seems to be gravitating towards this, but also other segments of society are partnering more closely, which is becoming more important. The ability for you to understand how you make progress against providing guardrails around a system that is probabilistic or that may not be fully understood yet, which is understandable given the state of where the technology development is, becomes incredibly important. And so it's something that I think underscores the emphasis and the need to continue in this direction.
What I would say is that one of the observations I would have is that unlike previous technologies, so if you think of the industrial revolution, for example, you had a technology that was diffused to a relative few, and the burden was on that relative few to really transform existing industries and introduce new industries.
With AI, you actually have a technology that can be almost overnight diffused to nearly everyone. And if you believe that statement, there's kind of a beautiful insight that comes out of that, which is that the future of how this is all going to play out no longer becomes a forecasting challenge, it becomes a design problem. And so if you do believe that it is a design problem, it's something that we have to actively invest into to make sure we design the right outcomes that we want.
Timothy Wu
Well, let me talk a little bit about the incentives though to invest in safety. On the one hand, I think you're right, obviously it is very hard, very expensive, takes a lot of resources. On the other hand, there is competition ongoing between different models, different systems. People want to get technologies out there. If I'm not mistaken, Mark Zuckerberg had said he kind of released the agent a little earlier than he might have wanted to, but he felt there was a strong competitive pressure. Now, I think we usually think of competition as a good thing, but I guess, do you really think the industry by itself has the incentives to invest in the kind of safety you're talking about given the challenges facing it, or does it need a little bit of a nudge?
And obviously I'm talking about government now.
Kareem Ayoub
So it's going to require much more than just the industry. I think the industry itself has talked a lot about that, so that's nothing new. What I would say is that companies and governments and businesses are having to grapple with what are the ways in which I'm able to deploy this in a way that fits within the guardrails that I've designed and not? And it depends on where in the stack you are thinking about as well. If you look at the model layer, you are seeing all of the different labs across the industry have conversations around exactly the tensions that we just talked about earlier and the way in which models are being developed.
And that I would say is true, not just in the US, but also internationally in Europe as well as China, Southeast Asia, even the Middle East as well. And so there are very productive conversations happening there, and that's very much industry, government, as well as other sectors as well.
When you think about deployments at particularly the agentic and the services layer, you then have to start thinking about what are the ways in which you can express some of those guardrails that you actually care about, and how do you actually verifiably understand the way in which those guardrails interact with the model and then with the agent system itself around the guardrails that you've designed? And that conversation is probably one of the more important ones for us to have because that will govern our ability for us to actually benefit humanity. Part of our mission is build the technology to benefit humanity and deploying it into these extraordinarily sensitive verticals or areas, if we really want to turbocharge human ingenuity and use it as a force multiplier for our problem solving capability, we have to understand how to apply the correct guardrails in order to maximize that.
Timothy Wu
One of the, I think, tropes in the AI conversation, I think, and you've been in this as long as anyone, has been that growth is exponential and linear is going up all the time. We're going to get better and better, and then soon we'll be curing all diseases. Obviously, many amazing things have happened. I don't know quite how to ask this question, but in technological history, most technologies eventually start to flatten out at some point. If we were sitting around, and this was a hundred years ago, people would be talking about the airplane and how in no time flat we'll be flying to the moon or something like that, and because it's going up and up and up, and eventually we ran into different kind of physics problems.
Most technologies at some point start to flatten out. Now, in your experience, you're much closer to technology than maybe most people.
Do you see there starting to be limits if you were going to try to guess what would eventually be the factor or the physics that would start to flatten out the progress of AI? Or are we in some kind of different dimension because AI can improve itself or it's like a technology that is unlike any other before? But just on this idea of whether the pace of improvement starts to flatten out at some point.
Kareem Ayoub
Sure. I'm reminded of something that Demis, the now chair of Google DeepMind, said earlier this year at Google IO back in May. He said, "When we look back at this time, we'll probably realize that we were in the foothills of the singularity." I think what he was referring to was precisely this kind of incredible growth in the exponential capabilities that we are seeing. If you were to tell me even five years ago, particularly 10 years ago, that something like the transformer architecture created at Google released as a research paper and this kind of notion of next token prediction could work in the way that it is working today, that would've been very surprising, and it seems unreasonably effective as a technology because of that.
And the progress on the areas where you do see that brilliance has been incredible. Year on year, you're not seeing modest growth, you're seeing incredible progress on very technical, non-technical areas. Now, I'd say one of the important things to maybe realize that comes out of that as a consequence is that there might be capability overhangs associated with the state of the technology now. Have we had enough time to explore all of the potential that the technology is capable of doing today? And maybe not because the technology is developing so rapidly. Model releases are essentially coming, it seems like every single week, every month, whatever it is, from across the world, that the ability for us to have explored all the capability that is offered by any generation of model is always reset as we go to the next generation and understand what that next generation can also offer us.
So then the harder question becomes, what does a company or a business or a government or a person do with that exponential increase in capability and how does it benefit them on a day-to-day basis?
Timothy Wu
But to get back to the question, you don't see there's… I guess maybe you're saying there's something either you don't know or maybe there's something different about AI as a technology that unlike other technologies, or maybe just haven't started to see the curve start to bend. I guess I'm trying to pin you down on this a little bit.
Kareem Ayoub
That's fair. We just got done saying it's not a forecasting challenge. I like that though. we don't know in the sense that this is a technology we're continuing to invest in at the infrastructure layer, the model layer, the agent layer, services, and you're able to build exponential progress and make great progress and value at all of those layers. And so you're seeing compounding exponentials that as an industry, businesses and governments and companies are trying to focus on. And so we'll have to see how that plays out. But again, there's an active role for us to play.
That's the design challenge part that I was talking about earlier too.
Timothy Wu
You mentioned the singularity, which is a term that has been thrown around now for decades and a very interesting term. If I'm not mistaken, singularity refers to the point at which artificial intelligence exceeds the ability of human intelligence to understand it, if I've got that roughly correct. Do you personally believe there's such a thing as a singularity, a point in which things like that happen? And do you believe within DeepMind, is it widely taken as a goal of the company? you said we're in the foothills of the singularity to try and achieve that?
Kareem Ayoub
The goal stays the same. The goal is build AGI and use it to benefit humanity. And thankfully at Google, there's a pretty short distance between the technology development and path to impact. I'd say one thing, and this is related, one thing I've been thinking about more recently is the way in which we continue to strengthen the muscle of the mind as we approach that level of capability. And what becomes uniquely human that we have to invest into, not just for our own benefit, but for our colleagues and coworkers and just generally for society. And I don't know if this is right or the right way to think about it, but a few of the uniquely human things I've been thinking about that really we have to continue investing into are curiosity, empathy, and moral courage.
And what I mean by that is curiosity is important because you've got this technology, you can create abundant knowledge.
And so the question becomes, how do I ask the right questions or how do I explore the right spaces that will benefit people rather than all of the spaces? Empathy because the technology again can be very good at telling you what people will want, but to really build technology and steer it towards a future that benefits humanity, you also have to ask the question of what people need. And sometimes those two can diverge. And then moral courage, because again, you have a technology that is probabilistic. So every output, every token of text or code or image or soundbite or video is the daisy chain of probabilistic kind of linking.
And it outputs and you read it as a piece of text or a video or an image, but there is ambiguity there. And so you need to invest into the strength of moral courage to navigate through that gray zone in a world where that output does become probabilistic.
Timothy Wu
And what makes you think these three are particularly human? You know what I mean? I think it's a big question. In fact, it's a big question in psychology generally as to what makes humans different than animals. But is it just your intuition or does it come from working with the technology that makes you think these particular properties remain the province of humans in some special way? Or can you program AI to have moral courage, just to take one example?
Kareem Ayoub
Again, I go back to the future is not a forecasting challenge. It's a design problem. And if you take that to be true, we should understand what the technology is capable of doing. It's capable of producing this incredible knowledge abundantly. And in that future where you want to maximally benefit people and humanity with that context, you want to understand with that abundant knowledge, how do I ask the right questions? How do I focus on what people actually need and how do I navigate that ambiguity? Maybe it is programmable, maybe it's not programmable, but I'd say that's not actually the point.
The point is we should design in such a way that future that those three things do become valuable for us because that allows us to steer the benefit and impact we want to see.
Timothy Wu
Yes. No, that's a good point. You mentioned AGI and I just want to take advantage of the fact you've been in the AI industry I guess for 12 years and AGI in DeepMind among the firms I think has been most open about AGI has been our interest. We want to develop artificial general intelligence, which is defined in various ways, but let's say equivalent to human level, general level intelligence, you might define it differently. Over these 12 years you've been wrestling with this, has your understanding of that goal or what it might be changed at all?
Has it come to seem like a different kind of problem than you originally expected?
Kareem Ayoub
It has. So again, talking about AGI back in 2014, you'd get looks, but at the time you could really think of AGI or a generalized AI system as having many components that may emulate the biological human brain. And I'd say since 2017 when the transformer was released, the field has, whether rightly or wrongly, come to see AGI or these generalized systems as the ability for us to scale up and build out these kinds of generative AI models as well as agents that you see deployed every single day that the room here was raising their hands about.
And that was really surprising because if you think about something like AlphaGo or AlphaZero, which came after AlphaGo, you kind of also had to incorporate other systems into the technology. You had to incorporate things like planning or long-term memory or some sort of cognitive reasoning.
And so I would say that the notion of what does it take to reach generality and our understanding of it, and maybe this is just because it shows how little we knew about the human brain and still know about the human brain, has continued to evolve in terms of what we think is possible to drive towards that generality. And by the way, not generality for generality's sake, it's not AGI just because that is a certain threshold you want to achieve. It goes back to achieving a certain level of capability that allows us to ideally drive incredible good for the world, for people and for businesses and for governments because this is a technology unlike, if you believe the full upside, most other or any other technologies that we've developed as people.
Timothy Wu
So I'm sort of trying to process so you found the challenge, maybe I'm trying to capture exactly what you're saying about how the challenge has changed for you. It's more about developing more specific capabilities than you thought. Maybe I'm going to re-ask it exactly what you mean by that, how your understanding of generality has changed. Poke on that one idea.
Kareem Ayoub
Sure. All I was saying was artificial general intelligence used to be, or maybe it still is, this concept of bringing together different biological equivalents together or at least different technological components to build towards that generality. And the field has kind of converged in a certain direction on how it thinks about development of the technology and impact, which is fine, maybe it's not fine, but the point is that the notion of what it takes to achieve general capabilities across the natural sciences, the other parts of human knowledge that we care about were different than maybe what we thought five or 10 years ago, which is important from a development perspective as we think about what does it take to improve the jagged edge that we were talking about earlier of flash in the pan or incredible brilliance on the one hand and the ways in which companies or businesses or people are trying to use it on the other hand.
It all ties back to that level of generality and the broad capabilities that come from the tech.
Timothy Wu
Let's tackle one last topic. I know we're a little bit over our posted time, but we got some permission to go on a little further with your indulgence. So I'm interested in competition and entry into the market. You got in the market 2014, you were then at the startup. Now, first of all, when it comes to competition, particularly model competition, I think the understanding has been that the necessary inputs to be an effective model are talent, the best engineers and programmers and others, access to large amounts of compute and then access to training data. This is I think at least or other forms of data.
Do you think that's changing or has changed or are there other factors that I'm not mentioning that go into what are the inputs to be a competitive model at this stage?
Kareem Ayoub
I would say that the inputs have diversified greatly, and the evidence of that is that you are seeing incredible diversity and range in the kinds of companies that are being built at every single part. So from the infrastructure layer to the model layer. With models, you're seeing both proprietary and open models and different sizes of models, different kind of configurations of latency or cost or control that people care about. You're seeing it internationally dispersed across all parts of the world in a way that is exciting in the sense that you're seeing this multi-headed level of development.
At the agentic layer, you see different inputs even still from the perspective of companies or people are building their own tools and harnesses around the models or maybe around other agents to wrap around which exhibit and allow us to surface other kinds of capabilities. And then on top of that, yet you've got other layers of products and services that people are able to build, which again, encourage this incredible diversity.
And some of the benefits within the ecosystem come from, for example, the fact that a lot of this comes from open protocols. So you've got agents that are able to interact with different open standards and open protocols that allow a very tiny lavender grower in the south of France and Provence to work on Etsy and sell to these different agent infrastructures in ways that weren't possible before. So you're enabling this incredible diversity, this incredible ecosystem that people are able to build at different parts and enter at different parts, and there's this level of interoperability. And so that's being built on top of as we explore how you become successful.
Timothy Wu
And I take your point that there's a lot more room for different companies, different parts of the ecosystem. So maybe I'll just make this the last question. For the younger people getting out there now where you were when you were joining DeepMind, how are things, I guess, different? Obviously they're different, but for people who are seeking a career or interested in starting a startup in this space, how are things different, do you think, than they were in 2014 with the proviso, they're obviously a lot different.
Kareem Ayoub
Okay, we'll end with this. So I know it sounds a little bit cheesy, but the thing I would say is you have to become a practitioner of the technology regardless of if… I talk to students all the time and business leaders all the time. And when I say practitioner, I don't mean using the free version of a chat app. I don't mean just using a chatbot. I mean using the technology in such a way that it annoys you to start with and you work with it until it improves something that you actually care about in your day-to-day work.
And the reason that becomes so important is if you're a business leader and you're not a practitioner, you're trying to evaluate a portfolio of investments that you're making and trying to understand where you should double down, where you need to cut, and also steward your people through maybe one of the most consequential changes in certainly our generation, but maybe in human history.
And you need to do that authentically. And so my last thing, you must become a practitioner in a meaningful way in order to help steward that change, not just for yourself, but for the people around you that you impact as well.