What Is the Computer For Now?

Thorsten: We've been talking about, like, the end of local dev, the local environment is dead, and obviously we say, blah, blah, blah is dead every second day. But I'm looking at how I work. Like, so much stuff in these Orbs. That's stuff I would have bet money on it, like half a year ago, that this will be a local dev env thing. And now, you're kind of doing it with software on the spot and talking to the model at the same time in this other environment.

Thorsten: And I'm no longer just skeptical of local development. I'm now wondering, what is the role of the computer as a personal device in the future? I think it's wild, like I'm getting nauseous just thinking about like, what's happening to the computer, you know?

Quinn: Yeah. Well, the most enjoyable tool is the one that has just what you need and that the agent can drive. At the rate that I am using my computer or the distinguishing characteristics of my computer, if that continues, yeah.

Thorsten: People know this, I love dev tools, like, as much as anybody. And then, the first time I used the Cursor autocomplete, there was this moment of, ‘Oh, shit.’ Like, all of that stuff that I'm doing in Vim does not matter. Like, it does not matter how fast I can jump like the model will beat it—and then, everything that I did before looked like a black and white movie. And I guess what I'm wondering about is, is that what's happening to the computer?

Thorsten: Welcome to Raising an Agent, season two, episode four. We're back, we're excited, a lot of stuff is happening. Basically said, like, we've got to record now. GPT-6, Astra released. We got access, like, to four w—four days ago on Saturday. It's now Wednesday, my Wednesday when we record this, right? It feels like a lot of stuff is happening again, right? Like, it's exciting times.

Quinn: Right. And Fable 5.1, all the math advances. And every time there's a new model, I think it's partly because you know it's a new model and so you want to try more, and partly because the capabilities actually are greater than before, you learn what a longer leash looks like that you can give the models, what else that you have to do can you hand off to the models. And we've been discovering a lot of other things that we do, and you just always have to have your mindset to be, what can I not do?

Quinn: What can the agent do for me? And it's really taxing to always think that, but that's part of our job, and we've found some pretty cool stuff that we're doing now.

Thorsten: Yeah, I think that's actually—I think that's a good point. Like, every time a new model comes out, there's this reset where you go, ‘I've got to aim higher.’ And—and maybe you should have done this with the previous generation, but now, you know, my—my Astra moment on Saturday was, I got up super early, I don't know why, but I slept well, 6:00 a.m., dark out, I'm like—and I saw on our Slack we got Astra. ‘Let's go.’ And I fired up so many Orbs.

Thorsten: And I can't pinpoint to one thing where I'm like, ‘Oh, this was the Astra moment, here's the—this thing it did.’ And—and this was incredible. I can maybe say that it just felt like flying at such an high altitude that I didn't bump into any issues where I'm like, ‘Oh, you know, here it broke down, and I had to do this.’ Like, maybe the absence of stoppers, that's what makes this good. But to your point, like, what I did on Saturday, I had, in our main repo, I had Astra build something to be used by two other repos.

Thorsten: So these are all projects in Amp. I didn't touch any—I just never left Amp. Amp full-screen. I told it, ‘Here's the two other projects, here's this, build me this and make it be used by these two other projects which are deployed.’ And it just started spawning threads and Orbs in these other two projects, which fine, models can do this, like they've—they've been able to do this for like two, three months now in Amp. They can spawn other Orbs.

Thorsten: But it just went for like two hours, and you could see the messages flying back and forth, where like Astra would like think and say like, ‘Oh, I noticed that in the project, this. Let me blah, blah, blah.’ And it would send these messages, ‘Now I'm waiting for them to report back.’ And they would report back, and then it would say like, ‘Now roll this out.’ And then it—and I'm like, this is crazy! Like, this is high altitude.

Thorsten: Like, I just describe what I want, and then it just went and coordinated the work across multiple projects, which is, obviously, what the most AI-pilled people have been saying for two years now, right? Like, this is what we're doing, but it feels within reach now. Like, it feels it possible, you know?

Quinn: Yeah, it actually works. And for me, you might have gotten an email from me or from someone else in the Amp team when you've sent in feedback. And now, we make an effort to follow up to every single one with an email. And in the past, we'd try to do this manually, because, you know, agents have been good for a while, they'd be good at getting you maybe 90, 95% of the way there on the actual core work, but then not on following it through, not on monitoring feedback, on following up with the user, on—on all of that.

Quinn: And part of it is the new tools that we've given Amp ourselves. You know, the thread can archive itself, for example, but we've also given it tools to look at our GCloud logs, to look at the bug reports that have come in, to be able to send an email that we can review before it goes out. And so now, so much of what we're doing is not, ‘Oh, the work is done, let's push to main,’ which, you know, already was kind of radical to a lot of people, and then the thread is done.

Quinn: No, it's, ‘Let's push it to main. Let's have it monitor for when it's deployed.’ And we have a few different services, so it can even monitor is it deployed to this or that. And let's then, once it is usable, email the user that sent in the bug report, telling them that it's fixed now, and also give them some free credits as a thank you. And also, let's look through all of the other feedback we've gotten on Slack, on X, via email, via bug reports to see who else has requested this.

Quinn: Let's also look through our logs to see what users might have hit this. Maybe it was a bug, and we can tell those people, ‘Hey, I'm really sorry. We saw three days ago that you probably hit this bug, now it's fixed.’ Or maybe, you know, for a new Orb size that we just shipped, I looked through the logs to see who are the users that have been using the previously largest size of Orb, who, according to our logs, which is data that, you know, we keep on our side, were hitting out of memory conditions, and email them to tell them about this.

Quinn: And all of that, and then you have it monitor the feature and see is it producing errors that are unexpected, is it actually being used, is there any other interesting insights that you can give to me so that we can maybe do another few passes of feedback. And that means that the thread's lifetime is way longer, that so much more of that loop, and, you know, many loops, in a way, are happening in that thread. And that's something that the agent was not good at, and we didn't even really—we we could not do just a few weeks ago.

Thorsten: Yeah. I mean, oh, man, there's so much to unpack. But zooming in on one thing, I—I tweeted about this, which is you added this tool where we can now in a thread, we have like a Amp plug-in, so everybody can build this, we have an Amp plug-in that talks to our Amp server admin interface API, and with that, it can send emails to customers. Like, it can send emails to a user. So basically, we have this bug reporting tool, which also Amp can access, again, with a plug-in.

Thorsten: And I had this moment last week where Amp suggested, like, ‘Hey, we fixed the bug, should we tell the user about it?’ And—and we had a UI pop up where it had pre-written an email, and I could just hit the send or cancel button. And I was like, that's wild. And that's a nice experience. Like it—it—and the model fully gets it, like it understands we fixed a bug for this user, now I'm going to email the user, and it's a nice email, and that's wild.

Thorsten: To your point, like, I think, like, when people think you've got to aim higher, maybe they think like more complex stuff or like more, I don't know, hardcore engineering stuff or, you know, something like this, like maybe more engineering work, but there's another dimension here, which is that we—the whole lifecycle of a feature, including like conception, shipping, releasing, getting to users, bug fixing, incident response, whatever, like all of that can now happen in Orbs or in threads and can be done by a model.

Thorsten: And that's wild. I'm going to record a video. I actually—I—I typed a draft yesterday in the—in the Orb. It's still there in drafts. And then I was like, ‘I should record this.’ So sneak preview, I guess. I have a feature, I think Fable 5.1, I had it draft it, like the UI. Then I'm like, ‘Okay, let's get this out.’ But what I said was, ‘Wait a second, wrap this in a feature flag.’ Like, it's basically—you know that memory warning we have, where it's like, ‘Oh, your Orb uses too much memory.’ We've gotten some, you know, people said like, ‘Uh, that's not actionable.’ And I get it, like it's opaque, you know, it says like you use too much memory, but what does that mean?

Thorsten: So my feature is you can start a new Orb with a button, it's not automatic yet, kind of tricky, but also like you can ask Amp to investigate and some—like make it a little bit more actionable. And then add more data. So I had it build like the more actionable buttons. It's now sitting in an Orb. And what I'm about to send it when I hit record later is, ‘I want you to wrap this in a feature flag, ship it, then give me the feature flag,’ Amp has a plug-in to give users feature flags, ‘then give me the feature flag, then I want you to spawn two other Orbs that run into memory issues,’ which I know it will nail, ‘then I'm going to look at those, and then I'm going to investigate whether that feature is good or not.

Thorsten: And then I can say, yep, that feature is good, let's roll it out to more users. That at like, what is this? Like, this is like, I don't know, armchair feature. Like, I'm just, you know, pointing at things and say, ‘Make it like this and ship this and do this.’ And that is, again, like the altitude is so high, I feel like. It's—it's wild compared to like even two months ago.

Quinn: Yeah. Yeah, another—another example of that is now we, whenever we find that the agent or a model is not responding to something right in the system prompt, we will update the system prompt. But then Amp knows, because we've told it to, then go run a little mini eval. Go and come up with a bunch of synthetic scenarios of prompts where you would want this to be triggered or where you would not want this to be triggered, across all the different modes we have and other popular models that people use on Amp, and then see—and see where it missed it, and then iterate until it gets good.

Quinn: And that, you know, that is so much better than just YOLO putting stuff in a system prompt because, you know, then it just accumulates. And then we can also do the same for subtraction: What can we take out, like a Jenga game—

Thorsten: Yeah.

Quinn: —without affecting the outcomes that we want? And because—and this is where it all comes together—because we at Amp, obviously we all use Amp. Now look, we—we try other tools too because there's a lot to learn from them, but all of our intent, all of our history is in Amp, in threads, in something that is legible to the agent.

Thorsten: Yeah.

Quinn: You can also see, why was this added here? And you don't just get the commit message, you also get everything that the human said to get it in there. And you don't get that if you're a company that's using seven different agents or that switches or whatever. I understand why a lot of companies do that, that might be a reality, but you are going to lose out on agent power if you do not have all of that context accessible to the agent.

Thorsten: Yeah. And it happens much more than you would think, even than I would think. Like, I—you would think like, I don't know, like, ‘Here's a bug,’ and then you have to say, ‘Who did this?’ Or find out when this was added. But that's not what happens. If you see Amp in use in our repo, it will do this on its own. It kindly knows like how there's git blame and whatnot. I—I've—I don't think I've ever seen it use git blame, come to think of it.

Thorsten: But basically it will find threads and say like, ‘Oh, so the thing we're looking at, Quinn investigated this last week and he found out this and he shipped this, and he sent the user this email,’ because it's in that thread, ‘so now the user did—’ And—and then you're like, ‘Okay, so wait a second, so did the user not upgrade, or did the user, you know...’ And it's super valuable context, which is, again, like what are we doing?

Thorsten: Like, this is—the leverage is so high. And—

Quinn: Yeah. And increasingly you don't want to do things that are not in the sights of the agent.

Thorsten: Yeah.

Quinn: So, we found this with the shift to Orbs, when as soon as you started making the Orb environment good, that's all people ever wanted to use, and it wasn't just that Orbs were better, it was that local dev decayed way faster than you would expect because you were not maintaining it. And actually that took a ton of maintenance. And now, I think what we're seeing is when there's some tipping point when enough of what you do is done in the agent and is accessible, legible to the agent, then you feel a kind of anxiety in your head when you're doing something that you know the agent will never be able to figure out.

Thorsten: Yeah. I want to come back to something you said, like the—the mini evals. I think this is important for people to know. So, one of the holy—

Quinn: Yeah, I got—I got more examples of that, too. Actually, another one is performance, where we have the ability for even iOS and macOS to ask Amp to go and perform an action in our application, take a Safari or iOS profile, and it does that using Buildkite pre-flights, and then it can, you know, iteratively make performance improvements.

Thorsten: Yeah. I got—I got more examples. But one thing that I did yesterday, and I got so excited by, is—well, the last two days, I'm still working on it. But the holy grail: you want your codebase to be legible to agents. You want agents to be able to perform things in your codebase, right? You want to be able to have the agent run through the app end to end, or something like this. So, I added a new thing, which is production canaries.

Thorsten: So basically, it's really hard right now in a—in an Orb or local or whatever, basically not production, to kind of get a feel for how the performance will behave in production. So, um, I think, you know, this is years ago, I've done this in—in SQL databases. You connect to the production database, you open a transaction, you add a new index, you run the query, you see how it performs, and then you roll back the transaction. You can do this in Postgres, you know, you can create indexes in transactions and roll them back.

Thorsten: You want to use production as a testing bed. So what I did yesterday was I added something so that in the agents, when they make a change to a certain database, they should use this new tool to test how it would behave in—in production, using these safe things that I added. So the setup: I want my agents, when they make a change to the database, to use these things in production, so confirm that the performance actually goes up, and, you know, or it doesn't break, or whatever.

Thorsten: How do I do this? Like, you could say, ‘I'm going to ship this to main and then just see what happens over the next few days and what others do.’ But what I had Amp do was, I said in one Orb, ‘Okay, now we ship this, push this to main, and now I want you to spawn multiple other Orbs and give them a goal and see that they use that thing that we added without us telling them.’ And I didn't even spell it out like this.

Thorsten: I just said, ‘We need to make sure that, you know, new Orbs now use this,’ or something. So, what Amp did was, it created a bunch of Orbs, I think three in one eval run, and gave them different prompts, and it was so smart, it knew that it shouldn't spell out, you know, what they should do. It—it—it didn't put in the prompt like, ‘And make sure the performance is good,’ or, you know, something like this. So it just gave them three different tasks that would touch one of those database tables, and then they would—should report back on what they found.

Thorsten: But we knew it was like just an eval, so we're not interested in the results. And once they were done, the main Orb that I was developing in would then read the threads and check, did they actually use the tooling that we provided to them? And then it would figure out like, ‘Oh yeah, two out of three did this, but this one used this parameter wrong. Let's change this.’ And then it would start three other eval runs and make sure that those work.

Thorsten: So, that's first of all, that's fucking crazy. Second, you can use this flow to optimize your codebase for agents. Like, you can say, every time we develop a new feature, I want the agent to run through this end to end in a browser. And you can ask like one Orb and say, ‘Hey, this is the—what I want. Let's fix the tooling, and then start Orbs and monitor them, give them a little feature to develop, maybe even the same feature, and have them monitor it, and then it will run in a loop.’ And it's—it's wild.

Thorsten: Like, that's—it—it's crazy to see. And again, like, I think that's the holy grail. Like, you want your agents to do this stuff naturally. And with Orbs and automated Orbs and agent-to-agent communication, it's really good. Like, it's really good to see.

Quinn: Yeah. And all of this stuff is hard if you don't—if all you do is just try to use agents more. You need to change stuff about how you work and how you ship. If you have mandatory code review in a pull request, then if every single gate requires somebody else to look at it, well, that person—I mean, these are not full atomic changes in the way that you used to have really nice, clean commits. They're a lot more exploratory and it's going to slow you down.

Quinn: And if you don't have a team where you trust people end to end to take in customer feedback, to be the responder, to follow up directly, if people are not able to do that, then you cannot get the benefits. So, so much about the company needs to change. And that's part of what we are doing at Amp with our team. We have 20 people total, and we are trying to live in the future and—and make Amp the company that is able to benefit most from these agents, so we can get out ahead and find what works and what doesn't and build a product around that.

Quinn: But, to address the elephant in the room, well, we've had some outages, and we take this really seriously. For us to show here's how agents can be really good, we need to make our own house in order, too. And we've had downtime. If you look at ampcodestatus.com, you'll see, you know, a bunch of incidents in the last five, seven days in particular. And that's not okay. We are—we know that you all depend on Amp being up, and for us to earn that right to be the agent that you use and that your whole team uses, because we just talked about how important it is to have a—a single agent for your whole team, we know that Amp needs to be really highly available.

Quinn: So, wanted to share more of what we are seeing, what we are doing, and some interesting thoughts we have. But, you know, for anyone affected, I just want to say I'm sorry, and we are sorry, and we are doing a lot of things to get better. Um, at the same time, the answer is not to slow down. One, because you—what you expect from Amp is to explore the—the future and to always be pushing forward. But two, because actually there are so many ways that agents will help you make a more stable, more reliable, higher quality service.

Quinn: And we're seeing so many of those and we're already using a lot of those, uh, things. So we don't want to just say let's just stop and, you know, as they used to do in the past, like Slack, the team chat app, had a year-long reliability year where, I mean, as it turns out, as I've heard from people there, they not only did not actually make the product more reliable, but they also, you know, shipped almost nothing new.

Quinn: So we want to figure out how we can still be the product and build the product that you all want and increase our reliability and make it so that you can depend on us more.

Thorsten: Yeah, I can start with some numbers, right? I think I pulled them up on Monday. I think in the last six weeks, this is when growth took off, I would say. So that's the big ingredient here, right? Like, there's growth, people love the Orbs. We're a small team, shouldn't be an excuse, we have infinite agents.

Quinn: Yeah, none of this are excuses, so these are all reasons we're sharing behind the curtains.

Thorsten: In the last six weeks, we had, depending on how you count, I would say 18 to 20 incidents. And obviously we have to say we do love opening incident channels, like we're very eager to just put stuff on the status page. Like, there's stuff where you could argue, ah, it was like a 20-minute blip, you know, let's not put it on the status page, but I think we're pretty honest about it. And I went through all of the incidents to analyze this, um, because obviously, like, we don't like having incidents.

Thorsten: So I think there's three buckets. The smallest bucket, um, of incidents, like less than one-third, is upstream provider issues. Like, you know, we've had—when was it? Last week, I think, where like, uh, OpenAI had some issues, OpenAI wasn't available, so we put that on our status page: upstream provider is having issues. Then our sandbox provider had some issues where they couldn't give us new sandboxes. Then we had an issue—this was maybe, let's dig into this. So at the start of August, we realized that we need to—like, our infrastructure is going to have to scale up.

Thorsten: We had more and more data, we knew there's more and more data coming, so at the same time we wanted to compact the data and compress the data without taking production down, without pausing anything. That turned out to be like a challenge, um, and we've had multiple attempts. That caused some outages at the—at the start, where we turned it on, it put too much load on the system, we had to turn it off, the system recovered.

Thorsten: With all that, there's never any data loss, it's always like the system recovers, and we had a bunch of these issues at the start of August. Then last week Monday, in a meeting, we were like, okay, we made a lot of progress, blah, blah, blah, but now the next step is we need to add more redundancy, like we need to add more storage nodes. This, you know, we have redundancy, it's good, but we need to add more so we have more headroom.

Thorsten: And the very next day, this was some people in—using US Central in GCP, they had an issue that killed one of our storage nodes. Our system recovered and handled it fine, but our system basically prioritized no data loss above everything else, above availability, which is, I think, a good thing you want to have in a storage system, and we can make this better, but basically our system was like, ‘Oh yeah, I lost a disk, I'm spinning up a new node.’ We got a new node, but then to keep redundancy, you have to move the data, right?

Thorsten: So it's always in multiple places. And that took a couple of hours for us to recover from, even though all we did was basically watch the thing move data around and watch the graph and tell users, ‘Sorry guys, like we're still moving data.’ It was embarrassing to sit there, not be able to do anything. But the point I'm making is it's a complicated system, and at the start of August I think that would have gone completely differently, and now we recover nicely from this.

Thorsten: And then yesterday we had a blip where, I mean, everybody who's done any ops or infra ever can tell a story about autoscaling, where some autoscaling basically said like, ‘Oh, the collector nodes that collect the metrics for Prometheus, they have memory pressure, so let's get more of them.’ And it kind of kicked off storage nodes, because by default on GKE, I think the priority of these collector pods for Prometheus has a priority of one billion, which is the max value allowed, so they kicked off a storage node.

Thorsten: And we were back online in 20 minutes, and then we were just monitoring the whole thing. So we went from, ‘Oh shit, what's happening?’ start of August, to last week, ‘Hey, we got to do, you know, more redundancy,’ to Google killing our disk, we recovered, but we were out, to yesterday, 20-minute blip and recovering nicely. That's the one bucket, like the infrastructure bucket. The other bucket is—I mean, I wish I could—I mean, we should do this.

Thorsten: Like, if we—if we clip this on the podcast, we should show the graph. Our commit rate tripled, I think, if I—if I remembered correctly, from May, June to August. Like, we're shipping so much. It's—I don't want to say it's a wonder that we only—like nothing broke, but basically, I think we had seven incidents where we shipped broken code. Like, you know, like a query, a database query was added and then in production it blew up because it put too much load on the database.

Thorsten: And that was kind of most of it, like most of them were database queries. I shipped a change where I upgraded the Git version in the Orb base image, and I assumed that was a safe thing to do. My fault. It broke, like, how to sign commits. So that was an incident where commit signing was broken. You could use Orbs, you could use agents, you could do anything you want, except in Orbs, you couldn't sign commits.

Thorsten: So that's an incident. And for all of these, we're pretty fast to recover. Like, we, you know, like I think the one thing that I'm thinking of now, Quinn, where you were like, ‘Oh yeah, I did—I think you shipped one query,’ um, we rolled back in like 20, 25 minutes or something. Like—

Quinn: Yeah. I mean, there's some less. And so, you know, you might look at how we're using agents, and everyone knows that we let our—our team push to main. And you might actually think that that is the main source of the outages. Actually, if you look at even the count, it's about a third, but if you look at the minutes of downtime and the number of users impacted, it's far less. So much of it is coming from this infrastructure that we have.

Quinn: And if you think about what is Amp, it's three components that are each totally new. You have the inference, and that is changing incredibly fast, it is scaling incredibly fast, and, uh, then you have the durable execution for the agents. So the thing where your agent can be thinking, you can shut your laptop, or, you know, if you're in an Orb, the Orb could even die, but the agent loop continues running. That is another new kind of component that's, you know, not completely brand new, but it's a different paradigm from how people usually run stuff.

Quinn: And then you have the sandboxes. You know, all of those are new pieces of technology where they're all scaling fast, they're all changing fast, and the three of them together, it creates an incredible experience, and it is the future, but there's a lot to figure out there. And so, you know, over the weekend we were talking as a team, and we had this moment when we thought, is this something where we just have to have a lot more redundancy?

Quinn: But actually, there are really big compromises if you have redundancy, if you start to say, well, let's build for the lowest common denominator. For example, for sandboxes, every single different sandbox provider has a different set of features. Some can do memory snapshots, some can't, some can't even be resumed. You know, they're all just built with different decisions, and I think we're all going to figure out in this industry what the right set of decisions is. You know, maybe there's actually four different products here.

Quinn: But if we were to say, well, sometimes your Orb can spin up on this provider, but then if that's down, it can spin up over here, then it creates a whole set of problems. Like, maybe you have to just accept a—a worse, slower, less featureful product. And also, if the provider where you spun up the Orb is no longer online, then, you know, we have to figure all those things out, and it increases the complexity of the system.

Quinn: So there's a lot of hard decisions that we have to make here, and the answer is actually not just go and add redundancy.

Thorsten: Yeah. I would add that I see it as our job to go as fast as possible and to use these tools in a way nobody's dared to use them before. Like, I would say the job of every engineering leader is you need to figure out, like, how fast you can go before stuff starts to fall apart. Like, that's—that's the job. Like, you—going slow and safe all the time, that's relatively easy. Just do nothing. Like, that's the default mode, right?

Thorsten: Like, you—this came out, like, what was it, 10 years ago, where Meta published, ‘Oh, over Christmas the number of incidents goes down.’ Yeah, it turns out because there's nobody pushing code. Turns out humans are causing the incidents. Like, it's not the system itself. So doing nothing is the safest option. So I think as an engineering leader, you should try and aim to go as fast as possible right up to the point where stuff is going to fall apart, and you need to calibrate and figure out, like, where that sweet spot is.

Thorsten: But calibration means you sometimes go over the line, right? Otherwise you're not calibrating. So that I see us doing. Like, that is our job. And we're up there on the frontier trying to do all of this, and that means we're getting the flies in the face, right? Like, we're getting the dust in the eyes. That's—that's what we do. And I want to say, like, we're doing this so nobody else has to do it, but that also comes at like a certain, you know, sometimes stuff breaks, which we hate.

Thorsten: And if you look at our incident channels or anything where anything breaks, the first thing is always: what customer is affected? What users are affected? Tell them about it. Is it on the status page? Do we communicate this? We need to credit them back, send out an email, collect all of the users—we're doing this in Orbs. Like collect all of the users, give them $10 in credits, $20 in credits, send them an apology. Like we know that an incident is a fucking embarrassing and awkward thing, and I wish we didn't have one, but when I look at what we're doing and what we want to do, it also seems silly to say there's never ever going to be an incident.

Thorsten: You know, like there's a certain thing where you know sometimes we're going to overstep that line maybe. Involuntarily, like ideally we're not going to overstep it, but, you know, like sometimes it happens.

Quinn: Yeah.

Thorsten: I don't know if we should have said that on record, but I think that's how I see it. Like if—look, like we're shipping all of the features, everything from Orbs. We're doing incident response in Orbs. It works incredibly well. I don't think I've ever worked on a system that sent around so much data as Amp. Like on a daily or minute-by-minute basis. Like people—I don't know if people realize, it's like every conversation with all of the tool calls, all of that, like it's, it's a lot of data.

Thorsten: Like it's much more than any web application ever sends across the wire, you know? Like it's a lot of data. And that whole—it's a complex, big system, and we're a relatively small team for that size, and I think we basically, two years ago we couldn't have built a system like this. And now we do, and we're learning it, and we're getting better at it, and I hope that soon we will have no incidents. Yeah, we got to go as fast as we, as we can.

Quinn: Yeah, we want to figure out the patterns for how you can have a really reliable product that

Thorsten: Yeah.

Quinn: uses AI to become more reliable, not to, you know, say, ‘Okay, we've reached a high degree of popularity, let's slow down.’ Because, you know, frankly, you look in the rearview mirror, and there's a lot of AI products that did that, and now we've forgotten them. And it's actually really valuable if we can figure out and share with all of you patterns on how you can use an agent to not just build the thing, but also make the thing robust, and, and, and maintain it, and, and ship it, and, and have the agent do more of this.

Quinn: So...

Thorsten: Yeah. Yeah, we're going to share what we did. Like there's maybe not that many people who already use agents in like incident response. Like I think people may dream of this, but maybe they're not allowed to do it, or whatever, but basically one of the things—

Quinn: Well, in most companies, their, their agents are not able to read their logs yet,

Thorsten: Yeah, yeah, yeah. That's what I'm saying.

Quinn: or traces.

Thorsten: Yeah. And so, but, but basically one thing you're going to bump into once your agents can read the logs is—and this is a, has been a human problem, but the solutions are different. So if you have an incident and you ask the agent, ‘What the hell is going on? How does our cluster look?’ like every false—I guess, I don't know, let's call it false positive. Like every error log or every exception that shouldn't be an error log or shouldn't be an exception will confuse your agent.

Thorsten: Like it will go, ‘Oh, I see!’ Like say an infra provider goes down sporadically. We don't know what's going on. So there's like, we know there's a cause. Then we point the agent at our production logs, like what is, what exactly is going on? If there's other stuff flying around while the rest of the system is health—healthy, but say it says like, ‘Storage nodes at capacity,’ you know, whatever, like anything like this, then it will go and think, ‘Storage nodes are at capacity.’ It's just like any other—like a human you would put in front of the logs, right, without any other context.

Thorsten: So one thing we're aiming towards right now this week is, first of all, get rid of all of the logs that are not actionable and that are not actually errors, and everybody who's ever logged anything knows that this is a common thing that happens. So we're going to get rid of those, we're going to push towards them. And then the other thing we want is the ability for an agent to see the same graphs and metrics that we see, so that if I say—and right now everything is healthy, you know, knock on wood—if I say, ‘Is the system healthy?’ the agent should be able to say, ‘Yes,’ if it's healthy, and if it's not healthy, it should be able to say, ‘It's actually not healthy.’ Like that is a good benchmark.

Thorsten: So that's what we're working on. And one way to do it is like you build custom tooling with Amp plugins that have access to different metrics, and you point it at the different metrics and explain the system. You make the system legible to the agent so it knows kind of what the different parts are, right? It's—in a normal web app, obviously the agent knows that the database is important and Redis is used for a cache and whatnot, but if you have a slightly more complex setup, you need to kind of explain how stuff fits together.

Thorsten: So that's one thing we're doing this week. The other thing that, again, I mentioned this before, is I'm getting the agent more feelers into production in a safe way so that it can kind of already anticipate what stuff will look like in production, which I think is very important. So, yeah, we need to share more of this, and I think it's, it's very interesting. And I mean, this is given in this conversation, like I think we've been saying this for a while now, if, if you haven't used an agent to do ops, prepare.

Thorsten: Like this is mind-blowing. Like you should definitely do this. Like it's—if you've ever used like kubectl by hand, and then you ask an agent, ‘Can you use kubectl and look at this cluster and maybe check all of the’—you know, whatever—it's incredible. Like it's so good. Like they're so good at this stuff. It's, it's incredible. Like you should definitely try it.

Quinn: Yeah. What I would recommend is if someone hasn't, then probably the first place to do it is not in an active incident. So find a way, using like OIDC, we published in our news posts how to do this, for it to get access to your logs and have it go and look for exceptions, errors that look bad, and have it go fix them, and then have it go confirm after it's pushed that those have gone away.

Quinn: And that will help you punch the right holes in your infra for it to get access to it. And again, this also comes back to these two things we've talked about where if your whole team is using a single agent, whether it's Amp or something else, then you can be sure that everyone is going to have the same ability to access these production resources. But if you're on a bunch of different agents, that's a configuration nightmare.

Quinn: And then also, in an incident, that's when you need the fastest access to, ‘Why did this person do it this way?’ If they're not in the war room with you, that becomes tough. So there's all these other things you have to do.

Thorsten: Quinn, MCP, that's the solution. A centralized MCP server.

Quinn: Yeah. You know, there are, there are ways to set up MCP so that it works across all of your agents, and then you also got to make sure that it stays good and all of that. I, I—we haven't seen it done well. We haven't seen an MCP configuration that is multi-agent done to the level of quality that you would get if you're using a single agent and you just make sure that that's really good.

Thorsten: Yeah. All right, should we switch gears? I want to talk about something a little bit more philosophical. Maybe, maybe abstract, but okay, so here's the pitch. It's—I'm, I'm not saying this in a nihilistic—maybe I am, but I don't think it's nihilistic. But we've been talking about like the end of local dev, the local environment is dead, and obviously we say blah blah blah is dead every second day. But I'm looking at how I work, and I'm in Amp all day, every day, like full screen, and we talked about this, right?

Thorsten: It's, it's kind of addicting. Like, you know, like people have been saying this a lot, obviously, but I'm in this and I'm replying to user emails, I'm monitoring production, I'm rolling stuff out, I'm doing like so much stuff in these Orbs. I'm, I'm a lot, like, you know, there's nothing I don't—that I do locally. So that is one bit of context. Then you look at what Brett is doing, and in case you haven't, check out our new docs page.

Thorsten: I added something yesterday where we document how people work with Amp. And Brett, he builds like these crazy tools in Orbs that he uses to edit videos, for example, or to create thumbnails. Like this was the newest thing. For the last episode of this podcast, we have like these thumbnails, and obviously if you want to create thumbnails, you go through the video frame by frame or you find some frames and you pick, ‘This is a good picture of Quinn, this is a good picture of Thorsten, I'm going to put them side-by-side, I'm going to put some writing over it.’ And I've done this, and the way I've done it is like an idiot, I would open the video in QuickTime and I would scrub back and forth and then it's like, ‘Oh yeah, he's laughing here.

Thorsten: Buh buh buh buh buh.’ Use the arrow keys, find a frame, screenshot, like it's all pretty gnarly, right? That's how I did it. And then we have a Figma file, you paste it in, you move it around. What Brett did was he built in the Orb like a tool to kind of let him select frames, and when he hit like return, it would already put it in the template for the thumbnail, so he could have a live preview of what it would look like in the thumbnail.

Thorsten: The point I'm trying to make here is not Orbs are amazing—they are—and, you know, building software on the spot and JIT software and liquid software, that's amazing, it is. Point I'm trying to make is that's stuff I would have bet money on it, like half a year ago, that this will be a local dev env thing that you do locally, where you're like, ‘Oh, I got to use my video editor and QuickTime and have like different applications and blah blah blah,’ and, you know, like that kind of stuff.

Thorsten: And

Quinn: Yeah, it's like you could imagine, oh, that's a Computer Use with a Screen Studio or something like that.

Thorsten: Yeah, exactly like this, or like it's—I think like it might be the perfect, you know, like an Apple commercial video where it's like, ‘Hey, here's the new MacBook for creative professionals.’ It shows Brett with like the resizing the thumbnails, you know, like you're using your Mac, you're doing stuff on your computer. And now, you're kind of doing it with software on the spot and talking to the model at the same time in this other environment, and what I want to say is, I'm no longer just skeptical of local development.

Thorsten: I'm now wondering, what is the role of the computer as a personal device in the future? And obviously, there's been people in suits using iPads with like the keyboard attached for many years, you know, as their computer. You see them on planes, you see them on trains, and whatnot. And as a developer, you're like, ‘Nah, like I'm not, you know, I need like my local shell, or like, like my terminal, I need my keybindings, and all of this.’ And I think there was an argument years ago where somebody would say, ‘You can do anything you can do with your Vim stuff, you can do it in IntelliJ or VS Code, doesn't matter.’ But the argument I would always make is, ‘Well, you need to enjoy your tools, because if you enjoy your tools, you're going to use them more and you're more productive and whatnot.

Thorsten: Like the keybinds matter.’ But now I'm like, does this, does any of this still matter? Like are we all going to iPads now? And I, I, I, I think it's wild. Like I'm getting nauseous just thinking about like, what's happening to the computer, you know?

Quinn: Yeah. Well, the most enjoyable tool is the one that has just what you need, and that the agent can drive.

Thorsten: Yeah.

Thorsten: Look at—I mean, people are getting sick of it, I guess, but like Astra with, um, using Blender, headless Blender, like everybody's building 3D stuff and people are tweeting, ‘Hey,’ like there was one person who tweeted, ‘I'm a Blender artist, and yes, I was one of the people saying, “Well, they're coming for coding, but they're never coming for me.”’ You know, like, ‘There's—the models can't do this.’ But it turns out, like, wow, like it can do a lot of stuff.

Thorsten: Like you just give it textures and outlines and photos, and it builds like 3D things. So

Quinn: Shout-out to my seven-year-old nephew, who used Amp to make a Minecraft mod that makes our whole town—you know, we live in the same town—in Minecraft. And now, I was just at back-to-school night, and all the parents have heard about that. So I mean, this stuff, it spreads.

Thorsten: What did he use? Did he use Google Maps as like the input, or what did he use? Like how did he get the town into—

Quinn: Uh, it was OpenStreetMap and like USGS, US—you know, geographical something public data. And of course, uh, Astra refused to use Google Maps and Apple Maps, 'cause that would be against the terms of service.

Thorsten: Of course.

Quinn: But then it said—and this is really interesting—it said, ‘I can build you an application where you can go to the various places in town and take a panoramic photo, and upload them for me, and that will help me get much better textures.’

Thorsten: Oh, wow.

Quinn: So that's what he's going to do with my brother, his dad.

Thorsten: That's an—what a nice kids' activity!

Quinn: Yeah.

Thorsten: Like, isn't this nice? Like, I mean, ‘I'm the data collector for my agent,’ you know? ‘Let me go out and do this.’ Yeah. I mean, I think if I were to do this, it would blow my daughter's mind and like my, my friend's son. Like if I show up with a Minecraft map that looks our street here because they live down the street, I think it would blow their minds. I don't know. What do you think?

Thorsten: What's the role of the computer in the future? Do you, do you think like in, in two years, we're going to use macOS and we care about the, the dock and I, I don't know, like what do you think?

Quinn: I am really torn. I have an iPad Pro with a keyboard that opens up, and that's amazing. And I have a MacBook Pro as well. And I usually put it in my backpack to go places, but it's rare that I actually open it up. And when I'm on the go, I just use my iPad Pro. And with Portals and just using the agent, you can get so much done now. And also, you know, on, on the phone, and when you have voice on the phone, when you have an idea on the go, it—yeah.

Quinn: And I think that if I could get a large-screen MacBook Air, that would probably be best. I do like the function keys 'cause I have them mapped, but these days I'm really only using Amp, which is F2, and Chrome, which is F5, and uh, Slack, which is F8, the pause key, because, you know, don't feel as productive in that. So at the rate that I am using my computer, or the distinguishing characteristics of my computer, if that continues, by Christmas, yeah, or maybe something else to get pixels in my eyes, or to take input from my brain via dictation that doesn't need to be a computer, because I'm doing so much less intricate stuff, and I don't—today, yes, I do need a mouse and all that stuff, but I could imagine a future where if I could just project my Portal into my retina or on the wall or something,

Thorsten: Yeah.

Quinn: it's, it's just such a change. And none of this could have been predicted a year ago.

Thorsten: Yeah. I mean, little pause here, little thing here. You're the only person I know who uses F-keys as religiously as you do. Like I don't know if you know this, like every time, like I—if I go through Slack and it mentions F-keys, like I think it's you involved in that thread. And every time I see a keyboard shortcut that uses the function keys, I'm like, ‘Quinn was here.’ And it's, it's a hundred percent hit rate. Like I don't use them.

Quinn: What are you cavemen using? Like Alt+Tab, clicking on a dock?

Thorsten: I don't—yeah, I don't know what I use. I don't know. I don't even know, like I have the default pause and, and brighter and, and whatnot. I, I think I would have to hit the function key. I don't know what the F-keys do, actually. Like I've never used them. But—

Quinn: It's like prime real estate for your fingers.

Thorsten: Might, might be. Might be. To your point, I—what I've been—you've been on the dictation game, but, um, what I've started doing is, so when I'm in the gym in the morning, I have a bunch of ideas of stuff that I want to do, and I'm, I'm too embarrassed to talk to my phone in, in English in the gym. It's—can't do this. Like it, it's just too much. I'm not getting over this hurdle.

Quinn: 'Cause in, in case the listeners don't realize, you're in Germany.

Thorsten: Yeah, I, I live in Germany. Yeah, and I, I'm, I'm, I'm—I don't know what image I'm trying to keep in that gym, like in the mornings when it's me and four other guys, but like whispering in English into my phone, I don't know, man. Like I can't pull it off.

Quinn: They're going to think you're a spy or something.

Thorsten: Yeah. So, but then, when I walk back, it's like a 10-minute walk, and then I walk back, and then I'm walking on the street, and I'm talking to my phone, like as if I'm holding it up to my face. And I sound like, ‘I want you to check on this and blah, blah, blah, and then let me know.’ Boop! And I send this. ‘Now, I want you to do this.’ Boop! And then I come back, and then I have breakfast and whatnot, and then I sit down and I see like the threads running, and I check on their results.

Thorsten: And it's, it's wild. Like it, it's just feels crazy. I went for a run, and I was pacing around the garden and just talking to my phone. My wife was like, ‘Who are you talking to, like this? Like, what are—are you on the phone?’ I'm like, ‘No, I'm starting Orbs here.’ But yeah, it's just wild times. Why—like what, what's happening to computer? It's—to round this off, like I think two years ago, I, I wrote this on my newsletter that people know this, I love dev tools.

Thorsten: I was this close many, many times to have Vim tattooed somewhere on my body. I love the keyboard shortcuts. I love all of this. Like, as much as anybody, I riced out my shell config before ‘ricing’ was even a term or whatever. Like, you know, like, I've been there. Like, I've done it. And then, the first time I used the Cursor autocomplete, this was, you know, like truly, I think, end of—towards the end of '24, there was this moment of, ‘Oh, shit.’ Like, all of that stuff that I'm doing in Vim does not matter.

Thorsten: Like it does not matter how fast I can jump or quick-scope or setting marks, like all of that, it does not matter. Like the model will beat it—and then, the trajectory of how these models are getting better and faster and you could just see like, okay, this is, this—everything that I did before looked like a black-and-white movie. And I guess the point I'm trying to make now is, or what I'm wondering about is, is that what's happening to the computer, like the, the, the thing that where I install software and I care about my environment, and I have like my menu bar, like is that going to feel like a black-and-white movie in the future, and we're not actually using these applications anymore?

Thorsten: And who knows? But—

Quinn: I think so. I mean, you could say that the pre-web experience, or at least, you know, before the web had applications that you used, that—I mean, we've already gone halfway there.

Thorsten: Yeah. The only thing left will be using your finger to hit the Touch ID for the passkeys. You know, that will be the thing. That's how they—and then it's just, ‘Neuralink activated. Please press unlock 1Password. You want to use this? Use your passkey. Press again.’ And it's just this all day long, one finger.

Quinn: Yeah. Yeah. Thousand times passkey authing per day.

Quinn: But that thing you said about, you know, getting away from Vim and just seeing that once you saw autocomplete, that just changed, and, you know, that, that progress, I think that pretty much everyone has opened their minds a lot to change. And just want to acknowledge it's still hard to always take a trajectory and extrapolate it. And just to be really concrete, we saw that when we shifted the default in Amp from queuing your messages, if you send a message while the agent is working, it used to queue it.

Quinn: So it'd wait to give that message to the agent until the agent was completely done. And now the default is that it does what we call steer. It'll wait for the, the next available opportunity, like when the agent is done listing something, and then it'll inject it. And that's the kind of thing where I think you and I are about 100% certain that no one will even think about queuing in six months. And there's a lot of people that say, ‘Look, steering doesn't work, I need queuing, I have this intricate workflow.’ We do give you the ability now, which we didn't in the past, I think we've changed here, to go, go back to the default so that it queues by default.

Quinn: But that's the kind of stuff where we feel like we're doing you a disservice by like letting you do that or letting you stay on that for that long. And you, you know, human nature, we have it too, is to want to cling on to something. But if you're using queuing, then the agent is going to be slower and it's going to have less input from your brain when it's doing its work. So, you know, in everything we do, this is just a, a plea to everyone using Amp to just, you know, get your mind more and more into this my—mode of extrapolating change.

Quinn: And you all are there so much better. I mean, we have the best users out there. But that's where we are and that's where we're constantly having to push ourselves. It's never easy, but we appreciate you so much when you get in that mode with us, 'cause we're in a golden age.

Thorsten: Yeah. There were two tweets yesterday where one person was saying something like, ‘Oh, they don't plan anymore,’ or planning is, you know, they don't use the plan files or whatever, the newest generation of models, Fable 5.1 and, and, and GPT-6. And then he ended the tweet by saying, ‘Time to change workflows again.’ And then I think Kenton Varda from Cloudflare, he tweeted something like, ‘The two models, they blah blah blah,’ and he's like, ‘Yeah, it would be good to pause so I don't have to rethink this stuff every two months, you know?

Thorsten: Like just, I don't want to change my workflows, I want to get stuff done.’ And that's a very real thing, like what both of them are pointing towards. And I didn't, but I was this close to saying, ‘Yeah, well, if you use Amp, you don't have to think about it. Like just trust us, like we'd—we're doing this for you. Like, we've never had a planning mode. Um, we kind of added compaction when people were still, yeah, you know, manual context management.

Thorsten: We make a lot of these calls because we think they now work better and you don't have to do this other stuff. And so if you use Amp, you can be—yeah, there might be an update where you're like, where did they move the button? But, you know, trust us that we put like some thought into should we make this a default or not and that we kind of try and always go where the puck is going.’ Yeah.

Thorsten: Puck.

Quinn: That's right. All right, well, yeah, I just want to end it with we are in a golden age, software development and just technology is changing so fast and it's so exciting and it is such a joy to be getting to build this with all of you and getting all your feedback, so please keep it coming.

Thorsten: Yeah. It's a golden age, it's Orbin' time.

Quinn: It's Orbin' time.