Producing Raising an Agent in Orbs

Hi, I'm Brett from the Amp team, and I just wanted to record a video chronicling some of the workflows and processes that I've been exploring using Amp for video production. Over the last month or two, Amp has increasingly taken on more and more of the responsibilities for video production, and it's fascinating for me to sort of watch my workflow go from juggling a bunch of different apps, like Final Cut and After Effects, to doing more of it in-house with Amp.

So I have a note that I've been working on for about a month now called Orb to (Post) Production, chronicling how these things have changed. And I really want to get this out, but I keep having to rewrite it because the process is changing day by day. So I have this VidGen repo here that I've been working on, and it started off easy enough with a lot of very small utilities, and it kind of escalated from there.

So let me show you how it works. I'm going to just start a new thread and say hello. The orb is where I've been doing most of the video production work in a format that is also kind of agent-friendly, that allows Amp itself to also be able to iterate on it with me. So you can see here, I've run the orb. I have my Amp session on the left, and then on the right, I have this portal into the orb, which has a bunch of purpose-built utilities that I've been working on.

What I think's really interesting about what I've been doing with Amp is I basically took the parts of the workflow that I needed from a tool like Final Cut and Adobe After Effects and smushed them into a piece of malleable software that basically works the way I need to work. So when recording Raising an Agent, we have Quinn and Thorsten record their separate streams, and I find that manually managing three different tracks is kind of annoying and error-prone, so I like to use the multicam features in non-linear editors.

Over the course of many prompts with Amp, I built this entirely purpose-built editor here. What it allows me to do is it allows me to open the videos with Quinn and Thorsten. It's using Mediabunny to do local media processing. So in this case, it's providing me one track for the edit. And what's really nice about this is it immediately creates a split-screen composition. I was going into After Effects to mask two videos so that I could have three time-synchronized angles: Quinn, Thorsten, and the split-screen angle. I added some controls to like make sure that the speakers were sort of centered up in it. Now if I play it back, I can hit two, and it will cut to angle two, which is Thorsten, to continue talking.

When Amp helped me build this software, it added features I didn't even know I wanted. So in Final Cut, when you're editing a multicam sequence, you only see an audio track representing the currently active audio angle. Amp, when it built this editor for me, it shows each of the individual audio tracks separately, which like, such a small thing, but like, you know, because I eschewed multi-track editing in favor of multicam editing in Final Cut, I gave up the visualization of seeing when a different speaker was talking. But when Amp built me this version, without me prompting, added a thing that lets me see like, 'Oh, Quinn's about to interject, let me switch angles here,' that sort of thing.

It has like, you know, good keyboard hotkey support. It has a lot of the things I want from Final Cut, so I can use up and down on the arrow keys to select clips, I can use X to select the current clip, I can even, you know, use arrow keys, and if I press X, I can use the square brackets to trim the in and out points, so I can really precisely edit down the footage as if I was using like a non-linear editor on the caliber of, you know, a Final Cut.

I'm going to open the project here, and we've got like, I think it's probably this one, v38 sounds about right, or one of the later edits. And you can see there's a lot of edit points, just like I would use Final Cut. And because I was building this editor in a collaborative session with Amp, I was able to ask Amp to add features and capabilities, whether it be hotkeys or whether it be entire features that make the editing process and the production process simpler.

What I got Amp to build for me was, as I'm doing the edit, if I see an interesting topic getting covered, let's just say this area here, I can mark an in and out point on the timeline, and then add to the intro or save as a new clip. And the purpose behind doing this is like I already have the edit points that I really like. Let's save this as a clip, which is a duplicate timeline which is specifically for the more tight social media edit. I can hit Save Clip, and that adds it as a clip to the end here.

If I go to the program tab, we have the title sequence that says Raising an Agent, and has the two names. And these names are programmatic props, comma-separated. We can say like 'you', and then if I sort of scrub the timeline, it will add the extra parameter there.

We also have this intro teaser, and this is the—I mentioned before when I'm looking at the full edit, when I'm editing here, I can add a section of clips to the intro. And that is a specific item in the program called intro teaser. And we have this looping audio that plays at the start with like the theme track, which is actually aligned to loop with the title sequence, which means it will automatically create as many duplicates kind of to the left as needed and then fade it in at the point it needs to.

And so then I can edit down this intro teaser to be even more succinct. And then if I click Full Program, this is like the entire intro, then goes to the title sequence, and then goes to the regular episode cut.

Just by being able to sort of like inject my process into the software, so, Amp—first I like to edit the sequence, then I like to find the intro teaser, and then, you know, whack it together in a full program. But then there's also the clips part. I had Amp build me the ability to be able to create clips, and there's a couple of pieces to it.

First, transcript and captions. So I have a process by which Amp transcribes the episode, and there's a bit of a visualization for this I think in the as-of-yet unreleased Orb to (Post) Production, but essentially Amp splits a low-quality MP3 export into like 10-minute chunks, reads those files, and looks for certain mentions and human names on our team and stuff like that, and then runs a Python script to align the actual word timings and set the cue points, and then generate subtitles in SRT and VTT format for Libsyn and YouTube.

I basically told Amp, 'Hey, look at that, and add that capability as a button that we can sort of like run as a job on this orb.' And what that allows us to do is, when it creates the clips, we have, you know, these features like the captions can like have the speaker. If I need to, I can click to toggle off a word if like, say, the timing is ever so slightly off and we've edited it in such a way that a word is omitted, we can do that. We can also double-click to actually edit the word or even add new words.

It has the ability to restructure the entire clip based on aspect ratio, and it has certain rules for, 'Okay, when it's a sort of narrow aspect ratio, put the side-by-side in like a vertical orientation,' or when it's square, and then when it's landscape, you know, do side-by-side like this, and you see the caption track moves from between the splits in the middle to moving 16:9.

Now the big thing I did this time was I actually asked Amp to analyze the transcript and find the suggested edits. And then I had it look at the project JSON format that it's helped to build, and 90% of these clips, I can tell because it went ahead and muted the other speaker track in the videos where it edited them, but these videos, it edited. Now, I did go through and tidy up a couple of things, and make sure that I liked the edit and the decision around whether it was on the speaker or split-screen, but most of these videos, Amp edited in terms of it like had the in and out points, it had the transcript, and so it actually clipped like seven of the ten that we've done. These are just editing down the sequence and adding the captions.

And then I was like, 'Okay, what if we then brought it into Remotion and got Amp to do some sequences?' I shared this on Twitter, but like basically I had the clip ready to go and transcribed. I had a vision, like I could imagine spending twelve hours in After Effects painstakingly recreating UI from Windows 95 and GeoCities, but I just sort of like gave it a bit of a high-level direction that I'd give to someone like a video editor or motion graphics specialist that I trust.

And it worked for eleven minutes, and it transcribed everything, and it came back with this, which for anyone who's seen the video will go, 'Ah, like that's 80% of the way towards what was actually used in the very end.' What it did initially was it did an entire sequence with no video of Quinn or Thorsten. It just used the audio and then created an entirely new video without any edits between the original clips and this motion sequence.

I saw it, and like I cracked up laughing because I'm like this is actually really funny. Like it's pretty on point tone-wise to what I wanted. Bunch of stuff I wanted to get rid of, like if you look in the original, there's a lot of like additional diegetic text that I don't think is necessary to make the point.

So after committing, I just got it to remove a bunch of stuff. And then, like, I tried something that I didn't expect to work. What the models can do continues to blow my mind, and I thought, these videos, to me, are like the read-only file at the end of the project. There's no layers, there's nothing in it. But an agent has so much more patience than I do and so much more capability to work around things than I do. What would it do if I gave it the initial video with Quinn and Thorsten, both with captions and without captions, and asked it to try and compose the two seamlessly using Remotion.

So I have here the—I think it was this one. Internet no captions, and internet with captions. And again, this is the, you know, the original video playing through that I exported from the multicam editor project. It's just got Quinn and Thorsten talking back and forth with the captions burned into the video. And then a version, you know, for Amp's benefit, with no captions burned into the video.

And what it did blew me away. It basically recreated the subtitles from the video with subtitles in Remotion. So it was able to basically diff the version that had subtitles and the version that didn't have subtitles, and create an editable version of the subtitles in Remotion components that more or less matched the version that was exported and burned into the video. And you can see that because when it plays back, and here's like the sequence—so, what it did was it, you know, it did like a graphic version, and then it created variations that combined the graphic version with the video version, and it tried different approaches. Some of them are like the presenter, and some of them are like the takeover. And so the presenter version is the one where when it has a graphic to show, it edits it such that the video goes into a picture-in-picture, and the captions that were burned into the original video have now been faithfully recreated in Remotion, and can be animated from the position to the bottom left as an editable parameter. And again, this blew my mind. Giving it the no caption version and the caption version seems to have unlocked something for Opus, where it recreated the caption design perfectly, using the same fonts, and basically, from two flat videos, created an incredibly like dynamic thing that I could then go, 'Oh, edit this, change this.'

And so for these ones, it only did the first 30 seconds that I then got Amp to stitch into the original videos using my CLI. It also did this one. This one was pretty funny. Presenter in internet analogy through the screen. For this one, this was an alternative, and it like put Thorsten into the screen and then sort of glitched it across to show the sequence. And I thought that was pretty funny. And then at the end, it actually shows like an alternative where when he's talking about how he used dial-up differently, it shows him playing in a pixelated, laggy RealPlayer video. Let me just play this part of the sequence.

I ended up going with the presenter version. And the takeover version is a version where it essentially animates away from the video and into like full screen of the visuals, still retaining the caption layer on top.

This thread where I started with the internet analogy, and this was like a Friday night Perth time for me, and I was blown away by what it did. And so I was immediately like, this can do it for all of them. All I have to do is provide the unvisualized clip without captions, and the visualized clip with captions, and then it can essentially reconstruct the captions and animate them as it needs to with the visuals on screen. And so it worked on a bunch more.

So I had Amp both select and edit and render like nine clips, eight clips from the episode, and then I brought it into this thread, and so each of them, I sent the captions and no captions variation. I'd give it really specific feedback about things I didn't like in it, and work through it. Most of what I had to do was just tell it to reduce or back off doing things.

We have this video where we're talking about orbs and runners, and I wanted us to use the same visualization for orbs and runners that we have in the Amp product. So I was able to get the working threads in the VidGen project, I was able to get them to spin up orbs in the Amp project to gather design resources like our orbs design, and even like things like our Amp logo, so that we could then, you know, use the same visual language that we use to represent runners and orbs currently in the actual videos we're publishing on Twitter. We're not just using generic components the agent invents.

And there's a story that Thorsten tells where he talks about how he opened his personal Amp account on his Raspberry Pi to check how much energy they'd used. And again, for this, I was able to actually ask Amp to open an orb in the Amp project, and actually gather all of the UI from the mobile app, so that we could actually have this relatively realistic and yet simplified version of the app, rather than just a generic transcript that doesn't look anything like Amp. The really nice thing about this approach, as well, is that the more components and reusable, composable parts that are in the codebase, when we have future things to visualize, this is becoming like this growing library of content to be able to use.

It's kind of blown me away, and like I don't even know where it's—by the end of season two, Amp is probably going to be editing the episodes itself. I don't know. You know, what it really shows me is that when there's like domain knowledge and expertise—so, like I have experience doing video editing, production, motion graphics, all of that, I know how to achieve certain things. But then there's a level of effort and patience and just time that I don't have to painstakingly churn through some of these sorts of tasks.

And the incredible thing about Amp is that, and frontier models, is that I can very easily have it do the grunt work for me, and still kind of be in the driver's seat, or at least a co-pilot seat for the time being. You know, I'm not just accepting the first generated result it gives me, but I can get to very specific results I'd like so much faster than if I was trying to do this all manually.

So that's a bit of a run-through of how I've been doing things. Thanks for watching if you made it this far.