Orbing Screencasts

Hi there. So I wanted to record a quick bit of a meta video time capsule, showing something I'm working on with Amp and an interesting way that our primitives are sort of coming together.

So yesterday, Thorsten recorded a video demo of using Amp and recorded it with Screen Studio. He provided a Google Drive link to me, which Google displays the package from Screen Studio as a folder, a directory full of metadata and assets.

And so I downloaded the folder. Google Drive did that thing where it basically zipped it up into individual different packages, so there's four zip files and a single MP4, which is the webcam stream. And so I was like, okay, cool. Now, I do have Screen Studio locally, and Screen Studio editing is an option, and it may be the fallback option I end up going with.

But I wanted to explore the ways in which we could, you know, make the most of this source of video. You know, maybe we want a version that's produced and highly edited for Twitter, or for YouTube, but then maybe we actually want a different format for the marketing site, for example. Chatting to Tim, it was like, well, what if we had like an interactive version of the video that lets you actually navigate to specific sections for different features being showcased in the day-to-day in Amp sort of video that Thorsten's recorded?

So when I first downloaded the Google Drive split version of the video, I was unable to get it to actually reconstruct a usable project just by trying to open the zip file. So the first thing I had to do was ask Amp for some help. I asked it to extract this multipart zip that Google Drive downloaded, and one of the parts of it had come from, you know, it was just an MP4 video.

And dutifully, Amp rose to the occasion and reconstructed the Screen Studio project with all the metadata and everything included in it.

Then I was hit by a sudden strike of inspiration. Like, I can open the Screen Studio project. Well, not like that I can't. I could open the original Screen Studio project, and then we have the video that Thorsten recorded, and we have all of the data in it, and we can make some, you know, wacky zooms if we want.

But I was more interested in the fact that with this bundle, we have access to the original source video, we have access to the original audio, and we have probably access to the other metadata as well.

So the first thing I actually asked it to do was extract the original audio recording, desktop recording, and webcam. So we have just a copy of those files sort of packaged up. So here, for example, we have the video of Thorsten in its full resolution. We have here the original video recording of the desktop that Thorsten recorded, and we have the original audio as well.

And with these pieces, we can actually potentially explore more interactive ways of displaying this content. Like, I want to edit it once, make sure that the sequence works well, but I kind of don't want to be held down to a specific editor format or anything like that. I want some edit points, and then we can look at the various ways that might make sense.

So I've been working a little bit with Amp recently on some random video generator utilities. So this morning, I sent Amp a pretty lofty prompt where I basically said, you know, 'We have some existing content to look at. We have existing code to look at in this VidGen project that I've been working on,' and I basically asked it for an optimized UI that lets me make edits to the video, keeping the microphone, video, webcam, and desktop recording in sync.

And, you know, go from there. So this is an orb thread running on a remote machine with a portal that I can look at that has the utility for me to add the screen recording, the camera, and the microphone so that, you know, everything's synced up. And then when I select a clip, you can see it's given me some configurable options like, you know, how much is given to Thorsten, how much is given to the video, and how much padding should there be around the video, and what's the corner radius, and I can kind of nicely trim out the little bit of desktop bleed you can see from the sort of like the side there.

So anyway, let me just go back to—there we go. I think that's good.

So that's pretty cool. And so I started to watch the video, and you know, lo and behold, it works. If I play it back, you'll be able to hear it.

Then I noticed as the video played that, of course—and you can see there's a little sad face there for the waveform peaks, but v1 of the software, you know, why not?

I noticed as Thorsten started interacting with it that, of course, Screen Studio does that thing where it dynamically records the cursor position so that it can do its own animations. And one of the interesting issues and challenges with that is that sometimes the smooth motion misses where the hover states are shown in the original recorded video, stuff like that. That can be a bit annoying, but it does let you do some interesting dynamic edits. It's just, you know, a thing that we have to deal with. But that means that in the original video, as he's moving around, there's no mouse cursor.

So I was struck by an interesting idea. You know, I'm not sure if Amp is going to pull this off, so I'm recording this as I'm working on it. So I sent this—and I'm going all over the place, I'm excited, and my thoughts are unstructured—but I gave, you know, Amp this big prompt, put it all together, started editing it, you know, knowing that the metadata I'm recording here is just being saved as JSON, and I could probably get it to export it into a format back to, you know, Final Cut or anything else if I decide not to proceed with this incredibly rickety vibe-coded, Jelly-coded approach.

But what I thought I'd do is I've got my local machine running, and I've got this orb thread running. This local machine has the 8 GB Screen Studio project.

And, oh, there's another part that I forgot to mention. I'm all over the place. So I mentioned that I want to look at—so we have the produced video for Twitter, and I thought it would also be nice if we had a format that allowed us to have an interactive player on the marketing site, where it wasn't burned in, where I made the choices around the cuts, and then Thorsten appears here, and maybe there's a section where he's picture-in-picture and it overlaps the video. What if we exported the webcam and the desktop separately, and someone could actually have an interactive UI where they toggle the layout based upon the device size, screen size, that sort of thing?

So what I did was I asked Amp to investigate the setup in the marketing site, and it created a subthread. So this is the VidGen project with all the code for the random video utilities I'm using. It spun up another thread in the Amp project, which I haven't really looked at, and it basically gave this thread the instruction to look at everything it might need to host an interactive version of a video that has separate webcam and desktop recording and provides a UI that lets people scrub through the video to different features. So it came back with a report on how it could sort of approach that, and that was given back to the screencast editor thread here, which kind of like explained some of the tooling and made some recommendations about a shared metadata approach that it could use to consume a format provided by this editor once it was all said and done.

So after that, it got to work building it, which is basically what we have here.

And then, as I'm editing and going, 'Yeah, cool. Like I'm just sort of like going through and trimming it for time and things like that,' I noticed that, of course, Screen Studio is not baking the mouse cursor into the video.

So now I've asked my local thread that has the Screen Studio project, I said, 'Hey, an interesting challenge has emerged. The mouse cursor is of course hidden in the recording because Screen Studio does that. If you take a look at the thread, we're working on a web editor based on your reconstructions. It would be super useful to be able to implement the cursor data from the Screen Studio project into the editor. We don't want too much in the way of eased mouse movements, as much of the original data as possible. The web editor includes the ability to trim edits. We'd want the projection mapping of the mouse coordinates to snap to the size and position of the desktop capture track, noting it may be cropped in certain layouts.'

So then Amp reads the attached thread, this one, on the orb, and finds where the mouse move format is in the Screen Studio project. And it says, 'The raw track is unusually rich. There's a lot of, you know, samples plus exact cursor changes and 152 click pairs. I preserved that event stream rather than pre-easing or reducing it to 60 fps. Key mapping detail is that Screen Studio—' So I have no idea how the Screen Studio mouse cursor stuff is done, but I have a sense of what I want, and it sent the real cursor dataset and implementation spec to the web editor thread. It will preserve all raw events.

Over here, I've been talking to—I couldn't get the M4A picker working initially, and now it works, so that's cool. We have audio from Thorsten. That's important.

And then in the middle of this thread interjects the agent from my local machine with an info dump of what it's done and all of this stuff, and some specs for what it needs to do. And like, now it's going ahead and it's still working. It's kind of like, you know, doing all this stuff, and it'll be interesting to see how this all works and if it works, to be honest.

But the really exciting thing about this is that, like, I thought about all the ways that I could edit this video from Thorsten, make sure that we get it into a nice sequence. I could use a proprietary nonlinear editor like Final Cut Pro X, where I'm very comfortable making edits. I could use After Effects if I really want cool, like, nice, you know, motion graphics and visual treatments. But what I'm really focused on first and foremost is the story we're telling around how Thorsten uses Amp, and I want to get that sequence as smooth as possible, and I want to preserve flexibility in the final export of it.

So I'm going to go through this video and try editing it in this editor of all things, because it's preserving all of the edits in a format that I could potentially then ask the agent to export to After Effects or to Final Cut. But I have all of the raw source audio, videos that I'm working with. I'm having Amp build out a reconstruction of the smart cursors that Screen Studio has, with an agent locally on my machine able to beam that information up to the orb. And that's just preserving optionality. We can still have a nice, highly produced video that's rendered for Twitter, but then maybe on our marketing site we'll have a much more bespoke player that allows people to actually, like, more interactively engage with the web player version.

A feature I haven't mentioned we have as well: I've added chapter markers here, which we can use to mark segments. And this is kind of what we'll probably be using to denote features in the player. So I'm sort of defining a spec, some metadata about the video that's going to live alongside the video, that can be then used in the Amp marketing site to allow people to like skip through, 'Oh, where does he use schedules?' or 'Where does he use this feature?' And I'm doing this all while, you know, watching just the source video and trimming it down for time and making it run smoothly.

So yeah, like, early days. The agent is still iterating on these features, and it will be very interesting to see how it goes here. Editor wired up, now the reference player. Okay, so the editor's wired up apparently.

All right, let's give it a try. 8 GB Screen Studio.

Okay. Oh my god. There it is.

I know people are deconstructing retro games and reverse engineering everything, but it just kind of blows my mind, this age we're entering. Like, you know, I was at every step along the way, I would have expected to have encountered something. 'Oh, proprietary format, proprietary app format, we're going to have to go back to the editor.' But Amp has just pulled in that cursor data. I've already made an edit here, and it's properly identified the cut-out segment, and put in the mouse cursor, and presumably, boom, I can increase the cursor size if I really want. So I get a lot of the flexibility that Screen Studio provides with the initial recording without having to have used Screen Studio.

So yeah. Like, that's incredible. You know, it's yeah, wild.