Deforum Stable Diffusion Is Awesome

2023-03-05

This is wildly cool.

First off, Replicate is a pretty interesting service. They basically host models and put an API in front of them, and you pay per second of GPU time. This is a nice model if you’re looking to do something AI related but don’t want to go through all the hassle of setting up your own hosting of a model and building an API layer. I’ve got a little side project in the works for a talk I proposed at an upcoming conference. I’ll likely end up using Replicate to cut down on the work involved.

Secondly, if you're plugged into tech at all, you've probably seen something like Stable Diffusion, Midjourney, or Dall-E in the last few months. AI Image generation has been a pretty big deal recently, and it's crazy to see what people have been able to produce as they've iterated on various techniques.

This model is particularly cool, though, because it produces videos instead of static images. Without knowing too much about how it works, it looks like what it's doing is basically prompting Stable Diffusion repeatedly to produce each frame of an animation. There are all kinds of parameters on the model to control various things like rotation and zoom, giving the user some degree of control over the output.

You may have seen some “AI generated lyric videos” a few months back when Stable Diffusion and Midjourney hit the scene - people were taking the lyrics of songs and using each line as a prompt for image generation.

I have always wanted a cool animated music video for the 20 minute prog rock epic Close to the Edge, by one of my all time favorite bands, Yes. So, with that in mind, I made two different videos with Deforum. The first is a custom prompt, meant to be played over the intro of the song. The intro has this soundtrack of running water and chirping birds that kicks off the album. My prompt for this instrumental/ambient section was "birds chirping, flying around in a floating island forest scene. waterfalls fall from the sides of the floating island. cool colors with lots of green, in the style of Roger Dean" [1].

I absolutely love what came out of the model - check it out:

The second video I made was prompted with nothing more than a lyric from the song: "the time between the notes relates the color to the scene"[2]. Again, I think what Stable Diffusion came up with is really cool, and perfectly fits the mood of the song at this point.

If you make an account on Replicate, you get a little bit of free usage - I generated these two videos and a couple static images from other models before I ran out of free tier usage. So if you’re interested in this, know that while you can play with it for free, the video generation stuff will burn through your limited free credits pretty fast, since you're generating 100 images in one go. There are some other nice models to play with over on Replicate - I had a lot of fun playing with Vintedois Diffusion, which is capable of some really nice output.

Anyway, that's all I have to say for now. As usual, I'm continually blown away by what we can do with these models, and excited to see what's next. And I guess I'm overdue for yet another listen to Close to the Edge.

[1] - Adding "painting by _some artist_ is a good way to point Stable Diffusion towards a particular style. Roger Dean is a rock art legend who designed album covers and logos for some of the big prog bands of the 70s and 80s, and did lots of art for Yes, so he's very appropriate for this job. Even if you're not a big ol' prog nerd like me, you'll likely find his work familiar. His style suffused through a lot of pop culture over the last few decades, especially the "floating islands" he seemed to be so fond of painting.

[2] - I have no idea what this lyric means, but I've always thought it sounded cool. Most of Jon Anderson's lyrics were like that, to be fair - Yes lyrics are much more of a "vibe" thing than any specific message. This lyric in particular comes after a kick-ass instrumental section with a killer keyboard solo by Rick Wakeman.

As a footnote to this footnote, Wakeman is probably (indirectly) one of the biggest influences on punk rock - punks apparently found the progressive rock movement of the 70s so excessive and overwrought that it spawned a whole musical genre in opposition. Wakeman's elaborate stage costumes (often involving CAPES) and insanely technical multi-keyboard playing is about as prog-rock excessive as you can get. It's also awesome.

If you liked this post, sign up for my email list via Substack so you can get notified whenever I publish a new post!