text-to-image (stable diffusion)

I created "a room too soon" by starting with a single source image then using stable diffusion (DALL-E 2) from openai I created 30 variations of this image. I downloaded these images and put them into a folder. I then composed it into a single image using the graphics magick command: "gm montage -tile 7x5 ./*.png -geometry 2000x2000+0+0 grid.png". You can install it on debian based linux distros with: "sudo apt-get install graphicsmagick".

Recently I decided to play around with some interesting tools for generating new audio visual art, and this was the result. I somehow ended up with some creepy photo's as a result of prompting DALL-e 2 stable diffusion image generator. I used these runwayML and processing to generate video files. I reprocessed these videos a bit and edited it together. All using open source software and some paid online ai tools.

You may know that text-to-image allows you to generate realistic images from text prompts like "a sunny blue sky", It is now also very easy to generate very nice sounding music from text prompts now if your willing to pay for a third party text-to-music ai tool or use huggingface.co to run experimental machine learning models and pay for the computing time. The issue of what you do with them is very uncertain from a copyright or legal standpoint. Most websites that offer ai generated music try to upsell the copyright ownership or warn that you must not upload it to spotify or publish it. The rationale for limitations is unclear so the method I prefer is to use huggingface.co and pay to have more flexibility and direct control.

text-to-music

It has been possible to use machine learning to generate MIDI data for ages but it wasn't that impressive. Having something that generated realistic audio using .wav or raw audio instead MIDI was an unattainable dream. That is until recently due to the emergence of machine learning projects musicGen and musicLM that can generate audio based on prompts like "pleasant sounding harpsichord". musicGen is available via huggingface.co so I am using that for the mean time because google have not opened up their equivalent musicLM playground to researchers.

machine listening

There is currently MusicGen (for text-to-audio) and AudioGen for more realistic sounds. You can give Audiogen a 10 second sample and get it to keep generating an extra 20 seconds from it in sample mode. Or just ask it for "vocal chops for a dance or garage track" and then I used Pure Data and the bonk and sigmund objects to detect the beat or pulse in real time and randomly add these vocal chops and sound effects to the track whilst recording the output.

There has also been a number of interesting text-to-image and text-to-video tools emerging that will generate audio and video from text prompts. Currently the video's are only a few seconds long but stable diffusion tools are capable or realistic outputs. There is also a frame interpolation tool (on runwayML) that allows you to interpolate between a series of images and control the speed that the images warp into each other.

When using huggingface.co you will have to duplicate the space and pay for a GPU machine... be careful to set the sleep time to high enough length of time so that it doesn't switch itself off while generating audio. If you forget to set your space to sleep after your done you may end up paying for it when your not using it. I have accidentally left it running over night a few times and now and racked up a bill :/ so watch out for that.

The next idea I'm going to try out is using text-to-video or image-to-video from runwayML to generate a 3 second video. Then take the last frame of the generated video and feed it into the image-to-video again. In theory the video will dream or imagine new visuals and could produce some strange results. If the process could be automated it could be more interesting because you could generate longer sequences in less time without any manual effort.

I realise that I haven't actually explained how any of the underlying algorithms work.. this is more about the artistic processes involved.

tools used

images

DALL-e 2 : stable diffusion for image generation
artbreeder

video

processing -> generating a video from a folder of images
frame interpolation -> creating a video that interpolates between a series of images
image-to-video
text-to-video
vdcrpt-> video glitching software for datamoshing

audio

audacity
riffusion
landr - ai audio mastering

Interesting one I didn't use but worth testing:

ai audio stem splitting lalal.ai

video processes used

generate images using DALL-e 2 or runwayML text-to-image
use processing to create an .mp4 video from a series of images (configure the video frame rate)
or use image-to-video or frame interpolation from runwayML
use vdcrpt to create datamoshing glitch effects
use blender3D video editor to edit a final video

audio processes

run musicGen on huggingface.co and pay for GPU time.
run audioGen on huggingface.co to to generate sound effects.
generate up to 5 minute .wav files based on text prompts
use riffusion to generate some audio
use audacity to combine all the audio elements ready for mastering
upload tracks to landr to be mastered

links

huggingface.co - GrandaddyShmax/MusicGen_Plus_hfv2 artbreeder processing riffusion blender3D runwayML audacity MusicLM lalal.ai openai vdcrpt landr

author

m-onz