Images From Nothing

In today's episode, we're going to talk about image generation models, which can take a prompt and turn it into a compelling variety of images. This ability to generate with variety rapidly is great for experimentation and creative work. But image generation is also powerful for building highly specific, nuanced kinds of images to do things like extend a product experience. We're going to explore the full breadth of these possibilities and some of the techniques you'll need to use to reach them.

Show Notes:

• Jay Alammar’s visual guide to how StableDiffusion works: https://jalammar.github.io/illustrated-stable-diffusion/
• AWS’s blog on how to empower StableDiffusion with retrieval-augmented generation: https://aws.amazon.com/blogs/machine-learning/improve-your-stable-diffusion-prompts-with-retrieval-augmented-generation/
• AWS’s blog on getting started with StableDifussion using SageMaker Jumpstart:
  https://aws.amazon.com/blogs/machine-learning/use-stable-diffusion-xl-with-amazon-sagemaker-jumpstart-in-amazon-sagemaker-studio/ 

Official Transcript:

Ryan Ries:

Welcome to Mission Generate, the podcast where we explore the power of AWS and the cutting-edge techniques we use with real customers to build generative AI solutions for them. Join us each episode as we dig into the technical details, cut through the hype, and uncover the business possibilities.

I'm Ryan Ries, our chief data science strategist and your host.

In today's episode we're going to talk about image generation models, which can take a prompt and turn it into a compelling variety of images. This ability to generate with variety rapidly is great for experimentation and creative work. But image generation is also powerful for building highly specific, nuanced kinds of images to do things like extend a product experience. We're going to explore the full breadth of these possibilities and some of the techniques you'll need to use in order to reach them.

Let's dive in.

First, our usual disclaimer: just like how the images we'll be talking about today are generated using a computer, so is my voice! We've synthesized the voice of the actual Ryan Ries to let the computer do the talking. His nickname is RyAIn, by the way, and he handles the podcasting so I can stay engaged with customers.

We thought a great way to make a podcast about generative AI was to build it with generative AI. So if this is your first episode with us, welcome. I'll do my best to keep sounding real and alive for you and you can... well, presumably you are real already, so you can just sit back and enjoy.

Okay, back to the show.
In essence, concept-specific image generation lets you transform textual inputs into visual outputs. But rather than replacing traditional design or photography workflows, this technology shines in how it augments that work. It infuses the process with an intelligent partner to help you to more quickly experiment and materialize your ideas.

With image generation, suddenly your cycle time can be a fraction of what it was. Try 100 iterations in just a few minutes. Pursue the most interesting options and gradually evolve toward a design you prefer in real-time. Or dramatically reverse course and try something different--the time and effort cost of a major course correction will be nearly the same as a minor tweak, so experimentation is essentially free.

Seen in this light, concept-specific image generation is really a way to unleash artistry--not replace it. And by leveraging foundation models like Stability AI’s StableDiffusion and AWS native services like SageMaker, Bedrock, and S3, you can create a robust architecture for producing, managing, and refining digital assets with speed and precision.

It should be clear how this empowers traditional creative work, but this technology has major implications for other areas as well. Let's talk about a few of the solutions we happen to be working on right at the moment. A real estate platform is scrubbing sensitive and personally-identifying information from its listing photos. A plastic surgery clinic is creating post-operative photography to show their clients exactly how they'll look after the procedure. A global packaging firm is applying lighting and other effects to its 3D renders to help it generate product mockups more effectively. And an online photobook provider is adding new features to augment user photos with generated suggestions that both recognize their preferences and the context of each photo.

So as you can see, there's really a wide variety of possibilities with image generation. Unlike pure creative work, however, the stakes for these solutions are quite a bit higher than missing the mark on a branding exercise, for example. We'll come back to that, but let's talk about how the models can do this for a moment.

The techniques to achieve all this can be a bit intimidating if you're unfamiliar, but we'll do our best to break them down. Here's a quick survey of some of the most prominent ones to give you an idea of what's possible.

Inpainting allows parts of an image to be reconstructed or modified based on the surrounding pixels. It's useful for removing objects from images, filling in missing areas, or editing parts of images in a way that looks seamless and natural.

Outpainting can be thought of as the opposite--it extends the boundaries of an image beyond its original edges, which is useful if you need to enlarge something.

Generative Adversarial Networks pair your image generator with something called a "discriminator," which is acting sort of like an art critic and evaluating output. This is great for training and getting to higher quality outputs quickly.

Conditional Image Synthesis conditions the image on a set of inputs like labeled data, text input, or even other images. The idea here is to more closely direct and control the output of the model.

Sometimes it's useful to experiment with one or two of these approaches to see its effective limit in solving your problem before combining. But in general, this is where the complexity of effective image generation comes in.

You also need to remember that if your images are customer-facing, your solution needs to come with safeguards to ensure its suggestions are never inappropriate or out of line with a user's expectations. A great user experience should let the user get to something they like quickly without too much effort and do so in a reproducible manner.

For this reason, you may need to consider implementing a native service like Rekognition, which uses machine learning to analyze images for content, as part of your solution in order to vet suggested images before displaying them to the user.

And more targeted solutions, like our plastic surgery example, also need high accuracy. A photo in that solution that's even slightly off may give the customer a bad impression of how they will look afterward. And if you're trying to do something like intercept and remove personally identifying information, you need a solution which is ironclad as even one miss slipping through could be serious.

As we've said in other episodes, these models being probabilistic can be both their greatest strength and greatest weakness, and tuning and tailoring a solution to get predictable results remains a challenge. This is also one of the real differences between an MVP image gen solution and a production-grade one.

That's it for this episode. We hope you found it enlightening and that it enlarged your view of what's possible with image generation.

If we've sparked a great idea, or you've been working on a solution and would like to talk it through with someone, we'd also like to extend you an invite. My team will give you an hour of our time, free of charge, to explore what you're trying to achieve and discuss which techniques and architectures you might be a fit for. We understand that it's one thing to get a proof of concept working and another to build something that reliably scales to a real business problem.

If that's a conversation you'd want to have, drop us a line at mission cloud dot com.

We hope your images turn out just as perfectly as you envisaged them, and as always: good luck out there and happy building!