podcasts / Latent Space: The AI Engineer Podcast / episode
Latent Space: The AI Engineer Podcast cover art
podcast
📅 2026-11-16 ⏱ 98 min episode 📝 Transcript · 20,267 words

Building Visual Intelligence — Andreas Blattmann & Dustin Podell, Black Forest Labs

Latent Space: The AI Engineer Podcast · · Transcript & summary by WhipScribe

🔔 Get every new Latent Space: The AI Engineer Podcast transcript by emailOne email per new episode. Unsubscribe in every message.
One email per new episode, unsubscribe in every message. Not shared with the publisher.

Summary, chapters and key takeaways for this episode are on their way; the transcript below is complete.

▸Transcript

Our own transcript of the public episode, with a timestamp about every 90 seconds. Tap a timestamp or the link button on a paragraph to copy a link that opens this page there.

0:00Happy to bring to you our episode with Black Forest Labs. Our guests today are Andreas Blattman, he also goes by Andy, who is co-inventor of latent diffusion, the technique that eventually got scaled up by stable diffusion and stability AI. And they eventually split with a bunch of the core stability AI image gen folks to form Black Forest Labs. And Dustin Podell, who I also met with Ange at one of the stories that we tell, and who worked his way up from just being a regular engineer, a freelance web developer person, into a model and research lead at Stability AI, and now Black Forest Labs, creator of SDXL, Control Laura, post-training lead on SD3, Flux Ultra, Flux One, and Flux Context. And we go into all of these details.

0:43As you may know, sometimes we're very honored to have an extended history with some of our guests. And this is one of them. You might have heard in our Anjumita episode where we talked about introducing each other at one of the NeurIPSs and meeting Black Forest Labs for the first time. And this is right after Stable Diffusion. But what you may not have heard, and we only tell this story in this episode, is how Latent Space itself was started because of Stable Diffusion, directly crediting the launch for me, basically quitting my job to go all in on AI based on what we saw. And today's guest, Dustin, also had that experience and so did Vibhu. And so we have a very, very strong tie with the team that basically led up to Black Forest Labs.

1:30And now we're also interested in what's next for them. They have created some of the best open and closed weights models for image generation and video generation. I think especially Flux Context, if you're not aware of image editing models, I do think that this is, in the limit, one of the things to fully replace Canva and Photoshop and a bunch of other workflows and editors that you might use in a normal creative workflow. It is not going to replace the top end of the market, but for regular, normal, everyday edits, why would you not want anything else other than to query a model and ask it to do image editing for you? We do this, obviously, with this kind of video editing. It's all Clanker done, and I do review it, and they're also moving into

2:17robotics. And so this ties in with a mini-series of model labs that have been pursuing this direction, building world models from image gen, like with runway, as well as video models and now robotics models. And so I think a lot of things rhyme in a lot of our recent episodes, like even the artificial analysis episode that we're going to release soon, they are also going to robotics. And it seems pretty obvious that when you see a whole bunch of smart people who are at the frontier, all leaning one direction, that you're probably six to 12 months out from some kind of really big breakthrough. This has been a consistent theme over my last four years of podcasting. It's not going to change now.

2:55So look out for it. That's the most obvious call out I can make. But there's also a lot of interesting lore for people interested in the general diffusion space. We're going to do a little bit more over time. I'm thinking about an AIE LA where we talk about the generative media field and the creative arts and how that complicated relationship is evolving and I think maturing, to be honest. And finally, for those longtime listeners, we are starting to open up to advertisers, people that we really admire and respect and want to showcase what they're working on. The first one was with Superbase, which I think a lot of you enjoyed. And the second one today is brought to you by Deepgram. And you'll hear more from us at the midpoint of the episode.

3:39You should be showing a robot an action one time, and then it should be able to do this. And this would be truly a superhuman intelligence. Like add stability that we couldn't essentially leverage the value of what the models could be as well. Mistakes of not being able to actually capture the value of the models themselves. We raised all in all $450 million in funding so far. Currently, the last that was published in December was $100 million in revenue. BFL was not founded to be an image generation lab. Video modeling is actually the central means to understanding how our world basically works. It was maybe like six hours of training, four hours of training. It was a very, very short training run. And all of a sudden, I saw what felt like a clear generality coming out of it because it was... Okay, finally, our BFL pod.

4:28Welcome, guys, Andreas and Dustin. Great to be here. Thank you so much. Yeah, definitely. Thank you for having us. I know we've chatted about doing this for a long time. Yes. So we've had Ange on a couple of times, and that's where I met you at Vienna. I think one of the ICMLs heard. Was this like one of the, like, was it was the like rooftop event or? There was a year when there were two VNs back to back. Yeah. Yeah. That was first iClear and then it was ICML. I don't know how they do this thing. Anyway, anyway. And at the time you guys had, were like just about to launch BFL. Like we had a dinner and like, you were like, oh, and thinking about what's next after stability and all that. So anyway, congrats. How do you guys like to introduce yourself? Maybe

5:09we'll sort of get a crisp self intro up front. So I'm Andy. I'm a co-founder of BlackBose Labs. Been developing diffusion models for, oh, wow, I think seven years now. Met Robin and Patrick way, way back in the day. I think it's, it's seven years now since we met in Heidelberg. I'm from Germany originally. And there we started looking into just diffusion modeling for images. We were all having a kind of a physics background in some sense. And this paradigm of just mapping this general physical phenomena of diffusion to machine learning and generative modeling had a kind of a philosophical kind of beauty to us. That's why we got into this. And then, yeah, developed a lot of, I guess, fundamental algorithms that are powering

5:56diffusion models today, or image generation, video generation, and also action prediction today. I guess we'll talk about this later. We'll talk about it. It's not immediately obvious to people, image generation to action prediction. But we can talk about that. And then one thing I did want to comment is a lot of people don't know the relationship. Black Forest is a forest in Germany? It's a mountain range, actually. But yeah, it's a region in Germany, basically. And finally, Robin and me, we first met during our PhD. And then we figured out we were actually born, I don't know, 10 miles away from each other. And that was pretty fun. And then eventually that led us also to come back to Germany.

6:37Beautiful. All of Europe looks like this. You can see the lab right in the middle there. Yeah. So you named it because you're from here. Exactly. We named it because actually we founded this research lab and back in the day we could basically pick where we wanted to do it. No one said, okay, you have to come to, say, SF or whatever place. So we decided, okay, we want to be actually starting this at the place where we were born in Freiburg, which is like at the foot of the Black Forest, basically, or a city in the Black Forest. And that's how it came. And that's how then the company also started there. Yeah. Comfis, also Heidelberg? Comfis was in Heidelberg, exactly. That's where we met. We then later moved on to Munich.

7:19Which is the team that did Latent Diffusion. Through our PhD, exactly. And that is when we developed this latent diffusion algorithm. I really like your name, by the way. Your podcast name, Latent Space. I would actually be curious how that came together. I can talk about it, but yeah, go ahead. And exactly, we developed this latent diffusion algorithm at the university. And then based on this research project, we scaled out the algorithm, which led to stableable Diffusion then effectively and yeah since then it's been a constant journey on a rocket ship you could say which is super fun to have and we've met amazing people like Dustin as well who's one of the kind of oldest and like most I think

8:02relevant researchers that we have. But yeah, we'll learn more about that. We'll gas him up later. I think latent diffusion and just generally like VAEs as well, and then for us latent space, is this general idea. Even our logo is actually a VAE. And it's this idea that you can translate things into spaces and then explore things that are adjacent to what exists today. And then you can sort of retranslate back. And I thought that that was like such a cool insight for creativity and generation that I never really understood until I studied stable diffusion. Like the launch of stable diffusion changed my life. I started in space because of stable diffusion. I quit my job because of stable diffusion.

8:42Because I was like, okay, everything I know about AI is wrong because I can run this on my laptop. And like, if I was wrong about this, what else am I wrong about? So there's that. And then there's another relationship with Terry Pratchett, which I don't know if people have heard of. Writer, right? Writer. He has this concept of L-space, which is library space, where he has Orangutan that's like the librarian in Unseen University, which is Wizard University. And it basically is a library with all possible books. So if you have a library of all possible books, you can just theoretically go in there and find a book that is right for you. And that is Layton Space. So I originally saw it called L Space.

9:19One of my readers donated Layton Space and then became Layton Space. Amazing. Dustin, what was your journey into diffusion and AI in general? Yeah, so I came from a little bit of a different background than Andy and Rob and the team. So I'm not from the German side, but I've still had a fantastic time working with these guys. And honestly, it's funny you mentioned that you quit your job when Stable Diffusion came out because the original Stable Diffusion, I was not working with these guys at. And it was actually a cause for me to quit my job at the time so i am he caused mass unemployment i'm actually curious like the numbers of people that just like changed their life direction when stable diffusion dropped i would actually be very curious to know kind of the branching effects from

9:59that but i don't know we can we can theor on it. That's actually super interesting because here in SF, in the Bay, it has this huge impact. Whereas in Germany, where we stayed back in the day, people didn't really notice it. Because the people were so far away from tech. There are a couple of people, obviously, that have been involved in the open source community also in Germany. But the majority of people didn't really notice that drop in summer 2022, which was really interesting. And then you see what happens here. Like, yeah, to us, that was really kind of super interesting. Yeah. But anyway, so I guess to give a little background on me. So at BFL, I'm one of the founding researchers that joined in with the, we had a bit of a

10:44larger team that I'm sure we'll get into the history of kind of how that all came to be. But my background is not actually traditionally in, you know, the ML space whatsoever. I was traditionally a software research or software developer for around 10 years or so. Early in 20, you know, around the 2020s, I became very, very interested in just like open source AI. I saw like the kind of the early work in GANs and other, you know, in generative media. And I was tracking with it. I was experimenting with it a little on the side. It got me quite excited. I mean, and I remember following originally the latent diffusion work. I was actually, I was following that and quite excited about it. And then the one, you know, one of the things that kind of tipped me off to go, okay, maybe this is something a little bit more serious is actually a shout out to Midjourney V1.

11:31Wow. Back when that came out. And I remember thinking like, okay, this is actually going in a direction. I can see this being quite impactful. And then the one big moment, the big drop that really changed everything for me, as we talked about, was the Stable Diffusion release. And at the time, I was working at a software studio down in Los Angeles. Stable diffusion came out. And basically that day, I turned in my resignation and said, okay. What was the plan? The plan? So the way that I saw it, the way that I envisioned it at the time, I saw this thing that felt like, it literally felt like a magical thing that came out and I viewed, okay, the people that are working on this must be like arcane wizards.

12:09And I need to do whatever I can to understand like what magic or what ruins or what things that they did to imbue in this to make this happen and let this go out to the world and have this, have this impact. It wasn't like I was just totally disconnected at the time I was in the stable diffusion discord. I actually had friends in there At the time, I was in the Stable Diffusion Discord. I actually had friends in there. I was kind of interacting with the community. Was it viewed as basically open source mid-journey? Or, like, do people not make the connection? I mean, I was in both. I remember there were a lot of kind of discussions around this. Like, oh, Stable Diffusion was having such a, you know, there was a lot of discussion.

12:40And I remember the mid-journey Discord and the Stable Diffusion Discord around that time around the impacts it was having. I mean, it's been a long time so I can't remember the exact nature of it. I don't want to speak on, you know. MidJourney's doing fine. They're doing great. I mean, they're doing great. But for me, it was more just like I was very interested in the fundamental technology and then the fact that I could now run this at home and then really the thing I saw was then just this community of developers actually like taking it home, developing on it. I was hacking away at it. And then really the thing I saw was then just this community of developers actually like taking it home, developing on it. I was hacking away at it. I was meeting a bunch of

13:09other really, really smart, great people, some of which are actually at BFL now that have come along for the journey. That moment of just community interaction and learning and developing and building together was really an impactful moment for me. Luckily enough for me, I was able to find a kind of a job opportunity to get myself into Stability at the time where, you know, this was starting. And I didn't actually start working with these guys right away. It took me a while to kind of work my way up to that. And we can go into that history later if we want. Stability had a bunch of different teams, right? It was Stable LM, stable music. Harmony, the audio team. It was like, I don't know, the way I would kind of describe it is like if you grabbed a bunch of...

13:52Discord was kind of the space that a lot of this happened at the time. I don't know why the whole... It's like if a Discord had a physical manifestation. Technically, it was a melting pot of very different people that came together but just like interested in this emerging new technology that came, obviously. And that's also how we are. This is one of the most exciting things about our team. Our founding team comes from very different backgrounds. We have this hardcore academia research branch, and then we have people who were from the open source community, like Dustin, who actually got via a very different kind of trajectory into this space but i think this is what makes our team so strong

14:30we have these diverse backgrounds and there's yeah there's rarely a team i think that that has this history also this kind of joint history together yeah yeah yeah this is just amazing that that yeah yeah and we can go into more of your other co-founders first yeah I did one so I was also in that space right like literally you know I was also a very influential stable diffusion I did the wrong thing I basically went down the path of comfy UI where I build UIs for stable diffusion you seem to have done the right thing where you went to research well it's funny you mentioned comfy because like I used to work directly with I mean comfy joins him on the pod yeah and he moved here yeah yeah he's like my neighbor how does a It's funny you mention Comfy because, like, I used to work directly with, I mean, Comfy joins. We've had him on the pod, yeah.

15:05Yeah, yeah. He moved here. Yeah, yeah. He's like my neighbor. How does a, he's a researcher. How do you become a researcher? I'm someone that's, like, when I get interested in something, I am obsessive to the point of I cannot put it down until I pull it apart and figure out every aspect of it. And so, to me, I'd already been experimenting with pulling the models apart, talking to a lot of different people in that open source space. So not just like developers, but other researchers. I mean, remember, talk about another Discord, the, you know, the Luther Discord back then. At that time, I met a lot of people who I was just learning from and I was reading a lot of papers and just trying to uptake and understand for myself.

15:43Joining Stability was also another major point for me. Originally, I didn't join as a researcher. I joined as an engineer. But all of my free time I spent with essentially any researchers I was able to make friends with, you know, shout out to Dr. Scott Hawley there who worked on the audio team who mentored me quite a lot there. And at some point, I was able to sneak my way onto the cluster there and got myself access to, you know, some idle compute. And I really just, like, I treated it like I treated how I learned most things, which is just open source hacking, trying things, reading papers, trying to implement different pieces of work and just seeing how, I don't know, how models train, how they learned.

16:20And, I mean, that was really it. Like, there was no magic to it other than just I was excited and I just wanted to learn about it. I wanted to tear it apart until I understood how it works. I mean this is open source at its best. I think that's generally how you learn best just trying out things. And he was also Luther? Yeah, yeah. Affiliated but not on the image end side. But back then everything was like very hackable, right? Yeah. The delta between open models was like pretty drastic between what was closed and good right so people on language model side were doing post-training that's where like a lot of the news guys started in this you know you can look at early stable diffusion this took so

16:55where we get to today it's pretty crazy but you both had pretty you know impactful roles like you did sdxl You did latent diffusion. We'll get through all the history. But for people that haven't super followed the space, where are we today? What's the state of things? I know Flux 3 video is out. Flux 3 action, we're now like 7B doing robotics. What's the latest in the space with you guys? We started as an image generation lab when we started Black Forest Labs. As a kind of legacy of stable diffusion and the likes. And that has been, I think, our thing for the first year. But then we realized, okay, you actually, image generation is nice, but this technology can unlock so much more. And that's when we slowly went into

17:42what we call visual intelligence, which is basically trying to get our intuitive intelligence that we people have when we say when we catch a ball and just understand physics without actually knowing how to mathematically describe it or something. Or when we grab a bottle and just know exactly how to do this because we've done it 100,000, maybe not 100,000 times, I don't know, but a lot of times in our childhood, right? We want to be transforming that knowledge and that form of intelligence, which is very different from language, model intelligence, into models. And learning based on pixels, which means not only images, but also videos, and maybe also audio, say not pixels, but natural signals is maybe a better term

18:25that really unlocks this and that is what we realized and we realized hey we have the algorithm to actually do this which is diffusion or flow matching and that is when we just said okay we need to extend from image generation to basically all the natural representations which led to flux three basically but just one model that learns multimodal from images, videos, audios, also actions. We can talk about how this comes together later. To really get this, to keep it with Daniel Kahneman, I don't know if you read Thinking Fast and Slow. It's a system one intelligence. It's not system two intelligence. And you could maybe say... Everyone is system one models now. Maybe language modeling is system two. But we are focused for now on system one, which is extremely 1 models now. Maybe language modeling is System 2,

19:05but we are focused for now on System 1, which is extremely important for everything that is intuitive and that requires real-world interaction. That's why, on a high level, it's also so good for robotics. So that's been the journey, and now we're really focusing on that. And in the end, obviously, it's about combining both of these systems into this broad, very, very broad way of intelligence that humans also have. I want to go into sort of history and all these things. But just to also recap for people at the front, you're already making significant money as a generative media company, right? I don't know what the numbers are. And maybe you can sort of recap there. You raised a ridiculous amount of money.

19:47We raised, I think, all in all, $450 million in funding so far. So there's way more than stability. Any revenue milestones that you've announced? Currently, the last that was published in December was 100 million revenue we've crossed that since I can tell you that clearly clearly have crossed that if you guys 450 yeah exactly so you know yeah anyway I'm just saying like you did not have to pivot and to be clear we're not pivoting we're just extending like this model is really it's trained on everything it's always been the plan and obviously to create artworks you also need this visual kind of intelligence to create a video that we see here, you need to know how physics play out. I need to know that the model needs to understand, this is actually interesting,

20:34but the model needs to understand that the arm of this capybara or what it is, is not just falling away, that it has to follow some law that is described by physics. It's just not describing the law itself, it's just describing the output. In a similar way that we humans don't understand physics, but we understand that if I let a ball fall, it just falls down, right? Yeah. Well, I think there's different concepts of a world model, and obviously that's something that we can get into, where can you learn, this is sort of the bitter lesson version where you just learn on a bunch of input sources. Or you have to actually physically structure the 3D model and kind of learn from that. No 3D model at all.

21:14Just learning based on natural data. We also have a bit of triangulation from our eyes. We have two eyes. And that gives us a kind of slight understanding of 3D perception. But effectively, to a very large extent, even if we would close one eye, we could still kind of learn a lot about the world. That is what these models are basically learning. Like, you just learn from natural data, 2D video data, that obviously has a lot of 3D understanding because if you want to generate a pan around an object, you need to be hallucinating how it looks from behind. And if I look at you, I have a pretty good understanding of how your back looks, basically. Yeah. Because I've just seen it before, right? So this gives you an implicit 3D understanding,

21:54but it's really trained on kind of 2D videos. Yeah. I'm trying to steel man the counter argument, the opposite argument, right? It's like the Gary Marcus, like, oh, let's don't learn anything. Or I forget who are the famous critics of this kind of stuff but basically it's like it's like i have never lived i as a model i only see on inputs i've never i don't have personal experience you can model my back because you are also a human but and you can model other humans but maybe maybe this analogy isn't isn't exactly correct because i was obviously i'm sympathetic to your your side of the story but like i think that is how people criticize this approach to world modeling which is that yes you can approximate closer and closer closer but you'll never quite get something core

22:34to due to the existence of the thing because you never modeled the thing you modeled uh uh the most likely you know uh sort of of constraints around the thing. Yeah, I mean, I think to some degree you could argue that we operate under the same principle, right? Like, you can say, like, that we are, you know, okay, we have this embodiment and we're constantly, like, taking in data and, like, obviously we regularly update. But in some principle, like, we're still just taking in these, like, filtered signals from the world. I mean, there's all sorts of wacky things with how our eyes process and we have to, you know, part of our brain kind of has to hallucinate some of the info because we have the blind spot

23:08in the center and other aspects of this. But fundamentally, like, I don't think there's that much of a difference between, you know, how we, like, we're out here taking in this natural data and building this model of the world that we're able to act on versus, like, these models also learning from, obviously, the signals are different that they're, you know, okay, they're turned into pixels or, you know, into waveforms and processed through this other mean. But like the information is still there for the model to take in and build this like representation of the world that then you can essentially operate with. Yeah. I was actually arguing this with Lukas Beyer. He's also, he's with Meta now, I think,

23:45formerly Google and OpenAI. He's in Zurich, which is pretty close to Freiburg, and he's coming over from time to time. We were debating this on human learning. Like he was saying, okay, we actually don't need interaction, which would be kind of the, I think, what I would say is important for learning like higher forms of intelligence. It's like, first it's observation. That's also how we humans learn when we're babies, right? We just lie around first and we observe things and from that alone, we learn so much. This is what you can actually exactly map to how we're currently training these models. They just observe a lot of 2D videos, audio, and then they learn from it. And I think next step would also be closing the loop to actually interaction with the

24:29world, because that's also in how... I think that that's very important in how we learn, basically, around the world. We can then tap, and we see, oh, there's a correlation between I knock on wood, and there's a sound attached to it, right? I can just manipulate the world then to understand it better. That's how I think think humans are mainly learning, at least how I personally learned. And he was actually saying he thinks the majority is observation, which I found very interesting because if that would be true, then we just need to do what we're currently doing more, and that could basically unlock an insane amount of intelligence. I wonder how much of this is like emergence with good priors, right?

25:07We've seen adding new modalities to say language models. You don't get like the full benefit of, you know, doubling your performance since you've added vision to a language model. But if you have good priors, you know, is there light benefit? Is there a big benefit? It's interesting to see. I think there's a lot of technical stuff that'll be interesting to walk through. Like Gaussian splatting was a big thing. World Labs did a lot of 3D data. Where are the trade-offs between video generation output versus what else? But I think something fun for listeners of the podcast would be the history. You worked on very early stable diffusion, SDXL, latent diffusion even before that. Video diffusion as well.

25:44Video diffusion. Basically, there's a lot of buzzwords here. You know, LCM, LoRa, flow matching, old prompting. Can you teach us everything? What's interesting? How big are these? I've actually built a tech tree. I wanted to build this game where it's like a civilization, but the tech tree is AI. Like key AI papers. So to spend research points and get it. And so these are the tech trees, right? First it's GANs, then dot, dot, dot, like the end diffusion happened. But there's a lot of core points beyond that. I'd play that game. Okay, anyway, just to pitch the game, it's like you role play as a neolab, and you have to spend points on, like, GPUs or researchers. And then you have to collect data,

26:32and, like, there's all these, like, little things that you can sort of research. Come on, whip it up. We're in the vibe coding world. I want to play it tomorrow. I think game balance is really hard because I had AIs play. I built it. Oh, fantastic. But, like, game balance is really hard because I had AIs play I built it oh fantastic but there's like game balance is really hard it's actually like an AGI hard problem I feel like because you know once you find a way to win you just always win and it's not interesting as a game you need multiple ways to win and like you need high risk ways to you probably lose but maybe you win yeah is a great way for people that want to get into the space. Good interview questions if you're hiring as well. You should know most of

27:26these things. I think there's some points where there's spicy, you know, different paths that's popped up. So, autoregressive image gen versus diffusion. Two backbones or glass diffusion. All these things. What are the key things people should know? I'll leave it pretty blank. I mean, first would be SDXL maybe. Talking about SDXL a bit. I was going to say, what were the things first would be SDXL maybe. Yeah. Talking about SDXL a bit. I was going to say, what were the things we did in SDXL? The refiner. Oh, no, please, not the refiner. Wait, why that reaction? So SDXL was a big piece of work in trying to make the models feel good to use for the first time, if I was to kind of like lay anything out.

28:06Not to say like Stable Diffusion 1 and 2 weren't, you know, I mean, they were very exciting, they were very hard to use, but to anyone that was, like, in the space at the time between the period of SD, you know, SD1 to SDXL, we saw a ton of work out in the open source community working on how do we fine-tune these models, how do we make these models, you know, more aesthetic, how do we make these models into, like, you know, multi, you know, if anyone remembers, like, the original Stable stable diffusions were only able to do square images. And so SDXL was the first work of trying to actually release like the base foundation model kind of with a lot of these general improvements implicit in the model.

28:37So SDXL involved, you know, the first time we did large scale multi-aspect training. There was a, you know, at the time, this was before packing. There was, like, an efficient way we could do essentially bucket-style training. So training, you know, on all sorts of different resolutions of images and, you know, at large scale. This was the first time we did multiple actual training stages. So taking it from pre-training, actually developing, like, post-training sets. So there was a lot of data work that was done in SDXL. I'm trying to think like all the other aspects. And then I'll give you kind of like a little bit of a backstory on the refiner and maybe some things that didn't quite make it in.

29:15So here was the whole process of the final release model is essentially SDXL fundamentally works similar to the original stable diffusions. It was a latent diffusion model, took in text. It had, you know, you had a noisy latent, took the context of the text, actually generated the, you know, generated the imagery in latent space, and then a VAE decoded it. And then from there, the refiner was actually another full diffusion model that the whole point of it was to take essentially what was not a rough image, but SDXL wasn't the best at what we would call high frequency detail, which is this like the fine detail, like skin texture, texture of grass. Yeah, upscaling. Upscaling. It wasn't an upscaler because it didn't actually change the resolution, but it, yeah, upscaled in terms of the, it refined it.

30:00The reason I thought it was janky is because I worked, not just me, the whole team that worked on this was fantastic. I don't want to take all the credit at all because there were a lot of fantastic people that worked on this. We did a lot of work to try to push the detail quality of the model while trying to keep this like generality. And we just didn't quite get there. And so then there was this like sidetrack of work of like, OK, what are other methods to do this? And this is where the refiner came into being with actually – the refiner was actually a model that was – it was what was supposed to – so this is a little bit of the secret lore is there was a version of SDXL that was never released.

30:36It was a different architecture that didn't quite work as well, but it was trained pretty far along, even to, like, the post-training stages. The refiner was that model. It is a full diffusion model. Actually, I completely forgot about that. Yeah, yeah. Like the refiner was a full diffusion model. And the one thing about that model is it had a little bit of better performance on these high-frequency details. It just had very poor structure. And so this whole idea came of like what if we essentially like— Different model, different job. Different model, different job. We took it. We did a lot of work to really push the work on how well it could actually refine this high frequency detail and then we made this this essentially this pipeline yeah which is the excel

31:13part which which then became okay this together is like a more performative model you know or a more performative system altogether and then that was what we ended up releasing here which is actually interesting it was one of the early pioneering works of like actually that looked into, okay, what kind of properties learned in which part of the trajectory of this diffusion kind of trajectory, right? Because you have these structural things that are learned closer to the noise, if you come closer to the noise, and then you have the fine high-resolution, high-frequency stuff that comes further down and is closer to the clean image, basically. And today, there's a bunch of different works that actually analyze this.

31:51But you do actually, you generate from low frequency to high frequency if you go from noise to clean image, right? Along this trajectory. But that was not really known. It was informally maybe a bit known by then. And this was one of the first works to basically exploit this and say, okay, I actually dissect this kind of trajectory, and I have two kind of experts, one for the structure and the next for the texture, basically. And right now we have modulation, which is conditioned on the time step to actually account for that as well, and stuff like that in the architecture, but that was really one of the first works to really make this distinction, and as you see, it worked right. Yeah, it did work quite well.

32:30I guess the reason I still hold frustration, because at the time I wanted this one model to be so good at everything, and I felt... It was a cop-out. Yeah, maybe it was a little bit of a cop-out. Nowadays, I take a bit of a different view, because we see systems and models do incredible work together, but at the time, I remember feeling, of a different view because, I mean, we see systems and models do incredible work together. But at the time, I remember feeling, and I still just have this like kind of old frustration, you know, just from back then. Which is also very fair. Yeah. Because you should always go with the simplest solution that works. And if you then have a kind of a, you need to understand at which point do I need to involve a refiner.

33:03Yeah. point do i need a machine learning refiner approach yeah yeah it's just like when it comes to scaling whatever the simplest solution is that works you should go with kind of a bit of occam's razor uh as well and statistics basically so you know runway was also kind of doing their gen one and gen two and uh anastasis on this part also said admitted that gen 2 was uh pipelined like that works. They just didn't admit it. Whereas yours was public, so you had to admit it. I think if you go back to the time too, like people, I don't, people coming into the space today haven't seen 200, 500 steps of diffusion, right? Like, as you say, like you have a background and then minutes later you have little fine

33:40details coming in. It was also very tinkerable, right? This came out as what like a 3b works on eight gigs of ram so as much as you could bit or less and do the general approach if you're not a lab and you're just doing this at home you you have like one or two gpus you're gonna do pipeline stuff right it's like that's how you can see the differences right there yeah yeah uh so video i've been working on video for quite a actually already before stable diffusion i was working on a video generation for nvidia um had a paper there align your latents where we basically mapped the was actually in parallel to stable diffusion we basically mapped this latent diffusion algorithm to video which was the the natural next thing to to do it was

34:23very exciting to kind of then advance the things moving. And you can imagine the kind of compression reasoning behind why stable diffusion was so efficient compared to other models in the time. Also holds for video where you actually have this temporal dimension that you can also compress, right? So that made a ton of sense. Worked on this first for NVIDIA. And then when we basically joined stability, basically basically after i think after sdxl um we worked on this this um stable video diffusion which basically first applied this this latent diffusion um approach to video as said but importantly this work was a lot about investigating the effects of data filtering which as we felt or or as I personally felt

35:06back in the day, were like in research, and they still are, very much underappreciated. Data is just like when we think about what these models are doing, effectively we're estimating a probability distribution and the data set is what gives rise to this probability distribution, right? And if you have the perfect algorithm, the perfect algorithm, the best algorithm in the world that just learns anything you want, but your data is very bad, and you have, on average, blurry videos or images and not a lot of motion in the videos, your model will learn that, no matter how good your algorithm is. So what comes out of the model will actually be very bad and not nice. So the data is one of the most important

35:45levers that you have as one who wants to train these models and in the entire research field there was never much talked about data because a lot of people from the research community they just worked on benchmarks and like tweak the algorithms but if you talk about frontier models obviously we know that by now that that data is extremely important, right? And that is what we wanted to be really sharing more context on. So we investigated how much data, and then you have this trade-off, I want to be training on a lot of data. Quantity versus quality. Exactly, where's the sweet spot? That was what we investigated there. And then also, a very interesting chart. Is there a chart we should pull up?

36:24We can actually take it, the most exciting experiment that we did, it was not the data one, but actually, if you go down a bit, we then, and it comes back to what we discussed earlier, we then showed that just pre-training on plain 2D videos allows you to fine-tune an explicit 3D model extremely fast from it. So if you go to the last paragraph here of the paper, here we go, these are these 3D views here of the model. So what we actually did is we took this video model that we trained, and then we had, I think, 1K fine-tuning steps on 3D data, and we got a 3D model out of it. And that comes back to the question, okay, what do you actually need to kind of understand the world in 3D, right? Only one case that shows that this video model,

37:11although only trained on a plain video, actually have this insane amount of 3D understanding. And you just took a bit of a focus to get a 3D model out of it. And that is really like, that was one of the early experiments where we saw, okay, this is is actually kind of you can get world understanding through training on plain video and like from a research perspective this was like i think the most interesting experiment um in this paper also pretty short paper there's like a section here where you even explain the pipeline right there's like three four pre-training, some post-training. Yeah, stage one, image pre-training, video pre-training, video fine-tuning. Exactly. 1,000 steps, you can also make it 3D, right? Yep.

37:54Crazy. At the time, I don't remember what I did, but there was basically this temporal consistency problem where if you just stretch out diffusion and try to stitch frames, it wouldn't work. So a very old hackathon project where I forget the method we even used, but we fixed this temporal consistency to scale it out to 15, 20 seconds. It took forever, but it was a very hacky solution because you could do this in an afternoon. But critically, you need this kind of learning over temporal, kind of over the temporal dimension. And then if you just stitch together kind of a very fast diffusion model, you have this flickering. Even if you fix the seed and you do a shit ton of effects, but you realize you need to,

38:33like currently, today everyone is using transformers. So in the transformer speak, you need the attention over the temporal dimension as well to kind of learn from that, right? And then understand what is actually, how does consistent motion look like, basically. And as far as I understand, we asked this on the Grok Imagine part as well to kind of learn from that, right? And then understand what is actually, how does consistent motion look like, basically. And as far as I understand, we asked this on the Grok Imagine part as well. There's no compression in the video side that is useful, right? You just have frame, Exactly. Which is the simplest version. But it seems the most intense. Well, if you look at

39:01the MP4 codecs, you have very smart compressions, but that is lossy, obviously. And you need to kind of, you need to connect the frames in a way and like extract how kind of consistent motion looks like. Yeah. I don't think any video model exposes this, but kind of how far you look back in the temporal attention, because sometimes you do need longer or it needs to be more adaptive maybe. For example, if something gets occluded and then it comes out again, you want them to be the same. But what if, how long is it occluded? I mean, principally that depends on the problem you want to be solving, right? If you want to be creating a kind of a long memory model that can create your virtual worlds,

39:43then you obviously need to find a way to look back very, very, very long. Super memory hungry. And that is just your virtual worlds, then you obviously need to find a way to look back very, very, very long. I'm super memory hungry. And that is just, I think, also one of the most exciting research problems to work on these days. Long context is just extremely important. I don't know if there's more there before we cut over to Black Forest Labs. Anything else in the stable world? I mean, there's SD3. Yeah, SD3. That was the last paper that you did before. Yeah, SD3 was, I I felt like a pretty big paper. That was a, I think, a culmination of our work of instability, I would say. Yeah, absolutely.

40:13We got... Your graduation thesis. If you want, yes. No, this paper was really about understanding more in which part of this trajectory from data to noise does the model actually learn what properties and where do we have to spend most of our computation that we put in training to maximize the performance of the model. So before, and obviously that was also the transition from classic diffusion to flow matching was a slightly different paradigm basically. You call it rectified flow. Exactly. This was also the first time we used, like, diffusion transformers, right? Yeah. And then we took it a bit further, yeah. So lots of changes, like going from a convolutional architecture to a transform architecture,

40:56changing the probabilistic model from a diffusion to basically flow matching, which is way simpler mathematically, which is super, super nice. I think you should actually define some of those terms for people that haven't heard. Latent diffusion, flow matching. How did you come up with the concept of these things, right? Well, he didn't come up with flow matching. Exactly. We didn't come up with flow matching. We just saw this algorithm as something that is in a way more general because it can, like for diffusion, you always need, one of your two distributions between which you're basically translating needs to be a standard normal. That does not hold for flow matching anymore. And instead of approaching this kind of problem

41:36to transform one probability distribution into the other, seeing it as a solution of a stochastic differential equation, you're a seeing it as as a solution of a stochastic differential equation you're actually seeing it as a solution of an ordinary differential equation which simplifies everything completely like mathematically completely and makes also fast sampling much more um feasible you could say because what we always do obviously is when we have trained the model we want to be making as many steps, as little steps as possible to basically sample as fast as possible, right? Some people would say it's actually a special case of diffusion, flow matching. Others might say, okay, it's actually a generalization

42:15of diffusion, the flow matching algorithm, because you go from, you can translate between two distributions, no matter how they are shaped, and you don't need this kind of standard normal, basically, to be one of them. But in general, it can be seen as, again, a model that learns, basically, a vector field that translates between two distributions. And once the model has learned that vector field, you can just integrate it numerically, which is then as easy as solving a ODE by means of an Euler forward step, basically. So it's very similar in many respects. But it was just like, I would say, mathematically easier, more general from the kinds of probability distributions that it can translate between.

43:03And that's why we basically went with it. And maybe as a third-parter said, it's also easier to derive a fast model from those flow matching models, basically. It was much simpler and more scalable. It was much, much easier to work with and then had this nice performance benefit as well. So, I don't know, the diffusion space got quite complicated. Are we satisfied that there's no loss? Is it lossy in acceptable ways? Lossy? Lossy compared to full diffusion? No. Yeah, okay. So that is good. It's a strict improvement. Exactly. It's great. It is an improvement, definitely. Yeah, I mean, it's effectively just drawing a line between noise to data, really. It kind of took this, like, there was this complex space on how do you map between these

43:48two distributions, as Andy was saying, and it simplified it down to the simplest of just drawing a line. Because you're trying to do a shortcut, like, what if it's a bumpy distribution? Yeah, yeah. And the most important part of the paper, or one of the most important, like, on the probabilistic modeling side was actually looking at the trajectory from noise to image and which parts really matter and which don't. Because if you just like, we train this in kind of steps, right? We always on this part from data to noise, we only always train on one kind of step and then we approximate this direction in flow matching with a model. And if you just sample those steps in between data and noise, we only always train on one kind of step, and then we approximate this direction in flow matching with a model.

44:27And if you just sample those steps in between data and noise uniformly, you might actually be spending a lot of time optimizing for things that actually don't matter. So one central part of the paper was, where should we spend our compute? Which parts of this trajectory from data to noise carry the information that's most important? Because this is where you should spend the bulk of your compute at training, right? And oversampling these parts during training and then undersampling other parts that are less important. It's effectively like a further, like a much further, better exploration of part of what we talked about with SDXL, which was like the refiner. We spent time like training the refiner

45:03on the end of this trajectory to learn these high frequency details. And this was like more of an exploration of the entire space of where like you should spend time to make the most like general good model to learn the data. Yeah. Amazing. One thing I did pick up with SD3 was text in images, which you guys showed off a lot. There was a time when images could just straight up not do text, and then they did more and more text, and then now it's like, you know, here's our nice typography. And I feel like, can we attribute that to the diffusion transformer? I don't know if I could say it was the diffusion transformer. I mean, I think it really was just better modeling and better data. Like, it was fundamentally, like, as much as I want to, like,

45:44as much as, like, I don't know, you always want to attribute some fancy thing, it really was just that it could model the world better. It's a data thing. It's also, like, what you always need, like, for everything that you train on, you need a caption, right? If your caption doesn't describe the text that is visualized somewhere, your model will not learn it, right? So that's also something that is often... Sometimes you spend a lot of time talking about the nitty gritty details of some algorithm. In the end, it's again, the better lessons, right? Simple, scalable is almost always the most effective. I would say always the most effective. But you can obviously scale in a smart way. And still, if you spend tons of compute,

46:21you should be spending it as efficiently as possible and only where it matters. And that is also at the heart of this paper, basically, right? I think prompting also got a lot easier at this time. We used to have the 4K. Prompt expanders. 4K, Ultra, 8K. Yes, yes, yes. Who was the Greg guy? The poor artist. And now you can use him. Greg Rakowski, yes. I wonder how he's doing. I hope he's doing well. I hope he's doing well. And also, you know, for context, this is when Dolly 3, Mid Journey, Ideogram V1, nothing was open. So anything that you would see of this quality was either old stable diffusion or pretty much a closed tool. Then you had to make the big break to Black Forest what was that whole

47:07period like like basically you had to leave and start a lab from scratch I think it was probably one of the most exciting times we all had because when you're in stealth you're just like you're a small team we were 13 people I think when we started we first went to France had a fun little week in the mountains in France in a chalet. Spent the entire week just hacking, hanging out. Dustin figured out some really cool stuff to make our models way more consistent. I remember that. That was so fun. That was a magical week. That really was our kind of founding offside. That is when we started training Flux 1, which three months later resulted in this nice company launch, basically, with a model that I think blew

47:53pretty much everyone away. At least it blew us also away. Commercially, what this directly unlocked, like only having kind of incorporated the company for three months and then releasing this model, and it was like was like yeah just like was received very well right that was that was amazing uh so that was but like did you know that when you started this i mean you had to get all the the you know the clusters together like all the data that you had to start from scratch scratch we had started from scratch yes but we knew what to do, basically. Yeah, obviously. We had a lot of experience. But you just did SD3. Did you have a hypothesis that you were exploring of when you would do differently?

48:35We've obviously gathered a lot of knowledge in these entire times, even before stability as well, right? And that all culminated. And then we also had a team that just, like, was in one place, basically, that was very focused, which had never been the case before because, like, there's just, like, if you're working from very different parts of one company, also, as Dustin said, Dustin was actually not in our team. And that, like, we basically had a super nice team together, spent a lot of time together discussing how we would actually design the kind of learning process of this model, which was at the soft side. And then we just, kicked off and that resulted. Was it like discussions of what is Flux 1 or was it broader?

49:18Where do things go in a few years? How much of this vision was like? Well, I guess I can jump in and say like, I mean, one of the reasons the company was started, I will say there were some frustrations of time at stability. You know, we saw from our side, we saw the impact that our models had made out in the world. We saw not just the impact from, like, the open source world, but also we saw companies that were started and run and quite successfully used our models out in the world. Like, we knew there was a lot of draw and a lot of potential in what our models could do. And unfortunately, I think we also felt that at Stability that we couldn't essentially leverage the value of what the models could be as well.

50:00So not only did we have the belief in like our team's ability to actually do the research and perform and build these models and get them out in the world, but this would be our shot to kind of correct those inefficiencies or mistakes of not being able to actually capture the value of the models themselves. And so that was also a big part of the belief of why we thought we could, you know, go forward. Because we, I mean, we have this record of doing this and making this impact, and we figured that could continue on and we can actually make this, you know, successful company out of it, which now, I mean, I think it's gone pretty well. I think it was a good thesis. I think it was a good thesis.

50:34As I've said before. Yeah. We've been talking about how we can generate images and video and improve the perceptual understanding of models. But what about voice? And that's the subject of today's ad. We've been working with Deepgram for a while on the AIE side. And they recently acquired Pypecat, which has also been a big sponsor of AIE. So Deepgram is becoming one of the critical pieces of voice infrastructure for all agents. So this episode is sponsored by DeepGram. If you're building a voice agent, you need it to keep up with how people actually talk. That means understanding words in context and keeping latency low enough for a conversation to flow. That's what DeepGram's speech models and real-time infrastructure are built for.

51:13One in 5,000 organizations uses speech-to-text, text-to-speech, and voice agent APIs in production. The focus is accuracy, low latency, and cost efficiency so you can build and run voice experiences at scale. You can use the cloud APIs or deploy it yourself, including on-premises. And there's an offer on Flux text-to-speech, spend a dollar, get a dollar in credits, up to $500. To sign up for free or find out more, go to deepgram.com slash contextiseverything. Dgram. Context is everything. Thanks to Dgram for bringing us this episode, and now we'll go back to the interview. And what use cases were you seeing? I think a lot of people went straight to Hollywood. That's our goal. Or like Imad used to say,

51:51we will make Game of Thrones Season 8 again. I mean, I'm trying to remember this. I don't know if I'll be able to remember the name of this company. This might have even been before SDXL. But it was a company that for like a week or two, it went viral where anyone could take like a picture of themselves. And then it created it. The older version or so. The older version. And it used like the old Dreambooth style fine tuning, which was extremely inefficient. And then, but it like blewtuning, which was extremely inefficient. But it blew up. Everyone was using this. Everyone was excited about it. The models could make so many things, and we saw that there was a lot of potential in how everyone wanted to use it.

52:27And we saw where the models were going. At the time, they couldn't do typography. We knew, obviously, okay, typography is going to be important. And we eventually got there with SD3 and so on. It wasn't just one use case. It was, we saw that this was like a general technology that had a lot of potential and that was going to continue to grow and have potential in the future, which now is expanding into, you know, the physical world a lot more. And this was actually already the potential we saw that like. Yeah, yeah, absolutely. BFL was not founded to be an image generation lab. It was more of a kind of a matter of the flux moment being so good that we then capitalized on it as well because it came as a kickstart.

53:00But when we pitched the company to investors for the C-series, it was clear we want to be working on video because we see that as a... I think we shared this on a podcast actually back in 2023 already. When we were still at Stability, we were saying, okay, video modeling is actually the central means to understanding how our world basically works. Because again, what I said earlier, right? Essentially, we humans are also first observing the world, then starting to interact with it. And by that, we learn so much. So that was always the thesis. And then obviously, we had to kind of take this detour, which was very successful, I would say, to image generation. And then obviously we had to kind of take this detour, which was very successful, I would say, to image generation.

53:38And then you always face the issue of, okay, where do I spend my time on? That's unfortunately a thing we cannot get across, that the day only has 24 hours, unfortunately. But eventually, I think Flux 3 is really where we saw this going for a long time and what we wanted to be doing for a long time. And that's why we're so excited about like having the model out there, being able to create videos, being able to create images, being able to create audio, being able to act as a robot brain to actually kind of work in real manufacturing workflows. And there's a lot of stuff coming this year still that we have to announce on the physical AI side. But really, like, as Dustin said, our goal was always to create the most general technology.

54:24And we see this happening currently, which is why we're so excited, basically. Yeah, I mean, I think you guys also took the right paths to stay focused, right? Like, if I'm not mistaken, Flux 1 launched with XAI. It was their mid-gen model. Good partnerships. That's right. Before they had their own. But like, you know, that's a good way to solve do we build distribution ourselves. Even to the latest, Flux 3 Max is post-trained with FAL. They do a lot of inference distillation. They're good at that, right? So it's interesting. But it seems like you're expanding now towards you have, what's the latest one with robotics? Flux Reaction? Yeah, before we get there, there's one more one to cover, which is image editing in context.

55:04Sure. We have a K. Also, really nice history right here on the web. I was actually, it's weird. We're now at the stage of company that like things just happen. And I'm like, oh, we have this. But I was kind of sad Flux 1 wasn't And I'm like, oh, we have this. But I was kind of sad Flux 1 wasn't on there to view, but it's like the original one. Yeah, but it's fine. Pro was good. Even just the visuals here, like maybe someone will clip it off. Original stable diffusion index. You see the progression. But one thing that I do want to cover at some point, I no longer personally see the progression, right? You show me state of the art image video models. A lot of Chinese labs are putting them out.

55:41It's hard to eval. I know people care. I've talked to Sumit at BFL about, okay, no, actually people in the industry, aggressively A-B test industry, there's a lot that people care about on evals, but there's also a level of, it all looks very realistic to me. The rest is prompting skill issues. I think there's a lot that people care about on evals, but there's also a level of it all looks very realistic to me. The rest is prompting skill issues. I think there's a lot of little nitpicks. Like there were hands, there's still issues in text sometimes. There's still hands issues? No, there were hands issues. There's issues in perception and layout. Like I'm redesigning my living room. It's horrible.

56:14Controllability is a huge thing. I think controllability is the thing we need to solve like large scale. There's so many ways you can control these models and maybe that is a good segue into Flux context as well because that was a pain point that we saw for a lot of time. People were training so many Loras just to get this character consistency into models, right? Shout out to Replicate back then with all the Lor as you can. Oh, yeah. And then we had this amazing pitch of Dustin. It was also in Northside, also in France. It was another cabin where I, like, a lot of things happen in cabins. Also, I didn't even go to a cabin. We were really like, you should do more. And then we had this pitch of Dustin

56:51who actually kind of saw this potential and, okay, I need to get this, like, context. I need to be able to prompt not only with text, but also with images for image editing. Yeah, no, I mean, so I did this, like, I wish I could. Let me pre-hype as well. I've heard the story that Flux context was, like, you and one or two other people. Yeah, so for a long time, it was basically my side project, and then I stole one of our data guys a little bit into it and said, come, we're building this thing. project and then I stole one of our data guys a little bit into it and said, come word, we're building this thing. So for me, like, so yeah, as Andy was saying, it originally started as a pitch of mine at

57:22some cabin in France that I did essentially like a, I like to do these presentations every, you know, every offsite or every time we get together where I try to look at like the state of the field and where I think the field is going and kind of where, like what is something we're not doing or something where these models have a potential to do something novel and useful that, you know, maybe we're not looking at right now or maybe isn't solved. And so I actually did a presentation on, I know this is like the most overloaded word in the world at this point, but I was like, oh, it's the Omni, the Omni model presentation at that point, because there was a lot oh, it's the Omni model presentation at that point.

57:57Because there was a lot of research work out there at the time on, you know, other editing models. If we remember back to, I think there was the old prompt to prompt. And then there's, why can I not remember the name of this? The pitch was essentially these models are clearly learning like these deep relationships about the world. And this is going to keep coming up. It's about these relationships. And to me, you know, like the relationship of like, okay, this Coke can is sitting on top of a table. This bottle is sitting here. And you prompt the model and you go, okay, I want a photo of a dog wearing a hat. And it needs to know, okay, the hat is on top of the dog. It needs to have this general understanding. And my kind of theory in mind at the time

58:32was that since the model already has all of this like spatial knowledge, if we essentially build the right data that since the model already has all of this like spatial knowledge if we essentially build the right data and train the model like train a an already pre-trained model the right way we should be able to evoke essentially like general image editing with very very little training capturing these already like pre-built relationships of when things are are there when they're not there uh and how these, like, relationships, these spatial relationships inside of an image work. And then essentially what it became is like a three to form, I don't remember the timeline, of me here over in our SF office with the one data guy I stole, building out

59:08essentially just early test training data sets. And I remember the moment I saw that this was, like, clearly working quite well, when we made this effectively tiny data set of just very like high quality labeled, it was essentially describing like simple relationships in an image. And that was really it. Like it wasn't a way that you would prompt a modern like edit instruct model, it was just literally describing these relationships. And I trained the model for like, I mean, it was maybe like six hours of training for as it was, it was very, very short training run. And all of a sudden, like I saw what felt like a clear generality coming out of it. Cause it was, I think it was like a photography data set. It was just like photos of stuff. But even just from that, I was

59:49seeing like, I can put a logo on a shirt. I can put like, I can do, and it wasn't good. It was just like the model was clearly learning to attend between these two concepts and use these pre-trained relationships. And then I took that to our next off-site, showed results, and then it became this essentially more company-wide project for about, you know, a couple months where we really put effort in, brought people in, you know, really improved essentially. It was mostly a data set project. Like I'll say it was mostly a data set project like also i'll say it was mostly a data set product and then in the end of the installation and then there was a lot of there was a lot of very nice distillation work by axel that was done to speed it up then we essentially

1:00:25were able to put out put this model out in the world and i i was quite proud of yeah that was the first kind of model to really do this character consistentconsistent editing, right, in, like, high speed? Yeah. That made it, again, accessible to so many people. For most people, it's very common to generate something, chat with it, change it. Before this, image generation took quite a while, and you needed to recreate the image. You couldn't change the parent to be blue. Just getting the same character. It wouldn't be the same. You need to mask something that you don't like and then fill in the blank. I wanted to get away from the slot machine, essentially. Everyone had to do the slot machine, prompt and prompt and prompt and prompt,

1:01:00and it was very clear that iterative editing was much more effective. So my crappy coffee UI was that. It just slopped out 10 different versions. My favorite part, I still slop out. Like, I'm JGI, generate four, generate six, redoing something. Just make 10 iterations. And eventually, that one's good enough. slop out like i'm jj loving him i generate for generates like redoing something just make 10 iterations and and then yeah and eventually that one's good enough my favorite is that you pray to rn jesus this is a random number uh basically you know to me this concept of what is a foundation model is something like this where it's like okay you know all these little workflows that you guys do to like do

1:01:32that and do that and do that and then all these different tools we're going to replace that with one model yeah and that's that's actually what do that and do that and do that and then all these different tools we're going to replace that with one model yeah that's actually what we're building towards right and then that was where flux this was a very clear aha moment for me i don't know the i never tried prompt to prompt but i think uh to me the closest has come to the first context is quinn image which like the quinn folks are doing really good like they had a photoshop layer model yeah it was yeah no no i mean they're doing doing pretty cool stuff over there yeah it's interesting how much of this are doing really good. Like, they had a Photoshop layer model. Yeah. Yeah, no, no.

1:01:57I mean, they're doing pretty cool stuff over there. It's interesting how much of this just comes back to there's a lot of good data. Yeah, I mean, it was targeted good data for... Was it a good concept? I don't know how to even describe this here, but yeah. I think one thing that I'm surprised by is... So, you know, comment on stability one more time. Stable Japan, right? Japan had this, like, whole law of, like, we know, comment on stability one more time. Stable Japan, right? Japan had this like whole law of like, we will protect you if you train on copyright. Open anything up. As a country. And I was like, ooh, stable Japan. It's interesting. Nothing happened. I always wanted to visit that office.

1:02:28I was sad I never did, actually. I think on your Omni point as well, it was an interesting time. There were the two others. Gemini is obviously the latest Omni. ImageBind, which I interesting time. There were the two others. Gemini is obviously the latest Omni. ImageBind, which I didn't realize is 2023 for meta. They threw in one embedding. HeatMaps. HeatMaps. And Depth. That's a big one. Which, you know, I'm like, oh, very early, but like, it's cool. I think this work has progressed. There's definitely newer work on it, but like, I am you. You know, this is just another color, right? It's RGB, A, whatever that is. Yeah, yeah, yeah. Right? So like, how many colors are there? Oh, the fun thing, this was also all in, all out.

1:03:03So you can go text to heat map if you want. You can go IMU to depth. It's a big market for that. But interestingly, no video out, which is, I think, the kind of the central thing that we need to understand basically the world better, right? This is, I think, what's, as Ed has always been at the core of what we're doing. Yeah, okay. Well, before we leave the sort of data or IP discussion, is there a particular kind of data that is, everyone in an industry knows is like valuable? If people, if I have a data vendor listening, how do they sell to you how do they approach you it depends heavily on what type of problem you want to be solving that's just super important if I train a robot I obviously need

1:03:45a lot of eccentric data I think that's not a secret right because that is just how this thing will be egocentric I mean there's also a benefit for you guys right on the language model side there's a lot of people doing RL environments. Is there a parallel that people should come to? I'm trying to get you vendors. Oh, we have so many data vendors, actually, that are... Too many? It's a lot. Oh, okay. What is good versus bad? It, again, depends on which stage of the training you are and what your effective goal is. I think there's a trade-off between we had this already, right, scale of the data versus the quality. And that is what you have to, for every training run, like large-scale pre-training run you do again,

1:04:25you have to kind of find that sweet spot out because depending on the architecture you might be using, that might look very different again. And that is one of the central pre-training experiments or pre-scaling experiments that you need to do you need to do a letter basically um to understand at scale how much or to predict at scale how much data do i actually need and how should the quality look like so where's the sweet spot between like these two dimensions of data quantity versus data quality any other major jumps that we want to call out? So, Flux1 context was a big one. Flux2, FluxTools, Action, we talked a bit about. Any other? I mean, I think the three sizes, that's very important in terms of the dev

1:05:07models and the Schnell. Schnell? Schnell, yeah. What is Schnell? Schnell means fast in German. I think you probably covered that in the book. But yeah, I think that you have open source, you have the closed source, and you have the range of performance. I think you probably covered that in the book. But yeah, I think that like you have open source, you have the closed source and you have the range of performance. I think that makes sense. Yeah, absolutely. Yeah, I mean, I guess if like anything to track on is like, okay, the one like thing that made me quite sad that we didn't quite get in with Flux Context is that it was only single image edit because, I mean, it was very clear that there was like

1:05:36a huge need for multi-image, like people want to be able to put in many, many different images and references is a better word here. Many different references. Which I think Grog is like king right now. I don't know this statement. I know Midjourney does quite a bit on style books where you can build up references. They've always been very good at style reference, but this is more just like general like Flux context would only take one image, and so it was just, you could only iterate with that. So you could have a bad photo of one person, that's all you have, but you could have three different photos of the same person. But it was actually general enough, I remember it was actually quite funny,

1:06:06people were, like, doing workflows where they were stitching a bunch of, like, images into one image to, like, emulate it, but then with Flux 2, we finally were able to, like, spend the time to kind of build out the data and actually do this larger scale training for handling, I think it was 10 able to spend the time to build out the data and actually do this larger scale training for handling, I think it was 10 reference images at the time. Essentially building up much more context to work within. Which is, again, also a lot of to a large extent a data problem that you just need the data. Is there a practical limit? Let's say the industry standard is 10 characters or 18 characters. That there a practical limit? So, you know, let's say the industry standard is

1:06:37like 10 characters or 18 characters. That's it, right? Yeah. I don't need 100. Obviously, you can show crazy numbers on how many characters or like images you can get into one image or also video. But there are like, that is agent work then to get me 100 references. Which is the easy thing, right? It's agent work. But then I need to check it. Like, then I have to review 100 images. It's actually also a hard creativity problem. Like, where should I put these kind of 100 images? So I think without over-indexing on the number, everything that goes up, like over 10 references, is also hard to follow for a human, I think. If you say, okay, we want to be creating a video, right? I obviously can put in 50 people there but if you look at the move like the the most important

1:07:21movies how many central characters do we have like five six maximum unless you're avengers exactly that is then also it's at some point approaching the kind of limit of what we humans can actually kind of process in in a two minutes clip or something this also gets way way, like, coming back to the, I know you mentioned evals earlier, building evals for larger, like, contextual models becomes a much bigger process. Like, even for, we ended up releasing an open eval set for, I think it was ContextBench, which was built off the internal bench that we used to evaluate context. And then as you scale up how many references. Oh, I haven't seen this. Yeah, yeah. Please pull up context. This was a lot of work, but it was

1:08:01actually quite fun to put this together. Make the cat very fat. Yeah, I mean, I remember sitting in the back room of our SF office with like a few of the other guys, like me and Tim were laughing. Tim Dockhorn is another one of the great guys. So like, do you just take some edibles and just go like... We were sitting there like, do you just, like, take some edibles and just go, like... I mean, we're just... What's the process? We're sitting there late at night literally just typing in the... Like, we're just pulling images and typing in the stupidest things. I mean, it works. Yeah, that, like... But it's, like, real. Like, I don't know. Because... And in the end, you obviously need to kind of

1:08:29make one step to make it kind of... And wait, wait. What is ground truth? Yeah, you need to make sure that... We have a ground truth somewhere? Yeah. No, this is just a benchmark basically just inputs, right? But where's the how do you know? I mean that is the interesting thing, right? You just ask people. This is a public bench to give everyone Yeah, this is the evil and you have two models that you want to be Yeah, it's not a validation set in that sense? It's just an email set? This is just Dustin sitting in a room. Genuinely, like, 80% of this is that. Like, it's... Also heavily influenced by our, I think, by our Italy offside. Why don't you do, like, a Mad Libs, like, like a combinatorial explosion type thing

1:09:16that is more structured than just random exploration? So, I mean, now we definitely do more stuff like this, I will say. So one of the things is there were, I mean, there were certain things we had to like filter out for the public set on here because it's just there were some very dumb things in the internal thing, you know, in the internal set. But I don't know. I spend a lot of time. I like to see how people actually use our models. Okay. And I'm someone that actually how people actually use our models. Okay. And I'm someone that actually really likes to use our models. I think it's very, very important that you test and play, like really just play. I'll say play with your models. Like memes.

1:09:44Like memes. Like, yeah, like use it for what people actually want to use it for. Like I came, like I said, I came from Discord, open source communities on this. You see people doing the stupidest things, but it's like, this is people having fun with it. And it's so, a lot of the set, you know, besides trying to cover a bunch of actual use cases, like production use cases, which are very important, I also want to make sure it's just like, the models are fun to use. You know? Yeah. And so, yeah. Good vibe eval, vibe bench. Exactly. What I think is like, I underappreciated three years ago, which was like, yes, this is memes, yes, this is like a mid-30, whatever subscription. But like, it actually becomes very real work.

1:10:20Yeah, exactly. Absolutely. Because they've done it for fun. Then they're like, oh, I can do this. And effectively also like when you automatically eval these models, like there's a bunch of obviously human evals that everyone is doing in the field, but every evaluation will have their blind spots. And you should be combining as many independently as possible to get a picture that is as good as possible of how the model is performing. But that doesn't remove the burden from you. It's not a burden. It's actually super fun to then sample the model yourself and try to understand where are the strengths, where are the weaknesses. This is a core part of developing a model, is actually constantly sampling when developing it

1:10:59and checking, hey, is it where I would expect it to be, right? At some level, it's like what you guys care about too, right? Like, these are your use cases. It's a good way for, if you want something to get better, like, make a little benchmark and see how things perform. No, absolutely. I think it's very important that the people building the models actually, like, want to use the model. Like, I think this's very important that the people building the models actually want to use the model. I think this is very important. There's a big distinction in this and the microphone brand trying to use it for product photos. A very niche use case that they can min-max on compared to remove. Very general.

1:11:29Yeah. I think one thing I do doubt or question is that it was a very well-known mid journeys loop which is the it's discord everyone sees like other people prompting you get the four options you choose the four that is feedback for your next model how come you didn't do any of that we didn't do it publicly at scale so so every what isn't it valuable to do it can't be valuable but i think only up to a certain extent and if you then like cross certain limits, the signal that you get is pretty, pretty, like early days, we saw this insane potential, right? Of this tuning, say of majority models, like V5 or something crazy. But if your base model just reaches a certain, certain kind of level of strength,

1:12:11it's actually, you don't get these super good examples anymore where you can clearly see the model improving from this and this is also a lot of noise. So there are a lot of other ways how you can actually... I had this debate as well with someone where back then it was very useful because prompt adherence wasn't great, right? It's pretty judgmental on which one follows the prompt better when neither of them do it great, right? So you have a lot of preference tuning towards there's very high signal and okay, the people prefer this. Now back to you have a lot of preference tuning towards there's very high signal and okay, the people prefer this. Now back to my point a lot earlier, like if I take the top five models

1:12:43and tell them to remove a bed, it's all going to look the same to me. The preference, there's not that much signal, right? They're all filling it in pretty good because there's like, they understand this task very well. So that side of the eval for me is kind of gone. It all looks fine. But yeah. I was going to jump and say that was actually one of my favorite parts of moving into context is finally it wasn't just like, I don't know, it was finally just like, okay, did it actually just do the task? And I really enjoyed getting to these like very kind of clear concrete tasks. One more part of the tech tree and then we want to go into robotics and action. Autoregressive image gen. It seems like a big trend with both Gemini and ChatGPT doing it.

1:13:24Is that as big an impact in your mind as it is in ours? You know what I mean? It seemed like a very big change because then you could conversationally edit. It is the next obvious step from Flux Context, where instead of editing with an image and text prompt, you're conversationally working with a model. That, I would say, is not a feature of autoregressive models only. It's just based on the data you train on it. And you can get language capabilities into a diffuse model as well, right? Yes. Which we know is working. And as I said, long term, this is where we believe this will also go into and where we're working towards to also get more modalities including this language model, right? It's not a question of the algorithm there.

1:14:11It's just a question of what you focus on, basically. I think both approaches are extremely exciting and have a lot of strengths and a lot of shortcomings. I would say autoregressive models are way more data efficient. You can train them way better in data because for diffusion models, you have this trajectory from data to noise, right? And every training example you're training on is effectively, because you have this continuous path from data to noise in between it's giving you rise to infinitely many training examples because you can pick every possible spot to denoise from from this trajectory between data and noise for one example obviously the model would overfit if you would only train on this one example. Theoretically, it's just like one training example

1:14:56is giving, or one data point is giving rise to infinitely many training examples. Whereas autoregressive models can just train on all the tokens in parallel, basically, and get the maximum gradient from each training example. That's the training side. So there I would say autoregressive models are more data efficient. However, at sampling time, autoregressive models don't get across that. They have to autoregressively create one token after the other because they're iterative in this data direction, right? And diffusion models, you can just go down and distill 50 steps down to four. And by that, you can speed that up so much, which is on the autoregressive side just not possible because you have to autoregressively

1:15:38generate token after token after token. So these are just two... Wait, you make it sound like it's just worse than... So autoregressive generation is in that respect, it has a limitation. But on the other side, if Fusion has in the terms of data efficiency or training, there's also a limitation. So we actually, I think it's, I like both approaches. We've also been working on autoregressive models like long, long, long back and both are super exciting, I would say. So on data efficiency, autoregressive wins out though? Yeah. Okay. So scale. I guess one of the other big things that you see having talked to people that worked on both like Chachapiti and MidGen is how you can add in the loop URL like agentically.

1:16:18So models can do agentically, you know, generate something, reason on it, edit the image and continue that workflow. They can do that. But this is separate than pipelining in diffusion generation, right? It's all the one same model doing it where as you train that in on a data level, you know, you can scale that. I guess, is there a counter to this in diffusion or? There's no inherent limitation that you couldn't have a model that is diffusion-based do the exact same process that can do all of these aspects. Yeah, this is like really like I had the wrong, basically nano-banana happened and I was like, holy shit, like autoregressive is a thing, but you're saying it's not. I think, am I wrong? Isn't nano-banana diffusion? Nano-banana was diffusion. I was like, holy shit, like, autoregressive is a thing, but you're saying it's not. Am I wrong? Isn't it?

1:16:57I thought nano-banana was diffusion. I think it is diffusion-based. I think the auto, like, it is, and I don't have, like, insider knowledge here, so I might be wrong, but I'm pretty sure it's, it has the language component to do, like, autoregressive, but it isn't, like, I think it's what we'd call, like, a transfusion model, so, like, it's, you know, it can essentially do both, but there's no, yeah, there's no inherent limitation that says you have to do it one way or another, and it's not like we're... They're explorations. Yeah, yeah, yeah. What I'm getting is that it's not settled consensus, and I thought it was, that there was a wave of people shifting from diffusion to autoregression for MSG.

1:17:32I think there's a lot of exciting research on that end right now going on. I will say there's some very impressive work out there. Obviously, we've seen models work extremely well on that front. The same is going on for flow matching is also improving. I would say the debate is not settled yet. There's still significant research on both fronts going on right now. And I think it's important to watch both and try both, be excited about the... And finally, the same is happening for robotics as well. We have VLA's versus VAM's basically, which is super, super interesting as well. Yeah. The last other thing, I guess, is there's also the diffusion for text, right? So there's also some of that, so there's benefits.

1:18:09There's, of course, speed, right? We see that already. Like with text, we see it clearly, right? We got diffusion, Gemma, that people can play around with too. Will they ever reach parity? Is it just a matter of investment in money and data or like you know i would say a lot of it is data if it doesn't work why doesn't it work my simple answer is that we we reason left to right we don't reason you know and diffuse into into vision at the same time you like i don't know we with auto-aggressive models i think a lot of people like to focus on the the the you know step by step of the of the tokens but At the same time, you like, I don't know, with auto-aggressive models, I think a lot of people like to focus on the,

1:18:36you know, step-by-step of the tokens, but at the same time, it also can parallelize. Like, we can't just take in a whole book at the same time as well, which those models can. Yes. Yeah, yeah. And you can think in latents. Yeah. I totally get it. So, that's, I mean, that is the valid question. Like, how come diffusion for text never took off? And, you know, now it's like, slowly, slightly better, but like. I think it's just like a matter of which approach was earlier and has been becoming for language model the standard. I think as your point, you said if it's more data efficient and the world has been doing a lot of scaling, there's a side there. That doesn't mean that there aren't pockets

1:19:12where there's use cases that you can get really good things at a certain size. The example is like, yes, diffusion Gemma is pretty good and it meets a bar and can run really fast, right? If you want that, that's great. Yeah. I mean, on the autoregressive language front, it's, I think it really comes down to, it was simple, it was scalable, it was there early. It was like a very, I mean, it's clearly been a very successful trajectory. A lot of time and money and research has gone into making that work. There's right now a lot of very exciting research on the diffusion front, you know, and I'd like to separate the notion there because there's diffusion language is really two camps. And I think a lot of people tend to not realize this because there are discrete diffusion language models, which I almost don't, I don't

1:19:57like to call them diffusion models really. And this is where like the diffusion Gemma and like the Mercury code, like Mercury models live, which is, is more of this like masking style model where, where it jumps from a mass to a token, but it's still in the discrete space versus now there's a big wave of research right now going on continuous diffusion language models, which those are actually operating as like, you know, I don't want to say flow matching models, but they're operating in this like essentially denoising space. So more like… This was used for audio, if I'm not mistaken. The… Audio generation. Continuous diffusion. Continuous diffusion. I mean, continuous diffusion is what we use for…

1:20:33It's for everything. Visual modeling, audio modeling. My last fun question there is, do you have a good way to do like Jira PO style RL on pure diffusion text models? Yeah, we're doing this actually. Okay. So for Flux 3, we did this and it has had tremendous gains that it brought us. So Flux Action, this is coming to very, very recent launches and work that you're doing. How do you approach this? What is the feedback and impact so far? Yeah, it's actually like how we approach it. Basically, you pre-train on, again, videos, images, audio, everything together. And then you have this very general model that understands the world. And then actually you can take a bit of action prediction data. Basically, you add kind of predictions in the state space of this robot,

1:21:17which is basically a kind of a parameterization of here is this robot arm, which every joint will be described by a translation and a rotation, six degrees of freedom, basically. And then you predict that together with a video. And since the model can already reason about the video and what it needs to create or to fulfill a certain task in video space, it's actually very simple to then add this new dimensionality or this new dimension, this state space, how we call it, and then basically predict that as well, given a chunk of earlier video, condition it on that, enter an instruction. Basically, the model creates the next chunk of actions to basically fulfill this action. And this works extremely good.

1:22:00We can do it in the real world. Like for real world robots, we can do it in a simulation. Like computer use is also kind of a mapping of that to the digital world. And we can make the model play games, right, and stuff like that. So this is all enabled by this single Flux3 model. And then you see also that it's extremely capable because we have this very general pre-training, which is not only limited to, say, classic robot data, but actually all kinds of data that gives a model just this inherent world understanding that you also see then in performance. And being a diffusion model is also pretty speedy, as we see here with these distilled versions, basically. Would you call this a world model or world action do you have a term you like to use

1:22:42this this one is about an actual model i think the word the term world model i don't like to use it's so old i'm the same but it feels like a hominy to me at this point let's call this a visually intelligent model because it can do all these these basic things right i guess the the distinction that i see in these models is you're not just predicting video or world you're also prediction action states right so one is what is happening in the world what is the state but also what is the next action to predict so it's like yeah sort of two outputs right so yeah is there a target frame rate uh that you rate that is industry standard? Because you don't have to predict out that far. It's my idea.

1:23:21You mean in terms of the real-time factor? It's like 10 FPS, I think, what the model currently runs. And you're projecting out like 1 FPS or 5 FPS? Sorry, 1 frame, 5 frames, whatever, right? I would have to look it up. I think it's a small chunk of frames, I would say, like in between 9 and 21 frames that it then predicts again. And then the model is conditioned on the previous generations again. And when it runs on the robot, what the camera actually perceives, right? And then you basically predict this further. So it's a bit of a chunk. It's not running autoregressively, so to say, in time, basically, but it's a chunk prediction. What I'm driving at is this video prediction side, that is very much for predicting the results of what you do.

1:24:03But probably you need some kind of long-range planner existing outside of that model that will coordinate the actions. I mean, this is why we're discussing the language side earlier. Yeah. As it runs right now, what we see is that there's no high-level planner involved or something. This is just a model running on a loop. Some of the interesting things here are… This is what we did with Dyna Robotics. Yeah. Dyna has one. Runway has one. I guess talk about the fine-tuning side. So if I want to take your general model, fine-tune it for a new robot with a new task. Not something super crazy, but something that's out of distribution. It hasn't seen, say, it's my own home robotics. I can program in how to use a humanoid arm or whatever.

1:24:46What is that process like? How long does it take? How important is that in fine tune ability? ALFRED BORUCHYKWINENBAUGHEIRAWICZ- Also, again, since this model is so general and the pre-tune was so nice, it takes like compared... Basically, you need a bunch of hours. I think we showed one example where we compared it with, I think it was Palio 5, which was also open source, right? It took eight times less data to basically train on it. That was for an industry application. It was like four hours of data that we needed, basically, to get to a success rate that was good enough for this basic application. And is that a thing you kind of want to help climb on right now? We go back to the context

1:25:23thought. We shouldn't be doing fine-tuning. What we're targeting and what we're working on right now extremely carefully and with a lot of focus in context learning for robotics. You should be showing a robot an action one time, and then it should be able to do this. And this would be truly a superhuman intelligence. Like if you look at how long it takes us as children to learn walking, to learn using our thumb correctly to grab this can, for instance, right? This is like weeks and even months of learning that it takes us to do this. Our vision is that robotic autonomous robot should be able to observe any action once and then be able to do it. That is what would be a clearly superhuman kind of visual intelligence.

1:26:07And that is what we want to be reaching. What is the standard now? How many demonstrations do we need? It's around like four hours of fine tuning you need at least. But even four hours is pretty good. I would say 2027 will be the year where we will practice our learning. Mid-training is an interesting one in here too, right? So is this like I want to go from manufacturing to maybe medical stuff like filling pill capsules like specifically like different niches for example? Is that where? I mean the model is trained quite generally. So even the mid-train, like, before fine-tuning, it might be able to handle many of the... Like, even this version might be able to handle many of the tasks you throw at it.

1:26:44The fine-tuning is really just nailing... You know, in the robotics world, you want a certain level of accuracy to expect out of it. Exactly. This is just the standard plain, large-scale mid-training. Then we move on to this fine-tuning stage where we add the high-quality robotics data in, and then we get this further bump. But you see that already to a somewhat good extent. On mid-training, the model can actually do these tasks. I guess the other interesting thing is it's a 7B, right? Is that mostly just constraint from these things are deployed? Exactly. You want to be having a model that is as small as possible because you need to be able to act first on device if you want to be adding this to a manufacturing, basically,

1:27:26and then also in real time. And this is on the backbone of Flux 3? Yes. So it's like the image and audio, all that stuff. This is derived from this one model, basically. We have this multimodal model. That is really important to understand. This has been trained on a huge corpus of images, videos, audio. And from that, we can derive everything that we basically released this summer, which was Flux3 video, Flux3 video edit, Flux3 Action, Flux3 Image, Flux3 Image Client. We also derived an upscaler from it. We have a super nice 4K video upscaler, which is, I think, super fast also. Can only recommend. It's pretty good. And this is all powered by this one model. And this shows how training these very general models

1:28:11actually compounds then over time because you can derive all these applications from them, right? Is there a difference between a video upscaler and an image upscaler? A video upscaler, we talked about the temporal consistency. Obviously, if you would have an image upscaler that would have no temporal understanding, then you would have this flickering again, especially in the high frequencies. You don't see this too much when you look at two images side by side. You don't see that much. But if you switch between the images, that's one of the basic tasks that everyone who's looking at a lot of images does. Just opening two images in two taps and then switching left, right, you see every single difference.

1:28:48And that would just amount to kind of flickering in the temporal space if you wouldn't have this temporal component in the video offscaler. And that's what makes a difference from an image offscaler. Any notes on real robot evals and how to scale this? I see sponge and bowl three times. And at some level, if you look at our Dynapod, there are literally people just one-off testing robots because you have to test in real scale. But come on, two out of three runs on sponge and bowl. Just restart, do it again. You can be three out of three, right? It's a hard run. I think one thing that's constantly being worked on is just closing the SIM tree wheel gap. And this will get better and better and better over time.

1:29:30But to be honest, no matter where you do it, it's a painful thing. It's also painful when you look at the content creation evals. Like, ask three different groups of voters what a good image looks like and you would get you you will get three different answers like wars over it yeah but that's evils are just regenerative modeling evils are just hard yeah yeah is this already like how developed is the commercial side we We will have more of this coming. I cannot tell too much, but there will be soon. We can embargo news. BFL-powered robots in production, live. And in 2027 will be the year of robotics. Oh, absolutely. Okay, say more on predictions. Where do we go from this in 2027? How fast do things pick up?

1:30:15Single-shot learning. Yeah, in-context learning is a big thing. We'll see first commercial applications of these things in real manufacturers. That's why it's actually good that we have this German base, because there's a ton of industry data and obviously lots of potential for partnerships. People are really hungry for robotics, I would also say. So there's a good bunch of synergies that we want to be using to actually make 2027 the year of robotics. I'm very excited about it. The research on this is also moving very quickly, so I suspect that robotics will improve very quickly. What about the side of robotics is very physical manufacturing constraints? Like you can scale up tech, you can scale up models, but that's, it's pretty hard to affect, right? So

1:31:03predictions 2027, where, where, where, where are we realistically on robotics? Can I buy something off the shelf that can single shot learn? Like. I suspect like later 2027, like people will be able to get some, some quite interesting robotics in their house. I mean, I, whether it's, you know, is it going to be able to do, like, fine motor tasks and all this? I don't know. But I suspect, like, and again, this is just entirely predictive. I have no baseline for this. But I suspect next year there might be some interesting, like, you know, at least niche for, like, the tech side of people like us. Not like, you know uh the average person would be picking this up but some interesting potentially useful robotics in your house and for manufacturing

1:31:42i think 2027 will be uh a year because manufacturing is like more restricted right then then kind of it it isn't that long tail as like say autonomous driving and stuff like that right or or like consumer robotics we can just like do it you would ideally like to do this thing to do anything you want but for manufacturing i think there's a very good chance that we will see like a lot of applications actually being unlocked by robotics in 2020 very much so and that's what we are also working towards basically very much so awesome i mean exciting it seems like a very fun place you guys have robot labs image video models You're just missing the flying car. You mentioned Germany a lot, but you also have a pretty big SF office.

1:32:20I actually didn't know there was a big European side. There's a substantial SF presence. It's called Black Forest Labs. Inviting everyone to come over to our place, 410 Townsend Street. Super, super nice. And we're not a European company. We're a company with roots both in europe and the us absolutely we're acting globally basically that's fun we got you on because you're visiting in town and we're like okay i should yes yeah how can people help you yeah hiring yeah looking for i think just just uh supporting us in marketing talking about our models um helping us also here, I think in the US, getting our name out. In the community, we are well known. I think for like, in the broader society,

1:33:05not that much. There's the big players, right? That's also what we need to change in 2027, really kind of getting our name out because we have extremely strong technology. I think that's what we saw. We have a long history of having built the core pillars of what everyone is using, basically. It's now also time to get our name out to the broader audience. Yeah. The tech community, I think we're well-known, but apart from that... Tell your mom about Flux. So it's very interesting that the overall... Obviously, I'm a marketing guy, and I help companies in marketing all the time. You guys are very different, because generative media in general has a weird mixed reputation in the broader population, right?

1:33:42Like normally I would suggest, oh, have a TikTok or Instagram marketing campaign. But pretty much you'll be killed just because they hate AI so much, right? And so it's very interesting to find a niche where this isn't regarded as AI. It's just normal, a thing you can do with tech. But it's not AI. Because I think the moment you say it's AI, the population will just have pitchforks. I don't know if you have any responsive thoughts or explorations about that. I think it's important to just always also emphasize that we take this responsibly. Our models are not creating avalanches of slop or something. They're actually live in real use cases, lots of professional workflows and they just help making the lives of

1:34:27professional people who work with this way easier and just increasing productivity so much. It's Photoshop. Exactly. It's so crazy how much more productive you are then. And that's really the goal of this on the content side. It's not about putting people out of work, not at all. It's just making everyone's life so much easier. I wouldn't say that. I'm just like, if I worked on your team, this is my GTM problem right now, which is like, how do we get people to basically go viral in the same way that maybe Whisperflow goes a little bit viral problem right now which is like how do we get people to basically you know go viral in the same way that maybe Whisperflow goes a little bit viral because they're like

1:35:02oh yeah I understand how to use this it's not AI it's just Whisperflow it's Granola same thing I think that the generative media hasn't had that crossover except for Nano Banana and then whenever ChachiBT had the Studio Ghibli moment those kinds of like that is the crossover where it's like oh like this is a fun trend that i want to hop on there were moments that hit well like your point of the see yourself as you're older like that had a huge moment people would look at themselves as their 80s and it's the other family like there have been moments but nothing is super i think one of the difficult pieces is it's actually very hard to predict one of these viral moments. Like, Nanamanana was their internal name, and then they just kept it because people just liked the name

1:35:45and it stuck. For context, actually, the one big use case that we saw a ton of usage on that we didn't necessarily expect was image restoration as well. So there was a ton of, like, I saw amazing, amazing examples of people, you know, taking old degraded family photos and then restore, colorizing them, fixing them up. But, but, and it was able to keep like a very, very high level of, of similarity to the, to what the original would have been. And then, I don't know, that, that was actually one of my favorite moments that we've had of people showing like their family, these kind of restored photos and the reactions from it. And I really, really enjoyed that like positive moment that it brought.

1:36:24But like I said, like we – so often we can't predict like what the thing is going to be, you know. It's even with our old, you know, even back in the stable diffusion days, it was always surprising what the trend was, you know, online. It just kind of, they just kind of, they emerged. You kind of plan them. I think once you have them, you need to kind of exploit them, obviously. And you should be always asking what is the next big thing, the next capability that it has not been able, or like that models have not been able to do. What would that be? and then trying to bake this into your model making bets some of them will work some others will not work but this is just extremely important to kind of always have this mindset of hey what what is the next thing we

1:37:08should be working on that will just bring a lot of joy or like productivity increases to people right well that is the domain of research. You guys are doing an amazing job. So thank you for sharing your story. I hope more people come to know you as a team because they've seen a lot of the model buds. But I feel like this is also the first time I've seen, I've heard the full story of BFL. Thank you. Thank you so much for having us. Thank you so much. I mean, it's fun to finally have gotten to do this. Yeah, yeah. I mean, it's a year in the making. But also, you know, I bought a bunch of micro ducks, and I'm looking forward to them running around. Do you have, like... I want to get my hands on one.

1:37:47I feel like I missed the window of these being available. You have connections. I'm sure I can... Whoever out there can get me a micro duck, give me a micro duck. I'll get you a micro duck. Okay, perfect. I bought a bunch, because I know it's going to be a hot gift for Christmas. Yeah, yeah, yeah. It's like. But yeah, you know, robotics 2027, that's, you know, one year away. Like, you know, we'll see. We'll see where this all learns. But I'm very optimistic about this approach. And I think you guys have explained it very well. Fantastic. Yeah. Amazing. Thank you. Cool. Thanks so much. All right. Thank you. Cool. Thanks so much. All right. Thank you.

▸More episode summaries

The Daily cover art
The DailyWhy Do Some Memories Survive Dementia?2026-07-26 · 29 min
The Daily cover art
The DailyThe Iranians Who Want More War2026-07-28 · 29 min
The Daily cover art
The DailyEurope Is Burning2026-07-30 · 29 min
The Daily cover art
The DailyThe World Cup Final Is Here2026-07-19 · 30 min
The Daily cover art
The DailyKarl-Anthony Towns Knows the Knicks' Win Was Bigger Than Basketball2026-07-25 · 39 min
The Daily cover art
The DailyIsrael, AIPAC and Difficult Questions for Democrats2026-07-29 · 39 min
Transcribe your links. Paste a YouTube, Google Drive or podcast link and WhipScribe gives you the transcript. Transcribe a link →
New Latent Space: The AI Engineer Podcast transcripts by email
One email per new episode, unsubscribe in every message. Not shared with the publisher.