Latent upscale with FluxPipeline

[IAmMorrow]

I'm trying to implement high res fix (latent space upscale) with the flux pipeline but i'm struggling a bit. The goal is to do a first txt2img pass to generate a 1M image and have the best composition and performance while outputing latents.

Then a second pass at 2M pixels to add details while keeping the original composition.

I found out in the diffusers code that even though not documented you can use "latent" as an output type. I was then able to interpolate latents to a greater "resolution" using pytorch.

The issue is that I can't seem to use the resized latents as a pipeline input for the second pass. FluxPipeline has a "latents" input parameter but it complain about the tensor shape.

I'm a total newbie with tensors and couldn't find anyone doing that beside with Comfy. Anyone could point me to the right direction ?

[yiyixuxu]

the latents input is for pre-generated noise - you probably want to pass your latents as image no?

diffusers/src/diffusers/pipelines/flux/pipeline_flux_img2img.py

Line 620 in 8f2253c

image (`torch.Tensor`, `PIL.Image.Image`, `np.ndarray`, `List[torch.Tensor]`, `List[PIL.Image.Image]`, or `List[np.ndarray]`):

could you try and let us know? we should support it, but that use case was never tested, so I would not be surprised if it does not work as it is

[a-r-r-o-w]

I think he wants to do something like the latent upscale example here: #9231

I believe it should be possible with the pipelines already but maybe the latents are not prepared in the correct manner.

@IAmMorrow, after you upscale the latents using interpolation, you need to make sure to "pack" it again from a BCHW tensor to a BND tensor. Our pipelines assume that when you pass latents, you have already made them into ndim=3 tensors by calling _pack_latents.

Helpful references:

• diffusers/src/diffusers/pipelines/flux/pipeline_flux.py

  Line 596 in 8f2253c

  if latents is not None:

• diffusers/src/diffusers/pipelines/flux/pipeline_flux.py

  Line 526 in 8f2253c

  def _pack_latents(latents, batch_size, num_channels_latents, height, width):