Flux inference og

[echosyy]

Describe the bug

I use the optimization.quanto package to call the quantization function. When the model are quantized to fp8, the speed is much slower than bf16. want to know why, thank you?

Reproduction

`transformer = FluxTransformer2DModel.from_single_file("https://huggingface.co/Kijai/flux-fp8/blob/main/flux1-dev-fp8.safetensors", torch_dtype=dtype)
quantize(transformer, weights=qfloat8)
freeze(transformer)

text_encoder_2 = T5EncoderModel.from_pretrained(bfl_repo, subfolder="text_encoder_2", torch_dtype=dtype)
quantize(text_encoder_2, weights=qfloat8)
freeze(text_encoder_2)

pipe = FluxPipeline.from_pretrained(bfl_repo, transformer=None, text_encoder_2=None, torch_dtype=dtype)
pipe.transformer = transformer
pipe.text_encoder_2 = text_encoder_2`

Logs

No response

System Info

X86, torch2.4+cuda12.2

Who can help?

No response

[asomoza]

how much slower? I think this is kind of expected but ccing @sayakpaul for more insights.

[sayakpaul]

Are you not popping your model to GPU? I don't see any placements.

[sayakpaul]

Also, going to move this to discussions as this is not a library issue.