face embedding fail in stage1

[novicemm]

when i train the stage1, face embedding is failed and it doen't generate the image as expected. I doubt it happened because it cannot extract face information successfully in data preprocessing stage. But I cant find how to fix the issue. Is there anyone who faces the same issue and please let me know how to fix it.

[shlroot57]

the same problem, i print image_prompt_embeds,but it‘s nan. ImageProjModel don't work???：
(Pdb) image_prompt_embeds
tensor([[[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan]]], device='cuda:0',
dtype=torch.float16)

[novicemm]

Change weight dtype from fp16 to fp32 in config file. It works for me.

[shlroot57]

Change weight dtype from fp16 to fp32 in config file. It wotks for me.

Thanks!!!
It can work now, but I don't understand why fp32 is used in the inference stage. I found that when using fp16 in the training stage, the face_embed is okay, but in the inference stage, the face_embed becomes NaN.