load and train with bf16，saved torch_dtype is float32

[gyangfan]

System Info

• transformers version: 4.39.3
• Platform: Linux-3.10.0-1160.102.1.el7.x86_64-x86_64-with-glibc2.17
• Python version: 3.9.1
• Huggingface_hub version: 0.22.2
• Safetensors version: 0.4.2
• Accelerate version: 0.29.2
• Accelerate config: not found
• PyTorch version (GPU?): 2.2.2+cu121 (True)

load a bf16 model then train with --bf16 and log show [Using auto half precision backend], the final saved model parameter torch_dtype in config.json is float32
1、why saved float32？
2、how do i load the trained model？if load with float32，Can I get a better output?

[amyeroberts]

Hi @gyangfan, thanks for raising an issue!

It might be useful to look at the usage tips for other models which were trained with bf16. I'm surprised that the torch_dtype in the config if float32, it should be the same as the model's weights e.g. like here for llama 2.

Without a script showing how you're creating, training and loading the model, it's not possible to know exactly how the weights and saving/loading is being handled.

cc @younesbelkada who knows more about the weight handling

[younesbelkada]

Hi @gyangfan
Thanks for the issue!
Note --bf16 will not load the model in bf16, it will just activate the bf16 mixed precision training, in that case it is recommended to keep the model weights in fp32 for more stable training. If you want to load the model in bf16 you need to manually pass torch_dtype=torch.bfloat16 in from_pretrained

[gyangfan]

Hi @younesbelkada， thank fot the relpy！
That's exactly what I did，to pass torch_dtype=torch.bfloat16 in from_pretrained to load the model in bf16. If i want to train the model with mixed precision，as you say，fp32 is more stable for training，so do I start training with model = model.float() to upcast the model to fp32? This is something I didn't see in the Trainer process，It seems to be that I load with bf16, that is train with pure bf16，Is this compatible with bf16 mixed precision.

Please correct me if my understanding is wrong.

[younesbelkada]

Hi @gyangfan
Thanks for getting back !
Per my understanding:

• training with bf16 mixed precision = loading the model without torch_dtype=xxx to load it in fp32 + activate the bf16 mixed precision training with bf16=True flag in TrainingArguments
• training in pure bf16 = loading the model in bf16. You will need less memory than the first option but training might be less stable than the first option
  Let me know if this makes sense (I am also not 100% sure here) - cc @pacman100 @muellerzr in case I missed something here

[gyangfan]

Hi @younesbelkada
Thanks for your explanation！
I shouldn't use the torch_dtype saved in config.json if i want to finetune the pretrained llm model.
The torch_dtype provided in config.json maybe not indicate how it was trained and saved，for example torch_dtype=torch.bfloat16 may show it was trained with mixed precision with bf16，but not trained with pure bf16 and not saved the weights in bf16，it just helpful when i load the model with from_pretrained and torch_dtype="auto"

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[shelfofclub]

Hi @younesbelkada
Thank you for the explanations from you and your colleagues!

At first, I hope to summarize the background.

1. When using from_pretrained without setting torch_dtype=xxx, the model will be load in fp32 regardless of dtype in the model weight files (such bf16 for mistral model files)
2. Without torch_dtype=xxx, set bf16=True or fp16=True in training args means mixed precision training with bf16/fp16. In this case, there are both 16-bit and 32-bit version model weights according to the doc fp16 in Methods and tools for efficient training on a single GPU transformers v4.46

Then, I have some questions.

1. When I load a model with torch_dtype=torch.float16 or bfloat16 and train it with trainer following hf code examples, is the model trained in pure fp16/bf16? According to #24819 (comment), is --fp16/bf16 fully ignored? Both fp16/bf16=True and fp16/bf16=False are ok and I will get a full half precision train?
2. Is pure half precision training still unstable currently? Is there any new technology to solve this problem? Or it is not able to be solved? I wonder whether any popular LLM is pre-trained in pure half precision?
3. If I want to load the model in fp16/bf16 and train with lora in fp16/bf16, what should I do? Use torch_dtype=torch.float16 and --fp16? In this case, which is the lora adapter is trained in? Pure half precision or mixed precision? Sorry, I don't want to use qlora and load the model in 8-bit int. But loading a 7B model in fp32 costs 28GB mem. That is impossible for gpus like 3090. So, I want to load it in half precision.

[aflah02]

+1 @shelfofclub
Did you find any answers?