OOM when training

[LanesraL]

Hi,
Thank you for your excellent work on this project!

I'm currently trying to train on the blendedMVS dataset using 8x H00 GPUs (each with 80GB memory). I ran the training script via bash bash_scripts/train/examples/mapa_curri_4v_bmvs_48ipg_8g.sh 8, but I'm encountering an Out-of-Memory (OOM) error despite several optimizations.

To mitigate the issue, I've already:

• Reduced the number of views to 2.
• Add accum_iter to 12
• Enabled gradient checkpointing by setting model.info_sharing.module_args.gradient_checkpointing=true and model.pred_head.gradient_checkpointing=true.

However, the OOM error persists. What confuses me is that the model only has around 500M parameters—why does it require such a large amount of memory?

Could you please provide some insights or suggestions on how to resolve this? Thank you in advance for your help!

[Nik-V9]

Hi, thanks for trying out our training framework!

I believe that you would need to tune max_num_of_imgs_per_gpu. I would not recommend using accum_iter.

With a 80 GB H100, I believe the following params should suffice (we have tested 4 view training on our end with a H100 node): model.encoder.gradient_checkpointing=true, model.info_sharing.module_args.gradient_checkpointing=true, train_params.max_num_of_imgs_per_gpu=24