Size mismatch for gpt_model.load_state_dict(model_weight, strict=False)

[Jamie-Cheung]

I use "python autoregressive/sample/sample_t2i.py
--vq-ckpt /data/checkpoints/vq/vq_ds16_t2i.pt
--gpt-ckpt /data/checkpoints/t2i/ControlAR/canny_MR.safetensors
--gpt-model GPT-XL --image-size 512
--condition-type seg --seed 0 --condition-path condition/example/t2i/multigen/bird.jpg
--prompt 'A bird made of blue crystal'
--adapter-size small
--control-strength 0.6"

But it report:"File "ControlAR-main/autoregressive/sample/sample_t2i.py", line 68, in main
gpt_model.load_state_dict(model_weight, strict=False)
File "/root/miniconda3/envs/varsr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 2153, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for Transformer:
size mismatch for condition_mlp.uncond_embedding: copying a param with shape torch.Size([2304, 1280]) from checkpoint, the shape in current model is torch.Size([1024, 1280]). "

It seems that the weight is not correct when "/canny_MR.safetensors" weight is loaded, can you help me solve it?

[lzm2275965881]

I use "python autoregressive/sample/sample_t2i.py --vq-ckpt /data/checkpoints/vq/vq_ds16_t2i.pt --gpt-ckpt /data/checkpoints/t2i/ControlAR/canny_MR.safetensors --gpt-model GPT-XL --image-size 512 --condition-type seg --seed 0 --condition-path condition/example/t2i/multigen/bird.jpg --prompt 'A bird made of blue crystal' --adapter-size base --control-strength 0.6"

But it report:"File "ControlAR-main/autoregressive/sample/sample_t2i.py", line 68, in main gpt_model.load_state_dict(model_weight, strict=False) File "/root/miniconda3/envs/varsr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 2153, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for Transformer: size mismatch for adapter_mlp.fc1.weight: copying a param with shape torch.Size([1280, 384]) from checkpoint, the shape in current model is torch.Size([1280, 768]). size mismatch for condition_mlp.uncond_embedding: copying a param with shape torch.Size([2304, 1280]) from checkpoint, the shape in current model is torch.Size([1024, 1280]). "

It seems that the weight is not correct when "/canny_MR.safetensors" weight is loaded, can you help me solve it?

This command is not used to infer canny_MR.safetensors. You can change it to depth_base.safetensors. Just like this:
python autoregressive/sample/sample_t2i.py \ --vq-ckpt checkpoints/vq/vq_ds16_t2i.pt \ --gpt-ckpt checkpoints/t2i/depth_base.safetensors \ --gpt-model GPT-XL --image-size 512 \ --condition-type seg --seed 0 --condition-path condition/example/t2i/multigen/bird.jpg \ --prompt 'A bird made of blue crystal' \ --adapter-size base \ --control-strength 0.6

[Jamie-Cheung]

I use "python autoregressive/sample/sample_t2i.py --vq-ckpt /data/checkpoints/vq/vq_ds16_t2i.pt --gpt-ckpt /data/checkpoints/t2i/ControlAR/canny_MR.safetensors --gpt-model GPT-XL --image-size 512 --condition-type seg --seed 0 --condition-path condition/example/t2i/multigen/bird.jpg --prompt 'A bird made of blue crystal' --adapter-size base --control-strength 0.6"
But it report:"File "ControlAR-main/autoregressive/sample/sample_t2i.py", line 68, in main gpt_model.load_state_dict(model_weight, strict=False) File "/root/miniconda3/envs/varsr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 2153, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for Transformer: size mismatch for adapter_mlp.fc1.weight: copying a param with shape torch.Size([1280, 384]) from checkpoint, the shape in current model is torch.Size([1280, 768]). size mismatch for condition_mlp.uncond_embedding: copying a param with shape torch.Size([2304, 1280]) from checkpoint, the shape in current model is torch.Size([1024, 1280]). "
It seems that the weight is not correct when "/canny_MR.safetensors" weight is loaded, can you help me solve it?

This command is not used to infer canny_MR.safetensors. You can change it to depth_base.safetensors. Just like this: python autoregressive/sample/sample_t2i.py \ --vq-ckpt checkpoints/vq/vq_ds16_t2i.pt \ --gpt-ckpt checkpoints/t2i/depth_base.safetensors \ --gpt-model GPT-XL --image-size 512 \ --condition-type seg --seed 0 --condition-path condition/example/t2i/multigen/bird.jpg \ --prompt 'A bird made of blue crystal' \ --adapter-size base \ --control-strength 0.6

Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:27<00:00, 13.76s/it]
processing left-padding...
prompt 0 token len: 8
prompt 1 token len: 8
Floating point exception

May I ask if this question has come up before？This problem should come frome
" index_sample = generate(
gpt_model, c_indices, (args.image_H//args.downsample_size)*(args.image_W//args.downsample_size),#latent_size ** 2,
c_emb_masks, condition=condition_img.to(precision),
cfg_scale=args.cfg_scale,
temperature=args.temperature, top_k=args.top_k,
top_p=args.top_p, sample_logits=True,
control_strength=args.control_strength,
)"

Or Could you please tell me which cuda and torch version you are using？