How to adjust the hyperparameter r in the Lora modules?

[pzp-dzd]

Thank you very much for contributing the source code of the paper.
When using Lora for fine-tuning, what is the exact value of the hyperparameter r in Lora modules, and how should it be adjusted when running the code?

[vishaal27]

It seems like they hard-code r=4 in their code, see here (self.lora_attn_dim is the rank I think):

PEViT/vision_benchmark/evaluation/lora_model.py

Lines 455 to 463 in be6fb43

	'''
	LoRA setting
	'''
	self.lora_moe_lambda = 1.0
	self.lora_moe_act = 'linear'
	self.lora_r_dropout = None
	self.lora_attn_dim = 4
	self.lora_moe = 0
	self.lora_attn_alpha=128