[Model] Add PaddleOCR-VL Model Support #42178

zhang-prog · 2025-11-13T08:15:40Z

What does this PR do?

This PR adds PaddleOCR-VL model to Hugging Face Transformers from PaddleOCR.

Relevant Links:

PaddleOCR
https://huggingface.co/PaddlePaddle/PaddleOCR-VL

Usage

Use a pipeline

from transformers import pipeline

pipe = pipeline("image-text-to-text", model="PaddlePaddle/PaddleOCR-VL", dtype="bfloat16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/ocr_demo2.jpg"},
            {"type": "text", "text": "OCR:"},
        ]
    }
]
result = pipe(text=messages)
print(result)

Load model directly

from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("PaddlePaddle/PaddleOCR-VL")
model = AutoModelForImageTextToText.from_pretrained("PaddlePaddle/PaddleOCR-VL", dtype="bfloat16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/ocr_demo2.jpg"},
            {"type": "text", "text": "OCR:"},
        ]
    }
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=100)
result = processor.decode(outputs[0][inputs["input_ids"].shape[-1]:-1])
print(result)

zucchini-nlp

hey @zhang-prog , thanks for the PR! Great model to have in transformers!

The main thing to fix first is the naming, it should clearly include "PaddlePaddleOCR" and follow the usual pattern depending on the modality. The config format also isn’t right; it needs to be fully nested, with text and vision configs inside. Additionally there are no tests or docs, several files are missing. You can run transformers add-new-model-like which would generate a placeholder with the necessary files. I also left some smaller comments here and there. Let me know if you hit any issues

zucchini-nlp · 2025-11-13T10:46:28Z

src/transformers/models/paddleocr_vl/modular_paddleocr_vl.py

+    if height < factor:
+        width = round((width * factor) / height)
+        height = factor
+
+    if width < factor:
+        height = round((height * factor) / width)
+        width = factor
+


same as Qwen but with support for H/W smaller than a factor. I think we made qwen-VL support small images as well, so prob directly importing will give expected result?

Unlike Qwen-VL's smart_resize method, we have differences in factor and max_pixels that need to be preserved. Forcing the use of Qwen-VL's smart_resize method could result in decreased accuracy.

You mean the default values of min-max pixels? We're passing the correct value from within image processor when calling smart_resize, so it shouldn't be a problem, no?

Our method is not entirely identical to Qwen’s, therefore direct reusability is not possible. The differences are as follows:

+ if height < factor: + width = round((width * factor) / height) + height = factor + if width < factor: + height = round((height * factor) / width) + width = factor if max(height, width) / min(height, width) > 200: raise ValueError( f"absolute aspect ratio must be smaller than 200, got {max(height, width) / min(height, width)}" ) h_bar = round(height / factor) * factor w_bar = round(width / factor) * factor if h_bar * w_bar > max_pixels: beta = math.sqrt((height * width) / max_pixels) - h_bar = math.floor(height / beta / factor) * factor - w_bar = math.floor(width / beta / factor) * factor + h_bar = max(factor, math.floor(height / beta / factor) * factor) + w_bar = max(factor, math.floor(width / beta / factor) * factor) elif h_bar * w_bar < min_pixels: beta = math.sqrt(min_pixels / (height * width)) h_bar = math.ceil(height * beta / factor) * factor w_bar = math.ceil(width * beta / factor) * factor return h_bar, w_bar

the diff is shown, we need to use our method :)

src/transformers/models/paddleocr_vl/modular_paddleocr_vl.py

zucchini-nlp · 2025-11-13T10:53:51Z

src/transformers/models/paddleocr_vl/modular_paddleocr_vl.py

+    tokenizer_class = "AutoTokenizer"
+
+    def __init__(self, image_processor=None, tokenizer=None, chat_template=None, **kwargs):
+        self.image_token = "<|IMAGE_PLACEHOLDER|>" if not hasattr(tokenizer, "image_token") else tokenizer.image_token


can you add the token in tokenizer so we can assume t's always available?
https://huggingface.co/docs/transformers/en/main_classes/tokenizer#multimodal-tokenizer

comment not addressed. After adding it in the tokenizer, we can assume it exists and get directly tokenizer.image_token

I tried adding image_token to the special_tokens_map.json, but found it had no effect. What should I do? Do you have an example?

can you try with the above link? It should work unless smth got broken in the latest refactor

Sure. We add image_token to tokenizer_config.json to solve this problem

src/transformers/models/paddleocr_vl/modular_paddleocr_vl.py

zhang-prog · 2025-11-21T05:22:13Z

@zucchini-nlp
We have refactored the code to address the issues you mentioned in your comments.
Please review the code again when you have time.
Thank you for your efforts!!!

zucchini-nlp

@zhang-prog thanks for iterating!

There are a couple major comments which were not addressed and can be a blocker for merging.

The model seems to not support batched inference in current state. We need to enable batching before merging if possible. Should not be hard I think given that the image tower is quite similar to existing models
We also need tests to make sure everything actually works and a documentation page. These files are usually auto-prefilled with empty files when you run transformers add-new-model-like
Let the modular copy automatically when possible. I think there are a few more modules which can be copied from similar models. If you struggle with finding a similar model, you can try out a modular detector

src/transformers/models/paddleocr_vl/__init__.py

zucchini-nlp · 2025-11-24T11:07:33Z

src/transformers/models/paddleocr_vl/modular_paddleocr_vl.py

+    if height < factor:
+        width = round((width * factor) / height)
+        height = factor
+
+    if width < factor:
+        height = round((height * factor) / width)
+        width = factor
+


You mean the default values of min-max pixels? We're passing the correct value from within image processor when calling smart_resize, so it shouldn't be a problem, no?

src/transformers/models/paddleocr_vl/modular_paddleocr_vl.py

zucchini-nlp · 2025-11-24T12:00:19Z

src/transformers/models/paddleocr_vl/modular_paddleocr_vl.py

+            image_embeds = self.get_image_features(pixel_values, image_grid_thw)
+            image_embeds = self.projector(image_embeds, image_grid_thw)
+            image_mask = self.get_placeholder_mask(input_ids, inputs_embeds=inputs_embeds, image_features=image_embeds)
+            inputs_embeds = inputs_embeds.masked_scatter(image_mask, image_embeds)


image_embeds.to(device=inputs_embeds.device, dtype=inputs_embeds.dtype) so we can support multiGPU without problems

src/transformers/models/paddleocr_vl/modular_paddleocr_vl.py

zhang-prog · 2025-11-26T04:06:15Z

@zhang-prog thanks for iterating!

There are a couple major comments which were not addressed and can be a blocker for merging.

The model seems to not support batched inference in current state. We need to enable batching before merging if possible. Should not be hard I think given that the image tower is quite similar to existing models

We also need tests to make sure everything actually works and a documentation page. These files are usually auto-prefilled with empty files when you run transformers add-new-model-like

Let the modular copy automatically when possible. I think there are a few more modules which can be copied from similar models. If you struggle with finding a similar model, you can try out a modular detector

@zucchini-nlp Thank you for your valuable insights! We’ve carefully addressed all comments and responded to your overall recommendations.

We support bs > 1, like this:

from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("PaddlePaddle/PaddleOCR-VL")
model = AutoModelForImageTextToText.from_pretrained("PaddlePaddle/PaddleOCR-VL", dtype="bfloat16")
messages1 = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/ocr_demo.jpg"},
            {"type": "text", "text": "OCR:"},
        ]
    }
]
messages2 = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/ocr_demo2.jpg"},
            {"type": "text", "text": "OCR:"},
        ]
    }
]
batch_messages = [messages1, messages2]
inputs = processor.apply_chat_template(
	batch_messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
    padding=True,
    padding_side='left',
).to(model.device)

generated_ids = model.generate(**inputs, max_new_tokens=1024)
generated_ids_trimmed = [out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)]
result = processor.batch_decode(generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False)
print(result)

We still have some issues to discuss. I replied to your comment and will generate the final version of the document once it’s completed.
We also added the PaddleOCRVisionConfig and PaddleOCRTextConfig into modular.

Thank you for your efforts. ❤️
PTAL.

zhang-prog · 2025-11-26T08:54:04Z

@zucchini-nlp How do I properly add documentation pages and unit tests? I tried to use transformers add-new-model-like, which generates the new modular_xxx.py files, but this process might not be the right approach.

zhang-prog · 2025-12-03T09:19:10Z

@zucchini-nlp PTAL. Thanks❤️

zucchini-nlp · 2025-12-03T11:07:11Z

Sorry, taking a look.Got lost in my notifications

zucchini-nlp

Nice, only a few comments and replied to your questions above

For the docs and the tests, they need to be in source/docs/en/model_doc and in tests folder. You can take a look at the recently merged model for an example https://github.com/huggingface/transformers/pull/41112/files#diff-857421affc3c877bca95377cbb6adb3a8374b149fcbdcc6c759ea0408fa96897

zucchini-nlp · 2025-12-03T14:41:22Z

src/transformers/models/paddleocr_vl/modular_paddleocr_vl.py

+
+
+logger = logging.get_logger(__name__)
+rope_config_validation = partial(_rope_config_validation, ignore_keys={"mrope_section"})


We had quite a few changes with rope recently. Rebasing main will help to get rid of this. Ignore keys now can be passed to super and rope validation is called in base class :)

Nice work!

But right now, we can only pass ignore_keys_at_rope_validation via kwargs and can’t specify it directly:

kwargs["ignore_keys_at_rope_validation"] = {"mrope_section"}

Maybe setting ignore_keys_at_rope_validation as a default parameter would make it more flexible?

you mean as a class attriibute or similar? That could be an option yes, my initial idea was to not multiply the number of cls attributes though ignore_keys are static and can be defined once. We're still adjusting stuff and the current way is an rc0.

For this model, we can follow the api in main branch and I will try to make a better way to pass ignore_keys for subsequent releases

ok. I will remove kwargs["ignore_keys_at_rope_validation"] = {"mrope_section"} firstly ; otherwise, the make style will not pass.

I will adjust this part of the code according to the main branch (your subsequent modifications).

src/transformers/models/paddleocr_vl/modular_paddleocr_vl.py

zucchini-nlp · 2025-12-03T15:55:57Z

src/transformers/models/paddleocr_vl/modular_paddleocr_vl.py

+
+
+class PaddleOCRVLModel(Qwen2VLModel):
+    _checkpoint_conversion_mapping = {"^model": "language_model"}


do we need it or is the model state dict converted to correct format?

@zucchini-nlp
We need to maintain the weight mapping because vllm, sglang, and fastdeploy also need to load the current state dict. If we modify the state dict, we would have to submit PRs to these three repositories to change the weight mapping, which would be quite cumbersome.😢

However, I have noticed that the _checkpoint_conversion_mapping attribute was removed a week ago.

Are there any other methods to achieve this besides passing the key_mapping during initialization? We want to ensure that users are unaware of this process.

The modification for _checkpoint_conversion_mapping is here: #42396

Now, we need to pass in key_mapping to ensure correct inference, which is a bit complicated. Is there a better way to do this?

example:

from transformers import AutoProcessor, AutoModelForImageTextToText + key_mapping = { + "^visual": "model.visual", + "^mlp_AR": "model.projector", + r"^model(?!(\.visual|\.projector))": "model.language_model", + } processor = AutoProcessor.from_pretrained("PaddlePaddle/PaddleOCR-VL") - model = AutoModelForImageTextToText.from_pretrained("PaddlePaddle/PaddleOCR-VL", dtype="bfloat16") + model = AutoModelForImageTextToText.from_pretrained("PaddlePaddle/PaddleOCR-VL", dtype="bfloat16", key_mapping=key_mapping) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/ocr_demo2.jpg"}, {"type": "text", "text": "OCR:"}, ] } ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=100) result = processor.decode(outputs[0][inputs["input_ids"].shape[-1]:-1]) print(result)

I see, no prob with keeping the state dict as is. I was mostly curious if it's a typo or intended :)

I understand:
add paddleocrvl to VLMS in transformers/src/transformers/conversion_mapping.py to use _checkpoint_conversion_mapping

src/transformers/models/paddleocr_vl/modular_paddleocr_vl.py

zhang-prog · 2025-12-04T10:21:37Z

@zucchini-nlp

PTAL.❤️

_checkpoint_conversion_mapping and ignore_keys_at_rope_validation needs to be discussed.

I am working on the docs and tests.

zucchini-nlp · 2025-12-04T11:15:51Z

src/transformers/models/paddleocr_vl/modular_paddleocr_vl.py

-        assert self.temporal_patch_size == 1
-        flatten_patches = patches.reshape(grid_t * grid_h * grid_w, channel, self.patch_size, self.patch_size)
+        if temporal_patch_size != 1:
+            raise ValueError("temporal_patch_size must be 1!")


super nit: I'd frame it as "temporal_patch_size must be 1, but got {temporal_patch_size}!" to give more information

zucchini-nlp · 2025-12-04T11:17:32Z

Great, looking good already. We can keep the conversion mapping as is, no issue for us! There are also a few unresolved comments from the past iterations, if you can take a look

Ping me when the docs/tests are added and the CI shows ✅

zhang-prog · 2025-12-05T14:00:58Z

Don't merge. Working.....

github-actions · 2025-12-08T06:23:49Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto

init

de74e15

zucchini-nlp self-requested a review November 13, 2025 09:07

zucchini-nlp reviewed Nov 13, 2025

View reviewed changes

refactor

0fe7b78

zucchini-nlp reviewed Nov 24, 2025

View reviewed changes

update

70c2338

zucchini-nlp reviewed Dec 3, 2025

View reviewed changes

zhang-prog added 2 commits December 4, 2025 16:16

Merge remote-tracking branch 'origin/main' into feat/paddleocr_vl

d4034ca

update

f1cd9ac

zucchini-nlp reviewed Dec 4, 2025

View reviewed changes

fix unresolved problems

1839fab

fix how position_ids work with flash_attn_2

a7468c5



		logger = logging.get_logger(__name__)
		rope_config_validation = partial(_rope_config_validation, ignore_keys={"mrope_section"})



		class PaddleOCRVLModel(Qwen2VLModel):
		_checkpoint_conversion_mapping = {"^model": "language_model"}

[Model] Add PaddleOCR-VL Model Support #42178

Are you sure you want to change the base?

[Model] Add PaddleOCR-VL Model Support #42178

Uh oh!

Conversation

zhang-prog commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Usage

Use a pipeline

Load model directly

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhang-prog Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhang-prog commented Nov 21, 2025

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zhang-prog commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhang-prog commented Nov 26, 2025

Uh oh!

zhang-prog commented Dec 3, 2025

Uh oh!

zucchini-nlp commented Dec 3, 2025

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhang-prog commented Nov 13, 2025 •

edited

Loading

zhang-prog Dec 5, 2025 •

edited

Loading

zhang-prog commented Nov 26, 2025 •

edited

Loading

zhang-prog Dec 4, 2025 •

edited

Loading

zhang-prog commented Dec 5, 2025 •

edited

Loading