Skip to content

UTF8 validate and replace invalid in GGUF output decoding #3062

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
+5 −2
Open

UTF8 validate and replace invalid in GGUF output decoding #3062

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
356d545
utf8 validate and replace invalid in gguf output decoding
mzegla Nov 24, 2025
82994e2
update tokenizers
mzegla Nov 24, 2025
9f5e7f2
Merge branch 'master' into gguf_utf8_validate
mzegla Nov 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion src/cpp/src/gguf_utils/gguf_tokenizer.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -575,7 +575,10 @@ create_tokenizer_from_config(const std::shared_ptr<void>& shared_object_ov_token
ov::OutputVector inputs_for_fused_ragged(detokenizer_outputs.begin(), detokenizer_outputs.end() - 1);
auto outputs_fused_ragged = create_func("FuzeRagged", inputs_for_fused_ragged, {});
outputs_fused_ragged.insert(outputs_fused_ragged.end(), detokenizer_outputs.end() - 1, detokenizer_outputs.end());
auto packed_output = create_func("StringTensorPack", outputs_fused_ragged, {});
ov::OutputVector inputs_for_utf8_validate(outputs_fused_ragged.begin(), outputs_fused_ragged.end());
auto outputs_utf8_validate = create_func("UTF8Validate", inputs_for_utf8_validate, {{"replace_mode", true}});
outputs_utf8_validate.insert(outputs_utf8_validate.end(), outputs_fused_ragged.end() - 1, outputs_fused_ragged.end());
Copy link

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line appears to insert the last element of outputs_fused_ragged into outputs_utf8_validate, but outputs_utf8_validate already contains all elements from outputs_fused_ragged (copied on line 578). This pattern duplicates the last element. If the intent is to append additional outputs (like the original line 577), verify that outputs_fused_ragged contains the correct elements to insert, or if this duplication is intentional for the UTF8Validate operation.

Suggested change
outputs_utf8_validate.insert(outputs_utf8_validate.end(), outputs_fused_ragged.end() - 1, outputs_fused_ragged.end());

Copilot uses AI. Check for mistakes.
Copy link
Contributor

@apaniukov apaniukov Nov 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that the insert is not really necessary, indeed.

Copy link

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line appends the last element of outputs_fused_ragged to outputs_utf8_validate, but outputs_utf8_validate already contains all elements from outputs_fused_ragged (copied in line 578). This results in duplicating the last element. Consider removing this line or clarifying the intended behavior if partial duplication is needed.

Suggested change
outputs_utf8_validate.insert(outputs_utf8_validate.end(), outputs_fused_ragged.end() - 1, outputs_fused_ragged.end());

Copilot uses AI. Check for mistakes.
auto packed_output = create_func("StringTensorPack", outputs_utf8_validate, {});
packed_output[0].get_tensor().add_names({"string_output"});
auto detokenizer = std::make_shared<Model>(packed_output, ParameterVector{detokenizer_input}, "detokenizer");

Expand Down
2 changes: 1 addition & 1 deletion thirdparty/openvino_tokenizers
Open in desktop
Submodule openvino_tokenizers updated 8 files
+2 −1 .gitignore
+9 −69 README.md
+3 −0 benchmark/.gitignore
+1 −0 python/openvino_tokenizers/hf_parser.py
+2 −0 src/tokenizers_factory.cpp
+1 −1 tests/pass_rates.json
+269 −3,907 tests/stats.json
+1 −12 tests/tokenizers_test.py
Loading