[JS] Update tokenizer methods #3012

Retribution98 · 2025-11-13T20:47:20Z

Description

Using Tensor from openvino-node in Tokenizer encode and decode
Extended the Tensor API
Added constructors for Tensor
Moved Tokenizer to the separate TS file
Use BigInt for tokenId
Store openvino-node addon in the AddondData to manipulate with entity from the core part
Aligned tests with the new API and verify tokenizer and binding behavior.
Updated benchmark sample with using Tensor encode
Update documentation - https://retribution98.github.io/openvino.genai/

Ticket

CVS-174909

Checklist:

Tests have been updated or added to cover the new code.
This patch fully addresses the ticket.
I have made corresponding changes to the documentation.

Signed-off-by: Kirill Suvorov <[email protected]>

Retribution98 · 2025-11-13T20:48:14Z

This PR is blocked by both PR#32802 and PR#32850 in openvino repository

Copilot

Pull Request Overview

This PR updates the JavaScript tokenizer implementation to use Tensor objects from openvino-node and extends the tokenizer API with encode/decode methods. The changes enable direct manipulation of tokenized inputs using Tensor objects rather than raw arrays, improving type safety and integration with the OpenVINO ecosystem.

Key changes:

Added encode and decode methods to the Tokenizer that work with Tensor objects
Created a separate TypeScript interface file (tokenizer.ts) for better type definitions
Integrated openvino-node addon storage in AddonData for cross-module entity access
Updated tests to verify new tokenizer behavior and aligned imports across test files

Reviewed Changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/js/tests/tokenizer.test.js	Added comprehensive tests for new encode/decode methods and removed async markers from synchronous tests
src/js/tests/bindings.test.js	Updated import to use named export for LLMPipeline
src/js/src/tokenizer.cpp	Implemented encode/decode methods with support for multiple input types and chat template getters/setters
src/js/src/helper.cpp	Added conversion helpers for int64_t, vectors of int64_t, Tensor, and TokenizedInputs
src/js/src/addon.cpp	Added setOpenvinoAddon function to store openvino-node addon reference
src/js/lib/tokenizer.ts	Created TypeScript interface definitions for Tokenizer and related types
src/js/lib/pipelines/textEmbeddingPipeline.ts	Updated to use named import for TextEmbeddingPipeline
src/js/lib/pipelines/llmPipeline.ts	Moved Tokenizer interface to separate file and updated imports
src/js/lib/index.ts	Exported tokenizer types
src/js/lib/addon.ts	Integrated openvino-node addon and updated exports
src/js/include/tokenizer.hpp	Added method declarations for new tokenizer functionality
src/js/include/helper.hpp	Added template specializations for new conversion types
src/js/include/addon.hpp	Added openvino_addon reference to AddonData
samples/js/text_generation/benchmark_genai.js	Added example usage of encode method to get prompt token size

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/js/tests/tokenizer.test.js

samples/js/text_generation/benchmark_genai.js

src/js/lib/tokenizer.ts

almilosz · 2025-11-14T10:05:01Z

src/js/src/addon.cpp

    exports.Set(class_name, prototype);
 }

+Napi::Value init_ov_addon(const Napi::CallbackInfo& info) {


Suggested change

Napi::Value init_ov_addon(const Napi::CallbackInfo& info) {

void init_ov_addon(const Napi::CallbackInfo& info) {

This function is used as a JavaScript function, so we usually return a Napi::Value even if nothing is returned. I understand your confusion because it is called like init_class, but they have different usages. To be clearer, I will rename this function to set_ov_node.

I believe there are functions that return Napi::Value even if they are void, but I do not think we should repeat that pattern. Here it pop_back is used as js function and it's void

openvino.genai/src/js/src/chat_history.cpp

Line 55 in cf4bc0b

void ChatHistoryWrap::pop_back(const Napi::CallbackInfo& info) {

src/js/src/helper.cpp

src/js/src/tokenizer.cpp

almilosz · 2025-11-14T10:43:08Z

src/js/tests/tokenizer.test.js

+  it("getChatTemplate return string", () => {
+    const template = tokenizer.getChatTemplate();
+    assert.strictEqual(typeof template, "string");
+  });
+
+  it("setChatTemplate updates template", () => {
+    const originalTemplate = tokenizer.getChatTemplate();


Suggested change

it("getChatTemplate return string", () => {

const template = tokenizer.getChatTemplate();

assert.strictEqual(typeof template, "string");

});

it("setChatTemplate updates template", () => {

const originalTemplate = tokenizer.getChatTemplate();

it("setChatTemplate updates template", () => {

const originalTemplate = tokenizer.getChatTemplate();

assert.strictEqual(typeof originalTemplate, "string");

I think we don't need to have a separate test to check return type of getChatTemplate(). It'll be enough to get this error from e.g. setChatTemplate updates template.

yatarkan · 2025-11-14T10:58:31Z

Docs also can be updated with new JS tokenizer functionality - in Node.js Bindings page and in Tokenization Guide (add js code example)

src/js/lib/tokenizer.ts

Copilot

Pull Request Overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 11 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/js/tests/tokenizer.test.js

src/js/lib/tokenizer.ts

site/docs/guides/tokenization.mdx

Co-authored-by: Copilot <[email protected]>

Copilot

Pull Request Overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/js/lib/addon.ts

src/js/tests/tokenizer.test.js

src/js/src/helper.cpp

src/js/src/tokenizer.cpp

Copilot

Pull Request Overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-20T12:09:21Z

src/js/src/helper.cpp


+template <>
+int64_t js_to_cpp<int64_t>(const Napi::Env& env, const Napi::Value& value) {
+    OPENVINO_ASSERT(value.IsNumber() || value.IsBigInt(), "Passed argument must be of type Number.");


Error message states 'must be of type Number' but the function accepts both Number and BigInt. Update to 'Passed argument must be of type Number or BigInt.'

Suggested change

OPENVINO_ASSERT(value.IsNumber() || value.IsBigInt(), "Passed argument must be of type Number.");

OPENVINO_ASSERT(value.IsNumber() || value.IsBigInt(), "Passed argument must be of type Number or BigInt.");

Copilot · 2025-11-20T12:09:21Z

src/js/lib/tokenizer.ts

+  applyChatTemplate(
+    chatHistory: Record<string, any>[] | ChatHistory,
+    addGenerationPrompt: boolean,
+    chatTemplate?: string,
+    tools?: Record<string, any>[],
+    extraContext?: Record<string, any>,
+  ): string;


Replace generic 'any' types with more specific types. Consider using 'ChatMessage[]' for chatHistory parameter, 'ToolDefinition[]' for tools, and 'ExtraContext' for extraContext, which are exported from './chatHistory.js'.

[JS] Update tokenizer methods

884f0ec

Signed-off-by: Kirill Suvorov <[email protected]>

Copilot AI review requested due to automatic review settings November 13, 2025 20:47

github-actions bot added category: LLM samples GenAI LLM samples category: JS API GenAI JS API labels Nov 13, 2025

Retribution98 requested review from almilosz and removed request for Copilot November 13, 2025 20:48

Copilot AI reviewed Nov 13, 2025

View reviewed changes

src/js/tests/tokenizer.test.js Outdated Show resolved Hide resolved

src/js/tests/tokenizer.test.js Outdated Show resolved Hide resolved

src/js/tests/tokenizer.test.js Outdated Show resolved Hide resolved

samples/js/text_generation/benchmark_genai.js Outdated Show resolved Hide resolved

Retribution98 requested a review from apaniukov November 13, 2025 20:48

almilosz reviewed Nov 14, 2025

View reviewed changes

apaniukov reviewed Nov 14, 2025

View reviewed changes

src/js/lib/tokenizer.ts Show resolved Hide resolved

apaniukov reviewed Nov 14, 2025

View reviewed changes

src/js/lib/tokenizer.ts Outdated Show resolved Hide resolved

Retribution98 added 4 commits November 14, 2025 14:57

Fix after code review

fd3bbd7

Add constructor, use bigint for tokenID and fix tests

d886ad1

Merge branch 'master' into update_js_tokenizer

78f4171

Update docs

f3e48bf

github-actions bot added the category: GH Pages Docs Github Pages documentation label Nov 14, 2025

Retribution98 added 2 commits November 14, 2025 21:23

Add token_type_ids

9b6daa3

Create issues for found problems

03e6a6e

Copilot AI review requested due to automatic review settings November 14, 2025 21:00

Copilot AI reviewed Nov 14, 2025

View reviewed changes

Apply suggestions from code review

d85604b

Co-authored-by: Copilot <[email protected]>

Copilot AI review requested due to automatic review settings November 14, 2025 21:07

Copilot AI reviewed Nov 14, 2025

View reviewed changes

src/js/lib/addon.ts Show resolved Hide resolved

src/js/tests/tokenizer.test.js Show resolved Hide resolved

src/js/src/helper.cpp Show resolved Hide resolved

src/js/src/tokenizer.cpp Show resolved Hide resolved

Small fixes

3e22f65

Retribution98 requested review from almilosz, apaniukov and yatarkan November 14, 2025 21:27

Merge branch 'master' into update_js_tokenizer

3b7b57d

Copilot AI review requested due to automatic review settings November 20, 2025 12:08

Copilot AI reviewed Nov 20, 2025

View reviewed changes

	Napi::Value init_ov_addon(const Napi::CallbackInfo& info) {
	void init_ov_addon(const Napi::CallbackInfo& info) {

	OPENVINO_ASSERT(value.IsNumber() \|\| value.IsBigInt(), "Passed argument must be of type Number.");
	OPENVINO_ASSERT(value.IsNumber() \|\| value.IsBigInt(), "Passed argument must be of type Number or BigInt.");

[JS] Update tokenizer methods #3012

Are you sure you want to change the base?

[JS] Update tokenizer methods #3012

Conversation

Retribution98 commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Ticket

Checklist:

Uh oh!

Retribution98 commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

almilosz Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

Retribution98 Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

almilosz Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

almilosz Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

almilosz Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yatarkan commented Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Retribution98 commented Nov 13, 2025 •

edited

Loading

Retribution98 commented Nov 13, 2025 •

edited

Loading

almilosz Nov 20, 2025 •

edited

Loading

yatarkan commented Nov 14, 2025 •

edited

Loading