gen_ai inference spark-tts LLM, result error.

[liqianhao111]

using gen_ai to inference spark-tts LLM(Qwen2 Architecture)，but in OVModelForCausalLM,the result is right,but using openvino, every token always "!", that is 0(after tokenizer)

openvino_genai: version:2025.3.0.0

code:
import time
import os

import torch
prompt = "<|task_tts|><|start_content|>量子力学（quantum mechanics）是描述原子尺度及以下微观世界行为的物理学分支，是现代物理学的两大支柱之一。<|end_content|><|start_global_token|><|bicodec_global_2391|><|bicodec_global_1229|><|bicodec_global_1008|><|bicodec_global_3279|><|bicodec_global_273|><|bicodec_global_1590|><|bicodec_global_1232|><|bicodec_global_2201|><|bicodec_global_1356|><|bicodec_global_2700|><|bicodec_global_972|><|bicodec_global_1061|><|bicodec_global_225|><|bicodec_global_3848|><|bicodec_global_3128|><|bicodec_global_3572|><|bicodec_global_758|><|bicodec_global_4095|><|bicodec_global_2290|><|bicodec_global_3325|><|bicodec_global_3445|><|bicodec_global_2683|><|bicodec_global_972|><|bicodec_global_3911|><|bicodec_global_1265|><|bicodec_global_3342|><|bicodec_global_3305|><|bicodec_global_253|><|bicodec_global_113|><|bicodec_global_3665|><|bicodec_global_507|><|bicodec_global_316|><|end_global_token|>"
from transformers import AutoTokenizer
import openvino_genai

pipe = openvino_genai.LLMPipeline("./spark_tts_ov/LLM", device="CPU")
tokenizer = AutoTokenizer.from_pretrained("./spark_tts_ov/LLM")
t0 = time.perf_counter()
result = pipe.generate("你好",max_new_tokens=10)
print(f"warm up time:{time.perf_counter() - t0:4f}")

for token in pipe.generate(prompt,stream=True,max_new_tokens=500):
print(token)
print(tokenizer.encode(token))

[Wovchena]

LLMPipeline is for text generation. You can try https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/python/speech_generation with this model.

[liqianhao111]

spark-tts structure is a little different，it need a Qwen LLM to inference.This structure may have been fine-tuned. I've been getting the correct results using OVModelForCasualLM, but it's slow and needs to be sped up with ov_genai. ov_genai does turn out to be fast, but it fails to produce the correct results.

def inference(
        self,
        text: str,
        prompt_speech_path: Path = None,
        prompt_text: str = None,
        gender: str = None,
        pitch: str = None,
        speed: str = None,
        temperature: float = 0.8,
        top_k: float = 50,
        top_p: float = 0.95,
) -> np.array:
    """
    Performs inference to generate speech from text, incorporating prompt audio and/or text.

    Args:
        text (str): The text input to be converted to speech.
        prompt_speech_path (Path): Path to the audio file used as a prompt.
        prompt_text (str, optional): Transcript of the prompt audio.
        gender (str): female | male.
        pitch (str): very_low | low | moderate | high | very_high
        speed (str): very_low | low | moderate | high | very_high
        temperature (float, optional): Sampling temperature for controlling randomness. Default is 0.8.
        top_k (float, optional): Top-k sampling parameter. Default is 50.
        top_p (float, optional): Top-p (nucleus) sampling parameter. Default is 0.95.

    Returns:
        torch.Tensor: Generated waveform as a tensor.
    """
    print(f"inference device:{self.device}")
    start_inference_time = time.perf_counter()
    if gender is not None:
        prompt = self.process_prompt_control(gender, pitch, speed, text)

    else:
        prompt, global_token_ids = self.process_prompt(
            text, prompt_speech_path, prompt_text
        )
    
    
    model_inputs = self.tokenizer([prompt], return_tensors="pt")
    print("input_token_length:",model_inputs["input_ids"].shape)
    
    streamer = TextIteratorStreamer(self.tokenizer, skip_prompt=True)
    token_times = []

    def timing_streamer_callback(text):
        now = time.perf_counter()
        token_times.append(now)


    streamer.put = timing_streamer_callback

    t0 = time.perf_counter()

    generated_ids = self.model.generate(**model_inputs,
                                        max_new_tokens=3000,
                                        # do_sample=True,
                                        # top_k=top_k,
                                        # top_p=top_p,
                                        # temperature=temperature,
                                        use_cache=True,
                                        streamer=streamer # 用streamer回调!
                                        )
    
    t1 = time.perf_counter()
    token_durations = [t2 - t1 for t1, t2 in zip(token_times[:-1], token_times[1:])]
    # print(token_durations)
    print("output_token_length",len(token_durations))
    # print("token_times", token_times)
    print(f"first token latency: {token_durations[0]:.3f}s")
    print(f"average next-token latency: {(sum(token_durations) - token_durations[0]) / (len(token_durations) - 1):.3f}s")
    print(f"llm inference time:{time.perf_counter() - t0 :.2f}s")


    generated_ids = [
        output_ids[len(input_ids):]
        for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
    ]
    # print("generated_ids", generated_ids)
    # t1 = time.perf_counter()
    # result = self.pipe.generate(prompt,**generation_config) # the result is incorrect
    # print("genai result:", result)
    # print(f"genai inference time:{time.perf_counter() - t1:.2f}")
    # Decode the generated tokens into text
    predicts = self.tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

    # Extract semantic token IDs from the generated text
    pred_semantic_ids = (
        torch.tensor([int(token) for token in re.findall(r"bicodec_semantic_(\d+)", predicts)])
        .long()
        .unsqueeze(0)
    )

    if gender is not None:
        global_token_ids = (
            torch.tensor([int(token) for token in re.findall(r"bicodec_global_(\d+)", predicts)])
            .long()
            .unsqueeze(0)
            .unsqueeze(0)
        )

    # Convert semantic tokens back to waveform
    wav = self.audio_tokenizer.detokenize(
        global_token_ids,
        pred_semantic_ids,
    )
    print(f"total inference time:{time.perf_counter() - start_inference_time:.2f}s")
    print(f"wav length:{len(wav.squeeze()) / 16000:.2f}s")
    print("*" * 20)
    return wav.squeeze()

[Wovchena]

Is there a way to reproduce the problem give it's a custom model?

[liqianhao111]

May I send you the model files and running scripts via email or other means?

[Wovchena]

Yes. Alternatively you can upload the model files and running scripts to http://huggingface.co/. Would be nice if you reduced the content to the minimum contrary to the code posted in this issue.
You can also try constructing LLMPipeline with SDPA attention implementation. That's how OVModelForCausalLM works while the default for LLMPipeline is to use PA. LLMPipeline(models_path, device, {"ATTENTION_BACKEND": "SDPA"}). It's going to lower performance but it will be helpful to diagnose the problem

[liqianhao111]

I have uploaded the model and execution script to huggingface. The specific URL is:https://huggingface.co/liqianhao/openvino-spark-tts
In addition, setting ATTENTION_BACKEND doesn't seem to change the result.

[Wovchena]

GenAI and transformers tokenizers encode the prompt differently

import openvino_genai
import transformers
prompt = "<|task_tts|><|start_content|>量子力学（quantum mechanics）是描述原子尺度及以下微观世界行为的物理学分支，是现代物理学的两大支柱之一。<|end_content|><|start_global_token|><|bicodec_global_2391|><|bicodec_global_1229|><|bicodec_global_1008|><|bicodec_global_3279|><|bicodec_global_273|><|bicodec_global_1590|><|bicodec_global_1232|><|bicodec_global_2201|><|bicodec_global_1356|><|bicodec_global_2700|><|bicodec_global_972|><|bicodec_global_1061|><|bicodec_global_225|><|bicodec_global_3848|><|bicodec_global_3128|><|bicodec_global_3572|><|bicodec_global_758|><|bicodec_global_4095|><|bicodec_global_2290|><|bicodec_global_3325|><|bicodec_global_3445|><|bicodec_global_2683|><|bicodec_global_972|><|bicodec_global_3911|><|bicodec_global_1265|><|bicodec_global_3342|><|bicodec_global_3305|><|bicodec_global_253|><|bicodec_global_113|><|bicodec_global_3665|><|bicodec_global_507|><|bicodec_global_316|><|end_global_token|>"
model_id = "./spark_tts_ov/LLM"
ref = tokenizer = transformers.AutoTokenizer.from_pretrained(model_id).encode(prompt)
print(len(ref))
pred = openvino_genai.Tokenizer(model_id).encode(prompt).input_ids.data[0]
print(len(pred))

[liqianhao111]

I run it and add print(ref==pred),the result:
[ True True True True True True True True True True True True
True True True True True True True True True True True True
True True True True True True True True True True True True
True True True True True True True True True True True True
True True True True True True True True True True True True
True True True True True]
The encoding results are the same

[Wovchena]

I've rerun it and the problem is still there. Let @apaniukov break the tie

[pavel-esir]

GenAI and transformers tokenizers encode the prompt differently

import openvino_genai
import transformers
prompt = "<|task_tts|><|start_content|>量子力学（quantum mechanics）是描述原子尺度及以下微观世界行为的物理学分支，是现代物理学的两大支柱之一。<|end_content|><|start_global_token|><|bicodec_global_2391|><|bicodec_global_1229|><|bicodec_global_1008|><|bicodec_global_3279|><|bicodec_global_273|><|bicodec_global_1590|><|bicodec_global_1232|><|bicodec_global_2201|><|bicodec_global_1356|><|bicodec_global_2700|><|bicodec_global_972|><|bicodec_global_1061|><|bicodec_global_225|><|bicodec_global_3848|><|bicodec_global_3128|><|bicodec_global_3572|><|bicodec_global_758|><|bicodec_global_4095|><|bicodec_global_2290|><|bicodec_global_3325|><|bicodec_global_3445|><|bicodec_global_2683|><|bicodec_global_972|><|bicodec_global_3911|><|bicodec_global_1265|><|bicodec_global_3342|><|bicodec_global_3305|><|bicodec_global_253|><|bicodec_global_113|><|bicodec_global_3665|><|bicodec_global_507|><|bicodec_global_316|><|end_global_token|>"
model_id = "./spark_tts_ov/LLM"
ref = tokenizer = transformers.AutoTokenizer.from_pretrained(model_id).encode(prompt)
print(len(ref))
pred = openvino_genai.Tokenizer(model_id).encode(prompt).input_ids.data[0]
print(len(pred))

i have checked and on for me genai/tokenizer from master and encoded tokens also differ from HF.