Skip to content

Releases: NLPOptimize/flash-tokenizer

BPE

25 May 14:27

Choose a tag to compare

BPE
pretrained_bpe

BPE

packages

02 Apr 10:54

Choose a tag to compare

⚠️⚠️⚠️⚠️⚠️

The configuration file is downloaded from here by the from_pretrained method.

Required packages

31 Mar 06:21

Choose a tag to compare

  • VC_redist.x64.exe : Latest Microsoft Visual C++ Redistributable Version.

dataset

10 Mar 08:33

Choose a tag to compare

⚠️⚠️⚠️⚠️⚠️

dataset

  • texts_en.all.parquet
  • texts_ko.all.parquet
  • texts_cn.all.parquet
  • texts_multilingual.all.parquet