Michael Yang f52671ecc6 refactor: bpe and spm tokenizer merges ...

- merge candidates and pairs which are essentially the same other than
  the type for rank/score
- use binaryheap in sentencepiece instead of implement custom structure
- update merging algorithm so it uses about 15% less allocations

2025-12-17 13:36:32 -08:00

..

imageproc

deepseekocr

2025-11-18 16:11:37 -08:00

input

batch: use tensors for outputs (#12185)

2025-09-15 14:33:06 -07:00

models

re-enable new engine granite embedding

2025-12-17 13:00:51 -08:00

parsers

DeepseekV3 Family Parser (#13484)

2025-12-16 18:56:30 -08:00

renderers

DeepseekV3 Family Parser (#13484)

2025-12-16 18:56:30 -08:00

testdata

gemma2 impl

2025-03-11 14:35:08 -07:00

bytepairencoding_test.go

refactor: using testing.B.Loop

2025-10-10 13:25:29 -07:00

bytepairencoding.go

refactor: bpe and spm tokenizer merges

2025-12-17 13:36:32 -08:00

model_test.go

fix: leaf alt name (#12390)

2025-09-23 17:50:53 -07:00

model.go

fix: leaf alt name (#12390)

2025-09-23 17:50:53 -07:00

sentencepiece_test.go

model: implement bert in ollama engine (#9080)

2025-09-15 15:35:59 -07:00

sentencepiece.go

refactor: bpe and spm tokenizer merges

2025-12-17 13:36:32 -08:00

textprocessor.go

model: handle multiple eos tokens (#10577)

2025-05-16 13:40:23 -07:00

vocabulary_test.go

fix(tokenizer): add special tokens to empty inputs (#13091)

2025-11-18 11:16:56 -08:00

vocabulary.go

fix(tokenizer): add special tokens to empty inputs (#13091)

2025-11-18 11:16:56 -08:00

wordpiece_test.go

nomic-embed-text model implementation (#13071)

2025-11-18 18:28:10 -08:00

wordpiece.go

nomic-embed-text model implementation (#13071)

2025-11-18 18:28:10 -08:00