Michael Yang f52671ecc6 refactor: bpe and spm tokenizer merges ... - merge candidates and pairs which are essentially the same other than the type for rank/score - use binaryheap in sentencepiece instead of implement custom structure - update merging algorithm so it uses about 15% less allocations		2025-12-17 13:36:32 -08:00
..
imageproc	deepseekocr	2025-11-18 16:11:37 -08:00
input	batch: use tensors for outputs (#12185)	2025-09-15 14:33:06 -07:00
models	re-enable new engine granite embedding	2025-12-17 13:00:51 -08:00
parsers	DeepseekV3 Family Parser (#13484)	2025-12-16 18:56:30 -08:00
renderers	DeepseekV3 Family Parser (#13484)	2025-12-16 18:56:30 -08:00
testdata	gemma2 impl	2025-03-11 14:35:08 -07:00
bytepairencoding.go	refactor: bpe and spm tokenizer merges	2025-12-17 13:36:32 -08:00
bytepairencoding_test.go	refactor: using testing.B.Loop	2025-10-10 13:25:29 -07:00
model.go	fix: leaf alt name (#12390)	2025-09-23 17:50:53 -07:00
model_test.go	fix: leaf alt name (#12390)	2025-09-23 17:50:53 -07:00
sentencepiece.go	refactor: bpe and spm tokenizer merges	2025-12-17 13:36:32 -08:00
sentencepiece_test.go	model: implement bert in ollama engine (#9080)	2025-09-15 15:35:59 -07:00
textprocessor.go	model: handle multiple eos tokens (#10577)	2025-05-16 13:40:23 -07:00
vocabulary.go	fix(tokenizer): add special tokens to empty inputs (#13091)	2025-11-18 11:16:56 -08:00
vocabulary_test.go	fix(tokenizer): add special tokens to empty inputs (#13091)	2025-11-18 11:16:56 -08:00
wordpiece.go	nomic-embed-text model implementation (#13071)	2025-11-18 18:28:10 -08:00
wordpiece_test.go	nomic-embed-text model implementation (#13071)	2025-11-18 18:28:10 -08:00