Commit Graph

4 Commits

Author SHA1 Message Date
Michael Yang af9708c72d
model: handle multiple eos tokens (#10577)
* get eos_token_id from generation_config.json

* refactor

* include both ids and strings in trace

* comments

* remove special case for gemma3 special vocab (#10743)
2025-12-29 06:38:01 -06:00
Michael Yang 4456012956 fix unmarshaling merges 2024-12-04 09:21:56 -08:00
Michael Yang 60e47573a6 more tokenizer tests 2024-08-27 14:51:10 -07:00
Michael Yang 3eb08377f8 detect chat template from configs that contain lists 2024-08-27 10:49:33 -07:00