Commit Graph

2 Commits

Author SHA1 Message Date
Jeffrey Morgan 2c371ff357
model: treat 'user defined' tokens as special tokens (#11077) 2025-12-29 06:38:15 -06:00
Michael Yang af9708c72d
model: handle multiple eos tokens (#10577)
* get eos_token_id from generation_config.json

* refactor

* include both ids and strings in trace

* comments

* remove special case for gemma3 special vocab (#10743)
2025-12-29 06:38:01 -06:00