ollama

Commit Graph

Author	SHA1	Message	Date
Jeffrey Morgan	e093db92c4	sample: temporarily use grammars for constrained generation in new engine (#9586 )	2025-03-10 16:17:39 +01:00
Parth Sareen	0682dae027	sample: improve ollama engine sampler performance (#9374 ) This change bring in various interface cleanups along with greatly improving the performance of the sampler. Tested with llama3.2 on local machine. Improves performance from ~ 70 tokens/s -> 135 tokens/s with topK(40) enabled. Without topK performance is ~ 110 tokens/s	2025-03-07 12:37:48 -08:00

Author

SHA1

Message

Date

Jeffrey Morgan

e093db92c4

sample: temporarily use grammars for constrained generation in new engine (#9586 )

2025-03-10 16:17:39 +01:00

Parth Sareen

0682dae027

sample: improve ollama engine sampler performance (#9374 )

This change bring in various interface cleanups along with greatly improving the performance of the sampler.

Tested with llama3.2 on local machine.
Improves performance from ~ 70 tokens/s -> 135 tokens/s with topK(40) enabled.
Without topK performance is ~ 110 tokens/s

2025-03-07 12:37:48 -08:00

2 Commits