5c5535c064
Currently Rows is called as the last step in a model computation to get the values for the output tokens. However, if we move it earlier in the process then we can trim out computations that never get used. This is similar to how models are defined in llama.cpp. Changing the model definition in this way improves token generation performance by approximately 8%. |
||
|---|---|---|
| .. | ||
| llama | ||
| mllama | ||
| pixtral | ||
| qwen2vl | ||
| models.go | ||