1108d8b34e
Although the vision component of multimodal models typically already call the optimized nn.Attention, it is converted into non-fused operations. That is because the backend-specific fused kernels may have requirements, such as padding, and they is performed by the cache, which vision encoders don't use. This implements a fallback path in the backend, softening the requirements into optimizations. In turn, this allows flash attention to be used for vision encoders, saving a significant amount of VRAM and improving performance. |
||
|---|---|---|
| .. | ||
| ggml | ||
| ggml.go | ||
| ggml_test.go | ||
| quantization.go | ||
| threads.go | ||
| threads_debug.go | ||