Commit Graph

16 Commits

Author SHA1 Message Date
Bruce MacDonald eed0ac2948 clean up vision model forward pass 2025-05-12 13:49:42 -07:00
Bruce MacDonald fcfad744ff fix patch merger 2025-05-12 13:49:42 -07:00
Michael Yang fb3c16f2a2 window index 2025-05-12 13:49:42 -07:00
Michael Yang 88b231f903 use maxgridsize 2025-05-12 13:49:42 -07:00
Michael Yang 7e920c8d75 fix: patch merger and convert
convert:
- split patch embedding
- split qkv

remove duplicate PatchMerger
2025-05-12 13:49:42 -07:00
Bruce MacDonald 2af76d0e7a default to 32 for vision block count 2025-05-12 13:49:42 -07:00
Bruce MacDonald 8d901825f0 reshape cos and sin 2025-05-12 13:49:41 -07:00
Bruce MacDonald 04936b719f Update model_vision.go 2025-05-12 13:49:41 -07:00
Bruce MacDonald 0f0136d419 simplify by doing operations in Go rather than with tensors
Co-Authored-By: Michael Yang <2372640+mxyng@users.noreply.github.com>
2025-05-12 13:49:41 -07:00
Bruce MacDonald 80498f76de fix build 2025-05-12 13:49:41 -07:00
Bruce MacDonald 5ff0d538b0 wip: implementing rope 2025-05-12 13:49:41 -07:00
Bruce MacDonald eedc969c35 grid refactor 2025-05-12 13:49:41 -07:00
Bruce MacDonald 963531215e update convert 2025-05-12 13:49:41 -07:00
Bruce MacDonald 3fe090f447 get patch embedding vals from config 2025-05-12 13:49:41 -07:00
Bruce MacDonald 1704072746 patch embeddings 2025-05-12 13:49:41 -07:00
Bruce MacDonald c1f9bcb4dd restructure
image processing

Update model.go

Update model.go

Update model.go

no projector

no projector

vision model scaffold

...

...

wip

...

rebase

fix patch merger

tidy

...

Update model_vision.go

server: do not attempt to parse offset file as gguf

This logic was causing issues for me when importing a gguf that had some padding at the end of the file. The valid gguf would be read, but then it would try to read the offset as a different gguf file. This does not seem right.

Update process_image_test.go

apply norm

prompt processing

prompt processing

fix post tokenize

fix gguf padding + populate the split patch embeddings

...

...

another shot at patch embeddings

...

patch embedding

Update model_vision.go

split pixels
2025-05-12 13:49:41 -07:00