 Daniel Hiltgen
|
30a7d7096c
Bump VRAM buffer back up
|
vor 11 Monaten |
 Michael Yang
|
4736391bfb
llm: add minimum based on layer size
|
vor 1 Jahr |
 Jeffrey Morgan
|
f0c454ab57
gpu: add 512MiB to darwin minimum, metal doesn't have partial offloading overhead (#4068)
|
vor 1 Jahr |
|
Daniel Hiltgen
|
34b9db5afc
Request and model concurrency
|
vor 1 Jahr |
|
Michael Yang
|
26df674785
scale graph based on gpu count
|
vor 1 Jahr |
|
Michael Yang
|
41a272de9f
darwin: no partial offloading if required memory greater than system
|
vor 1 Jahr |
|
Michael Yang
|
7e33a017c0
partial offloading
|
vor 1 Jahr |
|
Daniel Hiltgen
|
be330174dd
Allow setting max vram for workarounds
|
vor 1 Jahr |
 peanut256
|
a189810df6
Determine max VRAM on macOS using `recommendedMaxWorkingSetSize` (#2354)
|
vor 1 Jahr |
|
Daniel Hiltgen
|
7427fa1387
Fix up the CPU fallback selection
|
vor 1 Jahr |
|
Daniel Hiltgen
|
39928a42e8
Always dynamically load the llm server library
|
vor 1 Jahr |
|
Daniel Hiltgen
|
d88c527be3
Build multiple CPU variants and pick the best
|
vor 1 Jahr |
|
Jeffrey Morgan
|
c336693f07
calculate overhead based number of gpu devices (#1875)
|
vor 1 Jahr |
|
Jeffrey Morgan
|
08f1e18965
Offload layers to GPU based on new model size estimates (#1850)
|
vor 1 Jahr |
|
Jeffrey Morgan
|
c7ea8f237e
set `num_gpu` to 1 only by default on darwin arm64 (#1771)
|
vor 1 Jahr |
|
Daniel Hiltgen
|
a2ad952440
Fix windows system memory lookup
|
vor 1 Jahr |
|
Daniel Hiltgen
|
d966b730ac
Switch windows build to fully dynamic
|
vor 1 Jahr |
|
Daniel Hiltgen
|
7555ea44f8
Revamp the dynamic library shim
|
vor 1 Jahr |
|
Daniel Hiltgen
|
6558f94ed0
Fix darwin intel build
|
vor 1 Jahr |
|
Daniel Hiltgen
|
35934b2e05
Adapted rocm support to cgo based llama.cpp
|
vor 1 Jahr |
