 Jeffrey Morgan
|
0ee87615c7
sched: don't error if paging to disk on Windows and macOS (#5523)
|
vor 10 Monaten |
 Daniel Hiltgen
|
af28b94533
Merge pull request #5469 from dhiltgen/prevent_system_oom
|
vor 10 Monaten |
 Daniel Hiltgen
|
955f2a4e03
Only set default keep_alive on initial model load
|
vor 10 Monaten |
|
Daniel Hiltgen
|
3c75113e37
Prevent loading models larger than total memory
|
vor 10 Monaten |
|
Daniel Hiltgen
|
cff3f44f4a
Fix case for NumCtx
|
vor 10 Monaten |
|
Daniel Hiltgen
|
3518aaef33
Merge pull request #4218 from dhiltgen/auto_parallel
|
vor 10 Monaten |
 Blake Mizerany
|
cb42e607c5
llm: speed up gguf decoding by a lot (#5246)
|
vor 10 Monaten |
|
Daniel Hiltgen
|
9929751cc8
Disable concurrency for AMD + Windows
|
vor 10 Monaten |
|
Daniel Hiltgen
|
17b7186cd7
Enable concurrency by default
|
vor 1 Jahr |
|
Daniel Hiltgen
|
6f351bf586
review comments and coverage
|
vor 11 Monaten |
|
Daniel Hiltgen
|
ff4f0cbd1d
Prevent multiple concurrent loads on the same gpus
|
vor 11 Monaten |
|
Daniel Hiltgen
|
fc37c192ae
Refine CPU load behavior with system memory visibility
|
vor 11 Monaten |
|
Daniel Hiltgen
|
434dfe30c5
Reintroduce nvidia nvml library for windows
|
vor 11 Monaten |
|
Daniel Hiltgen
|
48702dd149
Harden unload for empty runners
|
vor 11 Monaten |
|
Daniel Hiltgen
|
5e8ff556cb
Support forced spreading for multi GPU
|
vor 1 Jahr |
 Michael Yang
|
e40145a39d
lint
|
vor 11 Monaten |
|
Michael Yang
|
c895a7d13f
some gocritic
|
vor 11 Monaten |
|
Michael Yang
|
04f3c12bb7
replace x/exp/slices with slices
|
vor 11 Monaten |
 Patrick Devine
|
4cc3be3035
Move envconfig and consolidate env vars (#4608)
|
vor 11 Monaten |
 Sang Park
|
4434d7f447
Correct typo in error message (#4535)
|
vor 11 Monaten |
|
Daniel Hiltgen
|
ec231a7923
Remove VRAM convergence check for windows
|
vor 11 Monaten |
|
Patrick Devine
|
6845988807
Ollama `ps` command for showing currently loaded models (#4327)
|
vor 11 Monaten |
|
Daniel Hiltgen
|
4142c3ef7c
Always use the sorted list of GPUs
|
vor 11 Monaten |
|
Jeffrey Morgan
|
bb6fd02298
Don't clamp ctx size in `PredictServerFit` (#4317)
|
vor 11 Monaten |
|
Daniel Hiltgen
|
354ad9254e
Wait for GPU free memory reporting to converge
|
vor 1 Jahr |
|
Jeffrey Morgan
|
c9f98622b1
Skip scheduling cancelled requests, always reload unloaded runners (#4189)
|
vor 1 Jahr |
|
Jeffrey Morgan
|
dfa2f32ca0
unload in critical section (#4187)
|
vor 1 Jahr |
|
Daniel Hiltgen
|
f56aa20014
Centralize server config handling
|
vor 1 Jahr |
|
Jeffrey Morgan
|
942c979232
allocate a large enough kv cache for all parallel requests (#4162)
|
vor 1 Jahr |
|
Daniel Hiltgen
|
20f6c06569
Make maximum pending request configurable
|
vor 1 Jahr |
