 Daniel Hiltgen
|
a1dfab43b9
Ensure the libraries are present
|
vor 1 Jahr |
 Jeffrey Morgan
|
4458efb73a
Load all layers on `arm64` macOS if model is small enough (#2149)
|
vor 1 Jahr |
|
Daniel Hiltgen
|
fedd705aea
Mechanical switch from log to slog
|
vor 1 Jahr |
 Michael Yang
|
eaed6f8c45
add max context length check
|
vor 1 Jahr |
|
Daniel Hiltgen
|
7427fa1387
Fix up the CPU fallback selection
|
vor 1 Jahr |
 Daniel Hiltgen
|
de2fbdec99
Merge pull request #1819 from dhiltgen/multi_variant
|
vor 1 Jahr |
|
Michael Yang
|
f4f939de28
Merge pull request #1552 from jmorganca/mxyng/lint-test
|
vor 1 Jahr |
|
Daniel Hiltgen
|
39928a42e8
Always dynamically load the llm server library
|
vor 1 Jahr |
|
Daniel Hiltgen
|
d88c527be3
Build multiple CPU variants and pick the best
|
vor 1 Jahr |
|
Jeffrey Morgan
|
ab6be852c7
revisit memory allocation to account for full kv cache on main gpu
|
vor 1 Jahr |
|
Daniel Hiltgen
|
8da7bef05f
Support multiple variants for a given llm lib type
|
vor 1 Jahr |
|
Jeffrey Morgan
|
b24e8d17b2
Increase minimum CUDA memory allocation overhead and fix minimum overhead for multi-gpu (#1896)
|
vor 1 Jahr |
|
Michael Yang
|
f921e2696e
typo
|
vor 1 Jahr |
|
Jeffrey Morgan
|
f387e9631b
use runner if cuda alloc won't fit
|
vor 1 Jahr |
|
Jeffrey Morgan
|
cb534e6ac2
use 10% vram overhead for cuda
|
vor 1 Jahr |
|
Jeffrey Morgan
|
58ce2d8273
better estimate scratch buffer size
|
vor 1 Jahr |
|
Jeffrey Morgan
|
08f1e18965
Offload layers to GPU based on new model size estimates (#1850)
|
vor 1 Jahr |
|
Daniel Hiltgen
|
e9ce91e9a6
Load dynamic cpu lib on windows
|
vor 1 Jahr |
|
Jeffrey Morgan
|
c0285158a9
tweak memory requirements error text
|
vor 1 Jahr |
|
Jeffrey Morgan
|
77a66df72c
add macOS memory check for 47B models
|
vor 1 Jahr |
|
Jeffrey Morgan
|
5b4837f881
remove unused filetype check
|
vor 1 Jahr |
|
Daniel Hiltgen
|
7555ea44f8
Revamp the dynamic library shim
|
vor 1 Jahr |
|
Daniel Hiltgen
|
3269535a4c
Refine handling of shim presence
|
vor 1 Jahr |
|
Daniel Hiltgen
|
35934b2e05
Adapted rocm support to cgo based llama.cpp
|
vor 1 Jahr |
|
Daniel Hiltgen
|
d4cd695759
Add cgo implementation for llama.cpp
|
vor 1 Jahr |
 Bruce MacDonald
|
811b1f03c8
deprecate ggml
|
vor 1 Jahr |
|
Michael Yang
|
b9495ea162
load projectors
|
vor 1 Jahr |
|
Bruce MacDonald
|
195e3d9dbd
chat api endpoint (#1392)
|
vor 1 Jahr |
|
Jeffrey Morgan
|
00d06619a1
Revert "chat api (#991)" while context variable is fixed
|
vor 1 Jahr |
|
Bruce MacDonald
|
7a0899d62d
chat api (#991)
|
vor 1 Jahr |
