Prechádzať zdrojové kódy

ollamarunner: Provide mechanism for backends to report loading progress

This enables the runner to report progress back to the Ollama server,
both for showing status to the user and also to prevent the server
from killing the runner if it thinks things have stalled.

Most of the infrastructure was already there, this extends it to
be available to the backends.
Jesse Gross 1 mesiac pred
rodič
commit
0ff28758b3
2 zmenil súbory, kde vykonal 7 pridanie a 0 odobranie
  1. 4 0
      ml/backend.go
  2. 3 0
      runner/ollamarunner/runner.go

+ 4 - 0
ml/backend.go

@@ -60,6 +60,10 @@ type CacheConfig struct {
 
 // BackendParams controls how the backend loads and executes models
 type BackendParams struct {
+	// Progress is a callback function that allows reporting percentage completion
+	// of model loading
+	Progress func(float32)
+
 	// NumThreads sets the number of threads to use if running on the CPU
 	NumThreads int
 

+ 3 - 0
runner/ollamarunner/runner.go

@@ -783,6 +783,9 @@ func Execute(args []string) error {
 	}
 
 	params := ml.BackendParams{
+		Progress: func(progress float32) {
+			server.progress = progress
+		},
 		NumThreads:     *threads,
 		NumGPULayers:   *numGPULayers,
 		MainGPU:        *mainGPU,