소스 검색

ollamarunner: Provide mechanism for backends to report loading progress

This enables the runner to report progress back to the Ollama server,
both for showing status to the user and also to prevent the server
from killing the runner if it thinks things have stalled.

Most of the infrastructure was already there, this extends it to
be available to the backends.
Jesse Gross 1 개월 전
부모
커밋
0ff28758b3
2개의 변경된 파일7개의 추가작업 그리고 0개의 파일을 삭제
  1. 4 0
      ml/backend.go
  2. 3 0
      runner/ollamarunner/runner.go

+ 4 - 0
ml/backend.go

@@ -60,6 +60,10 @@ type CacheConfig struct {
 
 // BackendParams controls how the backend loads and executes models
 type BackendParams struct {
+	// Progress is a callback function that allows reporting percentage completion
+	// of model loading
+	Progress func(float32)
+
 	// NumThreads sets the number of threads to use if running on the CPU
 	NumThreads int
 

+ 3 - 0
runner/ollamarunner/runner.go

@@ -783,6 +783,9 @@ func Execute(args []string) error {
 	}
 
 	params := ml.BackendParams{
+		Progress: func(progress float32) {
+			server.progress = progress
+		},
 		NumThreads:     *threads,
 		NumGPULayers:   *numGPULayers,
 		MainGPU:        *mainGPU,