瀏覽代碼

Load all layers on `arm64` macOS if model is small enough (#2149)

Jeffrey Morgan 1 年之前
父節點
當前提交
4458efb73a
共有 1 個文件被更改,包括 2 次插入1 次删除
  1. 2 1
      llm/llm.go

+ 2 - 1
llm/llm.go

@@ -70,7 +70,8 @@ func New(workDir, model string, adapters, projectors []string, opts api.Options)
 			break
 			break
 		}
 		}
 
 
-		opts.NumGPU = 1
+		// TODO: implement layer splitting on macOS
+		opts.NumGPU = 999
 	default:
 	default:
 		if info.Library == "cpu" {
 		if info.Library == "cpu" {
 			slog.Info("GPU not available, falling back to CPU")
 			slog.Info("GPU not available, falling back to CPU")