浏览代码

Load all layers on `arm64` macOS if model is small enough (#2149)

Jeffrey Morgan 1 年之前
父节点
当前提交
4458efb73a
共有 1 个文件被更改,包括 2 次插入1 次删除
  1. 2 1
      llm/llm.go

+ 2 - 1
llm/llm.go

@@ -70,7 +70,8 @@ func New(workDir, model string, adapters, projectors []string, opts api.Options)
 			break
 		}
 
-		opts.NumGPU = 1
+		// TODO: implement layer splitting on macOS
+		opts.NumGPU = 999
 	default:
 		if info.Library == "cpu" {
 			slog.Info("GPU not available, falling back to CPU")