Prechádzať zdrojové kódy

runner: Set windows above normal priority (#6905)

When running the subprocess as a background service windows may
throttle, which can lead to thrashing and very poor token rate.
Daniel Hiltgen 7 mesiacov pred
rodič
commit
dbba73469d
1 zmenil súbory, kde vykonal 8 pridanie a 2 odobranie
  1. 8 2
      llm/llm_windows.go

+ 8 - 2
llm/llm_windows.go

@@ -4,7 +4,10 @@ import (
 	"syscall"
 )
 
-const CREATE_DEFAULT_ERROR_MODE = 0x04000000
+const (
+	CREATE_DEFAULT_ERROR_MODE   = 0x04000000
+	ABOVE_NORMAL_PRIORITY_CLASS = 0x00008000
+)
 
 var LlamaServerSysProcAttr = &syscall.SysProcAttr{
 	// Wire up the default error handling logic If for some reason a DLL is
@@ -12,5 +15,8 @@ var LlamaServerSysProcAttr = &syscall.SysProcAttr{
 	// the user can either fix their PATH, or report a bug. Without this
 	// setting, the process exits immediately with a generic exit status but no
 	// way to (easily) figure out what the actual missing DLL was.
-	CreationFlags: CREATE_DEFAULT_ERROR_MODE,
+	//
+	// Setting Above Normal priority class ensures when running as a "background service"
+	// with "programs" given best priority, we aren't starved of cpu cycles
+	CreationFlags: CREATE_DEFAULT_ERROR_MODE | ABOVE_NORMAL_PRIORITY_CLASS,
 }