浏览代码

runner.go: Fix off by one in batch size check

When adding tokens to a batch, the index is zero based but is
checked against being greater than the max batch size. This results
in an out-of-bounds access when the final token is added.
Jesse Gross 8 月之前
父节点
当前提交
5d34320b7c
共有 1 个文件被更改,包括 1 次插入1 次删除
  1. 1 1
      llama/runner/runner.go

+ 1 - 1
llama/runner/runner.go

@@ -178,7 +178,7 @@ func (s *Server) run(ctx context.Context) {
 
 				for j, t := range seq.tokens {
 					// todo: make this n_batch
-					if j > s.batchSize {
+					if j >= s.batchSize {
 						break
 					}
 					batch.Add(t, seq.nPast, []int{i}, !seq.prompt())