go - 上传大文件并非全部在内存中

Question

我使用此 bash 命令将文件上传到站点:

cmd2=$(curl -F "$FileField=@$FileName" -F "signature=$Sig" -F "ajax=$Ajax"  -F "params=$Params" "$Url")

现在我已经转换为 golang 并且我使用了这个函数:

// Open the file.
file, err := os.Open(k.Path)
if err != nil {
    fmt.Println("Open File", err)
    return
}

// Schedule the file to be closed once
// the function returns.
defer file.Close()

// Create a synchronous in-memory pipe. This pipe will
// allow us to use a Writer as a Reader.
pipeReader, pipeWriter := io.Pipe()

var mpWriter *multipart.Writer

body2 := &bytes.Buffer{}

// Create a goroutine to perform the write since
// the Reader will block until the Writer is closed.
go func() {

    // Create a Reader that writes to the hash what it reads
    // from the file.
    fileReader := io.TeeReader(file, body2)

    // Create the multipart writer to put everything together.
    mpWriter = multipart.NewWriter(pipeWriter)
    fileWriter, err := mpWriter.CreateFormFile(responseObject.FileField, filepath.Base(k.Path))

    // Write the contents of the file to the multipart form.
    _, err = io.Copy(fileWriter, fileReader)
    if err != nil {
        fmt.Println("Write File", err)
        return
    }

    // Add the keys we generated.
    mpWriter.WriteField("ajax", strconv.FormatBool(responseObject.FormData.Ajax))
    mpWriter.WriteField("params", responseObject.FormData.Params)
    mpWriter.WriteField("signature", responseObject.FormData.Signature)

    // Close the Writer which will cause the Reader to unblock.
    defer mpWriter.Close()
    defer pipeWriter.Close()
}()

// Wait until the Writer is closed, then write the
// Pipe to Stdout.
io.Copy(body2, pipeReader)

client := &http.Client{}

// Send post request
req, err := http.NewRequest("POST", responseObject.FormAction, body2)

req.Header.Set("Content-Type", mpWriter.FormDataContentType())

// Submit the request
res, err := client.Do(req)
...

对于小文件我没有问题，但是当我尝试上传大文件时，它会将文件全部加载到内存中并且程序崩溃。

我试图找到解决方案但没有成功，我可以修改什么？

谢谢

Answer 1

您正在创建流的内存镜像，然后将该内存镜像流式传输到服务器。相反，您应该直接使用管道，以便在任何给定时间只有大文件的小块在内存中:

req, err := http.NewRequest("POST", responseObject.FormAction, pipeReader)

不要使用 body2缓冲。