go - io.copyN 不是第一次调用时不能工作

Question

我正在尝试从网站下载图片，步骤如下:

• 使用http.Get获取图片
• 使用os.Create 在当前文件夹中创建一个新文件
• 使用io.copyN将图片复制到文件中

但是奇怪的是如果io.CopyN第一次失败了，以后似乎再也没有成功过

代码片段:

    download_again:
        copy_byte, copy_err := io.CopyN(file, res.Body, res.ContentLength)
        fmt.Fprintf(os.Stderr, "img(%s) size: %d\n", name, res.ContentLength)
        if copy_err == nil && res.ContentLength == copy_byte {
            fmt.Fprintf(os.Stderr, "Success to download img(%s)[%f KB(%d B)]: %s\n", img_url, float64(copy_byte)/1024, copy_byte, name)
        } else {
            if try_i > download_times {
                fmt.Fprintf(os.Stderr, "[fatal] fail to download img(%s) %s\n", img_url, name)
                fout.WriteString(name + "\n")
                return
            }
            fmt.Fprintf(os.Stderr, "error in download img(%s)[%f KB(%d B)]: %s, try %d times\n", img_url, float64(copy_byte)/1024, copy_byte, name, try_i)
            try_i++
            goto download_again
        }

和输出信息:

img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[171.447266 KB(175562 B)]: 11085 , try 1 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085 , try 2 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085, try 3 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085, try 4 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085, try 5 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085, try 6 times

任何帮助将不胜感激:)

Answer 1

您需要重试 http 请求。 (不仅仅是 io.CopyN)

这是一个(大部分未经测试的)可恢复下载的实现:

func downloadFile(dst *os.File, url string, offset int64) (int64, error) {
    req, err := http.NewRequest("GET", url, nil)
    if err != nil {
        return 0, err
    }
    // try to use a range header
    if offset > 0 {
        req.Header.Set("Range", fmt.Sprintf("bytes=%d-", offset))
    }
    res, err := http.DefaultClient.Do(req)
    if err != nil {
        return 0, err
    }
    defer res.Body.Close()

    // some http servers don't support range, so in that case we discard the first
    // offset bytes
    if offset > 0 && res.StatusCode != http.StatusPartialContent {
        _, err = io.CopyN(ioutil.Discard, res.Body, offset)
        if err != nil {
            return offset, err
        }
    }

    n, err := io.CopyN(dst, res.Body, res.ContentLength)
    return offset + n, err
}

您可以向该函数传递一个偏移量，它会尝试从该点获取数据(通过告知 HTTP 服务器，或丢弃字节)。它还会返回您取得的进度，因此您可以将此函数包装在重试循环中:

func example() error {
    f, err := os.Create("/tmp/pkg.png")
    if err != nil {
        return err
    }
    defer f.Close()

    offset := int64(0)
    delay := time.Second
    // try 5 times
    for i := 0; i < 5; i++ {
        offset, err = downloadFile(f, "http://golang.org/doc/gopher/pkg.png", offset)
        if err == nil {
            return nil
        }
        // wait a little while before trying again
        time.Sleep(delay)
        delay *= 2
    }
    return fmt.Errorf("failed to download file after 5 tries")
}

顺便说一下，你真的应该避免使用 goto 循环。