c - 下载未使用 Curl C API 恢复

Question

1. 我正在尝试恢复因互联网故障而失败的下载。我用来检查 curl 下载是否成功的函数是:

   curl_multi_info_read
   

   此函数在网络中断时首次调用时返回正确的错误代码 (CURLE_COULDNT_CONNECT)。如果我尝试再次调用它，它会返回 NULL 指针，这意味着没有消息。实际上，我正在使用返回错误代码来检查是否存在互联网连接。这让我很困扰，因为如果没有互联网，它不会在第二次调用时返回任何错误代码。任何人都可以告诉我如何使用这个函数来检查返回代码，因为这个错误代码(CURLE_COULDNT_CONNECT)对我来说非常重要检查互联网状态并相应地从它的地方恢复下载当我恢复连接时停止....

2. 为了恢复下载我正在使用

   curl_easy_setopt (curl, CURLOPT_RESUME_FROM, InternalOffset);
   

   每次我失去互联网连接时，我都会调用此函数来设置选项，以便在互联网连接恢复时可以恢复下载 ...

---

备注Daniel Stenberg :

以下是有关平台和 libcurl 版本的一些详细信息:

• curl 版本 - libcurl 7.21.6
• 平台 - Linux (Ubuntu)

评论:

1. 是的。你的看法是对的。我从堆栈中删除了 easy handle，通过设置新选项 (curl_easy_setopt(curl, CURLOPT_RESUME_FROM, InternalOffset)) 再次添加到 multi handle，最后我做了 multi perform。如果他们没有互联网连接，它会返回正确的错误。我的问题是:每次失去互联网连接时是否需要重复上述步骤才能获得正确的错误？如果我不执行这些步骤，curl_multi_info_read 函数将始终返回 NULL。

2. 我的另一个观察结果是，当互联网连接恢复时，下载开始恢复。它从之前停止的地方开始下载。这让我感到惊讶。当 curl 恢复互联网时，是否在内部负责恢复下载。如果这是对的吗？我真的需要小心恢复下载或让 curl 处理正确吗？

Answer 1

您没有明确说明您使用的是多界面还是简单界面 值得一提的是您正在使用什么平台以及什么libcurl 你正在使用等

以下是针对 libcurl/7.21.6 的最小 curl easy 和multi 测试。

网线拔了，http server 也关了之类的~好像还可以吧。

这些可能对你有帮助:

curl_easy_setopt(curl, CURLOPT_LOW_SPEED_LIMIT, dl_lowspeed_bytes); //bytes/sec
curl_easy_setopt(curl, CURLOPT_LOW_SPEED_TIME, dl_lowspeed_time); //seconds
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);

注意:您必须非常努力地使 curl 在连接中断时倒下。这是设计使然，但让一些人感到意外。

编辑

我怀疑您是否愿意使用 CURLOPT_TIMEOUT。这将使传输超时。如果您的下载量很大，那么它几乎肯定会花费比您准备等待的时间更长的时间来确定您的网络连接是否有问题 - 超时会受到影响。相比之下，CURLOPT_LOW_SPEED_TIME 超时可能永远不会被命中，即使在经过数小时的传输时间之后也是如此。

curltest_easy.c:

/*----------------------------------------------------
curltest_easy.c 
WARNING: for test purposes only ~ 
*/
#include <stdio.h>
#include <unistd.h>
#include <curl/curl.h>
#include <curl/types.h>
#include <curl/easy.h>
#include <sys/stat.h>



static int dl_progress(void *clientp,double dltotal,double dlnow,double ultotal,double ulnow)
{
    if (dlnow && dltotal)
        printf("dl:%3.0f%%\r",100*dlnow/dltotal); //shenzi prog-mon 
    fflush(stdout);    
    return 0;
}

static size_t dl_write(void *buffer, size_t size, size_t nmemb, void *stream)
{    
    return fwrite(buffer, size, nmemb, (FILE*)stream); 
}


int do_dl(void) 
{
    CURL *curl;
    FILE *fp;
    CURLcode curl_retval;
    long http_response;
    double dl_size;
    int retval=0;
    long dl_lowspeed_bytes=1000; //1K
    long dl_lowspeed_time=10; //sec        
    /*put something biG here, preferably on a server that you can switch off at will ;) */
    char url[] = {"http://fc00.deviantart.net/fs26/f/2008/134/1/a/Dragon_VII_by_NegativeFeedback.swf"};
    char filename[]={"blah.dl"};

    struct stat st={0};    
    if (!stat(filename, &st));    
    printf("st.st_size:[%ld]\n", st.st_size);  


    if(!(fp=fopen(filename, "ab"))) /*append binary*/
      return 1; 


    curl_global_init(CURL_GLOBAL_DEFAULT);   
    curl = curl_easy_init();

    if (curl) 
    {   
        //http://linux.die.net/man/3/curl_easy_setopt
        curl_easy_setopt(curl, CURLOPT_URL, url);

        /*callbacks*/
        curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, dl_write);
        curl_easy_setopt(curl, CURLOPT_PROGRESSFUNCTION, dl_progress);
        curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0);

        /*curl will keep running -so you have the freedom to recover 
        from network disconnects etc in your own way without
        distrubing the curl task in hand. ** this is by design :p ** */ 
        //curl_easy_setopt(curl, CURLOPT_TIMEOUT, 60);          
        //curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 30);
        /*set up min download speed threshold & time endured before aborting*/
        curl_easy_setopt(curl, CURLOPT_LOW_SPEED_LIMIT, dl_lowspeed_bytes); //bytes/sec
        curl_easy_setopt(curl, CURLOPT_LOW_SPEED_TIME, dl_lowspeed_time); //seconds while below low spped limit before aborting


        curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
        curl_easy_setopt(curl, CURLOPT_RESUME_FROM,st.st_size);

        /*uncomment this to get curl to tell you what its up to*/
        //curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);


        if(CURLE_OK != (curl_retval=curl_easy_perform(curl)))
        {                      
            printf("curl_retval:[%d]\n", curl_retval);
            switch(curl_retval) 
            {
                //Transferred a partial file
                case CURLE_WRITE_ERROR: //can be due to a dropped connection
                break;

                //all defined in curl/curl.h 

                default: //suggest quitting on unhandled error
                retval=0;
            };    


            curl_easy_getinfo(curl, CURLINFO_CONTENT_LENGTH_DOWNLOAD, &dl_size);
            printf("CURLINFO_CONTENT_LENGTH_DOWNLOAD:%f\n", dl_size);


            curl_retval=curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_response);

            //see: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
            printf("CURLINFO_RESPONSE_CODE:%ld\n", http_response);

            switch(http_response)
            {
            case 0: //eg connection down  from kick-off ~suggest retrying till some max limit
            break;

            case 200: //yay we at least got to our url
            break;

            case 206:
            case 416: //http://www.checkupdown.com/status/E416.html
            printf("ouch! you might want to handle this & others\n"); 

            default: //suggest quitting on an unhandled error
            retval=0;
            };            
        }
        else
        {
            printf("our work here is done ;)\n");
            retval=2;
        }


        if (fp)
            fclose(fp);

        if (curl)
            curl_easy_cleanup(curl);
    }

    printf("retval [%d]\n", retval);
    return retval;
}


int main(void) 
{
    while (!do_dl())
    {
        usleep(5000);
    }

    return 0;
}

/* notes ----

$sudo apt-get install libcurl4-gnutls-dev
$ curl-config --libs
-L/usr/lib/i386-linux-gnu -lcurl -Wl,-Bsymbolic-functions

#oook. lets do it:
$ gcc -o curltest_easy curltest_easy.c -L/usr/lib/i386-linux-gnu -lcurl -Wl,-Bsymbolic-functions
$ ./curltest
*/

curltest_multi.c:

/*----------------------------------------------------
curltest_mult1.c
WARNING: for test purposes only ~
*/
#include <stdio.h>
#include <unistd.h>
#include <curl/curl.h>
#include <curl/types.h>
#include <curl/easy.h>
#include <sys/stat.h>

typedef struct S_dl_byte_data
{
    double new_bytes_received;  //from the latest request
    double existing_filesize;
} dl_byte_data, *pdl_byte_data;

static int dl_progress(pdl_byte_data pdata,double dltotal,double dlnow,double ultotal,double ulnow)
{
    /*dltotal := hacky way of getting the Content-Length ~ less hacky would be to first
    do a HEAD request & then curl_easy_getinfo with CURLINFO_CONTENT_LENGTH_DOWNLOAD*/
    if (dltotal && dlnow)
    {
        pdata->new_bytes_received=dlnow;
        dltotal+=pdata->existing_filesize;
        dlnow+=pdata->existing_filesize;
        printf(" dl:%3.0f%% total:%.0f received:%.0f\r",100*dlnow/dltotal, dltotal, dlnow); //shenzi prog-mon
        fflush(stdout);
    }
    return 0;
}

static size_t dl_write(void *buffer, size_t size, size_t nmemb, void *stream)
{
    return fwrite(buffer, size, nmemb, (FILE*)stream);
}

////////////////////////
int do_dl(void)
{
    CURLM *multi_handle;
    CURL *curl;
    FILE *fp;
    CURLcode curl_retval;
    int retval=0;
    int handle_count=0;
    double dl_bytes_remaining, dl_bytes_received;
    dl_byte_data st_dldata={0};
    char curl_error_buf[CURL_ERROR_SIZE]={"meh"};
    long dl_lowspeed_bytes=1000, dl_lowspeed_time=10; /* 1KBs for 10 secs*/

    /*put something biG here, preferably on a server that you can switch off at will ;) */
    char url[] = {"http://fc00.deviantart.net/fs26/f/2008/134/1/a/Dragon_VII_by_NegativeFeedback.swf"};

    char outfilename[]={"blah.swf"}, filename[]={"blah.dl"};
    struct stat st={0};


    if (!(fp=fopen(filename, "ab")) || -1==fstat(fileno(fp), &st)) //append binary
      return -1;

    if (curl_global_init(CURL_GLOBAL_DEFAULT))
      return -2;

    if (!(multi_handle = curl_multi_init()))
      return -3;

    if (!(curl = curl_easy_init()))
      return -4;


    st_dldata.new_bytes_received=st_dldata.existing_filesize=st.st_size;

    //http://curl.haxx.se/libcurl/c/curl_easy_setopt.html
    curl_easy_setopt(curl, CURLOPT_URL, url);

    /*callbacks*/
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, dl_write);
    curl_easy_setopt(curl, CURLOPT_PROGRESSFUNCTION, dl_progress);
    curl_easy_setopt(curl, CURLOPT_PROGRESSDATA, &st_dldata);
    curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0);

    /*curl will keep running -so you have the freedom to recover from network disconnects etc
    in your own way without distrubing the curl task in hand. ** this is by design :p **
    The follwoing sets up min download speed threshold & time endured before aborting*/
    curl_easy_setopt(curl, CURLOPT_LOW_SPEED_LIMIT, dl_lowspeed_bytes); //bytes/sec
    curl_easy_setopt(curl, CURLOPT_LOW_SPEED_TIME, dl_lowspeed_time); //seconds while below low spped limit before aborting
    //alternatively these are available in libcurl 7.25
    //curl_easy_setopt(curl, CURLOPT_TCP_KEEPALIVE,1L);
    //curl_easy_setopt(curl, CURLOPT_TCP_KEEPIDLE,10);
    //curl_easy_setopt(curl, CURLOPT_TCP_KEEPINTVL,10);

    curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);

    /*uncomment this to get curl to tell you what its up to*/
    //curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);

    curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, curl_error_buf);


    do
    {
        if (st_dldata.new_bytes_received) //set the new range for the partial transfer if we have previously received some bytes 
        {
            printf("resuming d/l..\n");
            fflush(fp);
            //get the new filesize & sanity check for file; on error quit outer do-loop & return to main
            if (-1==(retval=fstat(fileno(fp), &st)) || !(st_dldata.existing_filesize=st.st_size)) break; 
            //see also: CURLOPT_RANGE for passing a string with our own X-Y range
            curl_easy_setopt(curl, CURLOPT_RESUME_FROM, st.st_size);
            st_dldata.new_bytes_received=0;
        }
        printf("\n\nbytes already received:[%.0f]\n", st_dldata.existing_filesize);

        //re-use the curl handle again & again & again & again... lol
        curl_multi_add_handle(multi_handle, curl);

        do //curl_multi_perform event-loop
        {
            CURLMsg *pMsg;
            int msgs_in_queue;

            while (CURLM_CALL_MULTI_PERFORM == curl_multi_perform(multi_handle, &handle_count));

            //check for any mesages regardless of handle count
            while(pMsg=curl_multi_info_read(multi_handle, &msgs_in_queue))
            {
                long http_response;

                printf("\nmsgs_in_queue:[%d]\n",msgs_in_queue);
                if (CURLMSG_DONE != pMsg->msg)
                {
                    fprintf(stderr,"CURLMSG_DONE != pMsg->msg:[%d]\n", pMsg->msg);
                }
                else
                {
                    printf("pMsg->data.result:[%d] meaning:[%s]\n",pMsg->data.result,curl_easy_strerror(pMsg->data.result));
                    if (CURLE_OK != pMsg->data.result) printf("curl_error_buf:[%s]\n", curl_error_buf);
                    switch(pMsg->data.result)
                    {
                    case CURLE_OK: ///////////////////////////////////////////////////////////////////////////////////////
                    printf("CURLE_OK: ");
                    curl_easy_getinfo(pMsg->easy_handle, CURLINFO_CONTENT_LENGTH_DOWNLOAD, &dl_bytes_remaining);
                    curl_easy_getinfo(pMsg->easy_handle, CURLINFO_SIZE_DOWNLOAD, &dl_bytes_received);
                    if (dl_bytes_remaining == dl_bytes_received)
                    {
                        printf("our work here is done ;)\n");
                        rename(filename, outfilename);
                        retval=1;
                    }
                    else
                    {
                        printf("ouch! st_dldata.new_bytes_received[%f]\n",st_dldata.new_bytes_received);
                        printf("ouch! dl_bytes_received[%f] dl_bytes_remaining[%f]\n",dl_bytes_received,dl_bytes_remaining);
                        retval=dl_bytes_received < dl_bytes_remaining ? 0 : -5;
                    }
                    break; /////////////////////////////////////////////////////////////////////////////////////////////////

                    case CURLE_COULDNT_CONNECT:      //no network connectivity ?
                    case CURLE_OPERATION_TIMEDOUT:   //cos of CURLOPT_LOW_SPEED_TIME
                    case CURLE_COULDNT_RESOLVE_HOST: //host/DNS down ?
                    printf("CURMESSAGE switch handle_count:[%d]\n",handle_count);
                    break; //we'll keep trying

                    default://see: http://curl.haxx.se/libcurl/c/libcurl-errors.html
                    handle_count=0;
                    retval=-5;
                    };


                    //see: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
                    curl_retval=curl_easy_getinfo(pMsg->easy_handle, CURLINFO_RESPONSE_CODE, &http_response);
                    printf("CURLINFO_RESPONSE_CODE HTTP:[%ld]\n", http_response);
                    switch(http_response)
                    {
                    case 0:   //eg connection down  from kick-off ~suggest retrying till some max limit
                    case 200: //yay we at least got to our url
                    case 206: //Partial Content
                    break;

                    case 416:
                    //cannot d/l range ~ either cos no server support
                    //or cos we're asking for an invalid range ~ie: we already d/ld the file
                    printf("HTTP416: either the d/l is already complete or the http server cannot d/l a range\n");
                    retval=2;

                    default: //suggest quitting on an unhandled error
                    handle_count=0;
                    retval=-6;
                    };
                }
            }

            if (handle_count) //select on any active handles
            {
                fd_set fd_read={0}, fd_write={0}, fd_excep={0};
                struct timeval timeout={5,0};
                int select_retval;
                int fd_max;

                curl_multi_fdset(multi_handle, &fd_read, &fd_write, &fd_excep, &fd_max);
                if (-1 == (select_retval=select(fd_max+1, &fd_read, &fd_write, &fd_excep, &timeout)))
                {
                    //errno shall be set to indicate the error
                    fprintf(stderr, "yikes! select error :(\n");
                    handle_count=0;
                    retval=-7;
                    break;
                }
                else{/*check whatever*/}
            }

        } while (handle_count);

        curl_multi_remove_handle(multi_handle,curl);
        printf("continue from here?");
        getchar();        
    }
    while(retval==0);

    curl_multi_cleanup(multi_handle);
    curl_easy_cleanup(curl);
    curl_global_cleanup();
    if (fp) fclose(fp);

    return retval;
}

////////////////////////
int main(void)
{
    int retval;
    printf("\n\ncurl_multi d/l test ~curl version:[%s]\n", curl_version());
    while (1!=(retval=do_dl()))
    {
        printf("retval [%d] continue?\n\n", retval);
        printf("continue?");
        getchar();
    }
    printf("\nend of test!\n\n", retval);
    return retval;
}

/* notes ----

$sudo apt-get install libcurl4-gnutls-dev
$curl-config --libs
-L/usr/lib/i386-linux-gnu -lcurl -Wl,-Bsymbolic-functions

#oook. lets do it:
$gcc -o curltest_multi curltest_multi.c -L/usr/lib/i386-linux-gnu -lcurl -Wl,-Bsymbolic-functions
$./curltest_multi

*/

在开始全新的测试之前，您可能需要记住删除 blah.dl 文件。该程序故意不这样做，因此您可以预先截断现有文件以进行测试；)

NB: 对于这样的事情，你也许应该*不只是依赖于 CURLE_COULDNT_CONNECT ~你的代码应该主要是错误处理(如果你的程序是严格用于个人的，可能更少使用)

编辑

我更新了 curtest_multi.c 以演示 easy_handle 重用。

并且请注意以下来自 the documentation 的引述:

When a single transfer is completed, the easy handle is still left added to the multi stack. You need to first remove the easy handle with curl_multi_remove_handle(3) and then close it with curl_easy_cleanup(3), or possibly set new options to it and add it again with curl_multi_add_handle(3) to start another transfer.