java - 用于文件下载操作的多线程代码，与单线程相比并不快

Question

我有一个任务，需要从网络下载多个实体(> 700)的页面。该代码的设计方式是，特定函数获取一个实体的名称，下载其资源页面，然后对其进行处理以挖掘一些属性并将它们放入全局 HashMap 中。请参阅以下代码:

处理每个实体时使用的全局数据结构:

static HashMap<String, HashMap<String, ArrayList<String> > > Table = new HashMap<String, HashMap<String, ArrayList<String>>>();
static ArrayList<String> allColumns = new ArrayList<String>();

单线程代码:

BufferedReader br = new BufferedReader(new FileReader(filePath_ListOfEntities));
String entityURL;
while ((entityURL = br.readLine()) != null) {
    String entityID = entityURL.replace("http://dbpedia.org/resource/", "");
    try{
        GetRowForEntityURL(entityID); // Downloads page, processes it and updates the global DSs
    }catch(Exception e)
     {
         System.out.println("Ignored: " + entityID + " Error: " + e.getMessage());
     }
 }
 PrintTable(); // prints the global hashmap

通过下载每个实体的资源页面，处理变得非常快，这意味着速率确定操作是下载资源页面。请注意，对实体的操作是独立于其他实体的，但对页面的处理只能在资源页面可用后进行。因此，我尝试为函数 GetRowForEntityURL(entityID) 创建一个单独的线程。以下是多线程代码，与单线程代码相比，它花费了更多时间:

多线程代码:

BufferedReader br = new BufferedReader(new FileReader(filePath_ListOfEntities));
String entityURL;
ArrayList<Thread> threads = new ArrayList<>();
while ((entityURL = br.readLine()) != null) {
    String entityID = entityURL.replace("http://dbpedia.org/resource/", "");

    Thread t = new Thread(new Runnable() {
        public void run()
        {
            try{
                GetRowForEntityURL(entityID); // Downloads page, processes it and updates the global DSs
            }catch(Exception e)
            {
                System.out.println("Ignored: " + entityID + " Error: " + e.getMessage());
            }
        }
    });
    t.run();
    threads.add(t);
}
for(int i = 0; i < threads.size(); i++)
    threads.get(i).join();
System.out.println("***********Threads Joined******************");
PrintTable(); // prints the global hashmap

考虑到每个实体都应该并行处理，因此下载应该并行进行，为什么多线程代码并不更快？这应该比单线程快得多。

编辑:

现在很清楚，即使在使用 T.start() 之后，下载也会在单个连接上进行。我需要改进下载代码以实际利用多线程。这是我的下载代码，其中我尝试在每个调用(以及每个线程)中创建一个新连接，但我想这不起作用。

public static void downloadFile(String entityID) throws IOException {
    String fileURL = "http://dbpedia.org/data/" + entityID + ".rdf";
    String saveDir = inputFolder;
    URL url = new URL(fileURL);
    HttpURLConnection httpConn;// = (HttpURLConnection) url.openConnection();
    int responseCode;// = httpConn.getResponseCode();
    do{
        httpConn = (HttpURLConnection) url.openConnection();
        responseCode = httpConn.getResponseCode();
    }
    while(responseCode != HttpURLConnection.HTTP_OK);

    // always check HTTP response code first
    if (responseCode == HttpURLConnection.HTTP_OK) {
        System.out.println("Downloading for: "+entityID);
        String fileName = "";
        String disposition = httpConn.getHeaderField("Content-Disposition");

        if (disposition != null) {
            // extracts file name from header field
            int index = disposition.indexOf("filename=");
            if (index > 0) {
                fileName = disposition.substring(index + 10,
                        disposition.length() - 1);
            }
        } else {
            // extracts file name from URL
            fileName = fileURL.substring(fileURL.lastIndexOf("/") + 1,
                    fileURL.length());
        }

        // opens input stream from the HTTP connection
        InputStream inputStream = httpConn.getInputStream();
        String saveFilePath = saveDir + File.separator + fileName;

        // opens an output stream to save into file
        //saveFilePath.replace(".rdf", ".txt");
        String downloadAt = inputFolder + entityID + ".txt";
        FileOutputStream outputStream = new FileOutputStream(downloadAt);

        int bytesRead = -1;
        byte[] buffer = new byte[4096];
        while ((bytesRead = inputStream.read(buffer)) != -1) {
            outputStream.write(buffer, 0, bytesRead);
        }

        outputStream.close();
        inputStream.close();

        //System.out.println("File downloaded");
    } else {
        System.out.println("Download Failed. Server replied HTTP code: " + responseCode);
    }
    httpConn.disconnect();
}

Answer 1

使用线程池大小为1的Executor来测试单线程速度。然后增加池大小以查看它如何影响时间。

然后请注意，当您的池大小为 500 时，由于发生所有上下文切换，性能实际上会如何减弱。