Java 多线程 HTTP

Question

我有一个工作程序，它循环遍历 UID 列表(20,000 多个项目)，构建、连接、序列化和保存找到的项目属性。工作正常。

我想要实现的是加快速度。它必须发出 20,000 多个 HTTP 请求以及之后的所有请求......它并不是特别快。

我尝试阅读多线程和下面关于连接管理器的代码。重新使用 HttpClient 等。但我无法理解或将给定的代码应用于我的情况。

如何创建代码以便同时发出多个 HTTP 请求以加快处理速度？

PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
        CloseableHttpClient httpClient = HttpClients.custom()
                .setConnectionManager(cm)
                .build();

下面是我当前的代码，如何才能使这个过程更快？

JSONObject httpJSONObject;
            for (int i = 0; i < missingUIDList.size(); i++)
                try {
                    HttpGet get = new HttpGet("https://api.guildwars2.com/v2/items/" + missingUIDList.get(i));
                    HttpClient client = HttpClientBuilder.create().build();
                    HttpResponse response = client.execute(get);
                    HttpEntity entity = response.getEntity();
                    String result = EntityUtils.toString(entity);
                    httpJSONObject = new JSONObject(result);

                    itemRoot items = new Gson().fromJson(httpJSONObject.toString(), itemRoot.class);
                    String name = items.getName().replaceAll("'","''");
                    connection = DriverManager.getConnection("jdbc:sqlite:gw2.db");
                    Statement statement = connection.createStatement();
                    statement.setQueryTimeout(30);  // set timeout to 30 sec.
                    String cookie = "INSERT INTO item VALUES('" + name +
                            "','" + items.getDescription() +
                            "','" + items.getType() +
                            "'," + items.getLevel() +
                            ",'" + items.getRarity() +
                            "'," + items.getVendor_value() +
                            "," + items.getDefault_skin() +
                            "," + items.getId() +
                            ",'" + items.getChat_link() +
                            "','" + items.getIcon() +
                            "');";
                    System.out.println(cookie);
                    statement.executeUpdate(cookie);
                } catch (ClientProtocolException e) {
                    e.printStackTrace();
                } catch (IOException e) {
                    e.printStackTrace();
                } catch (JSONException e) {
                    e.printStackTrace();
                } catch (SQLException e) {
                    System.err.println(e.getMessage());
                }
        }

编辑:

根据 Vadim 的提示，这是针对单线程的、希望更多的优化代码。

private void addMissingItems(List<Integer> missingUIDList) {
    HttpClient client = HttpClientBuilder.create().build();
    HttpResponse response;
    HttpEntity entity;
    String result;
    try {
        connection = DriverManager.getConnection("jdbc:sqlite:gw2.db");
        statement = connection.createStatement();
        statement.setQueryTimeout(30);  // set timeout to 30 sec.
    } catch (SQLException e) {
        System.err.println(e.getMessage());
    }

    for (int i = 0; i < missingUIDList.size(); i++)
        try {
            HttpGet get = new HttpGet("https://api.guildwars2.com/v2/items/" + missingUIDList.get(i));
            response = client.execute(get);
            entity = response.getEntity();
            result = EntityUtils.toString(entity);
            JSONObject httpJSONObject = new JSONObject(result);
            itemRoot items = new Gson().fromJson(httpJSONObject.toString(), itemRoot.class);

            System.out.println(httpJSONObject.getInt("id"));
            String cookie = "INSERT INTO item VALUES('" + items.getName().replaceAll("'","''") +
                    "','" + items.getDescription() +
                    "','" + items.getType() +
                    "'," + items.getLevel() +
                    ",'" + items.getRarity() +
                    "'," + items.getVendor_value() +
                    "," + items.getDefault_skin() +
                    "," + items.getId() +
                    ",'" + items.getChat_link() +
                    "','" + items.getIcon() +
                    "');";
            statement.executeUpdate(cookie);

        } catch (ClientProtocolException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        } catch (JSONException e) {
            e.printStackTrace();
        } catch (SQLException e) {
            System.err.println(e.getMessage());
        }
}

Answer 1

使用 Executor services 的解决方案

PoolingHttpClientConnectionManager cm = new   PoolingHttpClientConnectionManager();
CloseableHttpClient httpClient = HttpClients.custom()
            .setConnectionManager(cm)
            .build();

private final ExecutorService pool = Executors.newFixedThreadPool(poolSize);

for (int i = 0; i < missingUIDList.size(); i++) {
    HttpGet get = new HttpGet("https://api.guildwars2.com/v2/items/" + missingUIDList.get(i));
    pool.execute(new Worker(get));
}

class Worker implements Runnable {
    private final HttpGet get;
    private final CloseableHttpClient httpClient;
    Handler(CloseableHttpClient httpClient,HttpGet get) { 
        this.get = get;
        this.httpClient = httpClient;
    }
    public void run() {
        try {
            HttpResponse response = client.execute(get);
            HttpEntity entity = response.getEntity();
            String result = EntityUtils.toString(entity);
            httpJSONObject = new JSONObject(result);
            ....
            //rest of your code
            ....
            statement.executeUpdate(cookie);
        } catch (ClientProtocolException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        } catch (JSONException e) {
            e.printStackTrace();
        } catch (SQLException e) {
            System.err.println(e.getMessage());
        }
    }
}