java - 使用 BufferedWriter 在 Java 中更快地获取 353 个网页

Question

我查看了其他链接，但没有找到解决方案。我能够在大约 7 分钟内连接到 353 个链接并抓取我需要的数据。我需要将时间缩短到 1 分钟以内。

我在下面包含了我的代码。

URL urlChartLink;
URLConnection urlconn;

try {
    Class.forName(driver).newInstance();
    Connection mysqlconn = DriverManager.getConnection(url + dbName, userName, password);
    Statement st1 = mysqlconn.createStatement();
    Connection mysqlconn2 = DriverManager.getConnection(url + dbName, userName, password);
    ResultSet rs1 = st1.executeQuery(strSQL);

    while (rs1.next()) 
    {
        sElementID = rs1.getString(1);
        sSymbol = rs1.getString(2);
        sChartLink = rs1.getString(3);

        urlChartLink = new URL(sChartLink);
        urlconn = urlChartLink.openConnection();
        urlconn.addRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)");

        sCurrentPrice = "";
        sPriceChange = "";

        try {
            BufferedReader in = new BufferedReader(new InputStreamReader(urlconn.getInputStream(), "UTF-8"));                   
            String currentLine;

            int iLine = 0;

            while ((currentLine = in.readLine()) != null) {
                //Get data from page
                iLine += 1;

            }
            in.close();
        } catch (IOException e) {

    }

    st1.close();

    mysqlconn.close();
    mysqlconn2.close();

}

我在没有 URLConnection 的情况下尝试过，但收到 403 错误。

如果有人能给我一个更好的解决方案，那就太好了!!

埃迪·雷

Answer 1

将 ExecutorService 与线程一起使用。此代码并非 100% 没有语法错误，但它应该为您提供正确的想法。

Class.forName(driver).newInstance();
Connection mysqlconn = DriverManager.getConnection(url + dbName, userName, password);
Statement st1 = mysqlconn.createStatement();
Connection mysqlconn2 = DriverManager.getConnection(url + dbName, userName, password);
final ResultSet rs1 = st1.executeQuery(strSQL);
ExecutorService service = ExecutorService.newFixedThreadPool(30);

while (rs1.next()) 
{
    service.execute(new Runnable() {
        public void run() {
            String sElementID = rs1.getString(1);
            String sSymbol = rs1.getString(2);
            String sChartLink = rs1.getString(3);

            URL urlChartLink = new URL(sChartLink);
            URLConnection urlconn = urlChartLink.openConnection();
            urlconn.addRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)");

            sCurrentPrice = "";
            sPriceChange = "";

            try {
                BufferedReader in = new BufferedReader(new InputStreamReader(urlconn.getInputStream(), "UTF-8"));                   
                String currentLine;


                while ((currentLine = in.readLine()) != null) {
                    //Call syncronized method to perform operations that need to be thread safe
                    addLine()

                }
                in.close();
            } catch (IOException e) {

            }
        }
    });
}

executor.shutdown();
while (!executer.isShutdown() {
    Thread.sleep(100);
}

st1.close();

mysqlconn.close();
mysqlconn2.close();

public void addLine() {
    syncronized (OBJECT_LOCK) {
        iLine++;
    }
}