java - 页面完全加载后读取页面源代码(执行JavaScript)

Question

我有以下代码等待 JavaScript 渲染完成。

ChromeDriver driver = new ChromeDriver();
driver.get(url);
WebDriverWait wait = new WebDriverWait(driver, 3000);
wait.until(new Predicate<WebDriver>() {
    public boolean apply(WebDriver driver) {
        return ((JavascriptExecutor)driver).executeScript("return document.readyState").equals("complete");
            }
          }
        );

问题是，页面源码完成后如何读取？下面的方法不保证它会根据其文档返回修改页面的源。

driver.getPageSource();

Answer 1

也许，这并不是真正的答案，但它的文本太长，无法发表评论

首先，你想实现什么目标？

据我尝试发现，ChromeDriver的getPageSource()返回“当前”源代码。不是最初从服务器发送的源代码。

JavaScript 加载页面的问题似乎是，浏览器接收页面内容，并在最初解释和渲染它之后，对于浏览器，文档已准备就绪

通过 javascript 的请求不会改变(不能改变？)document.readyState

因此，在页面加载并最初呈现后，其他所有内容都“只是”dom 操作

一些建议只要到达页面末尾和/或在页面上找到某个元素就向下滚动

但是

假设您必须向下滚动 50,000 像素到页面末尾，如果前面的元素被 javascript 删除，您将永远无法立即获得“完整”源代码

在我看来，你必须问自己一个问题:你想要实现什么目标

对我来说，类似下面的代码是有效的:

private void performScrollToEndOfPage(final WebDriver driver) throws Exception {
    final JavascriptExecutor js = (JavascriptExecutor) driver;
    Long prev = null;
    while (true) {
        this.checkItems(driver);
        long val = 2000;
        if (prev != null) {
            val += prev.longValue();
        }
        final Object current = js.executeScript("window.scrollTo(0, " + Long.toString(val) + ");return window.pageYOffset;");
        if (!(current instanceof Long)) {
            break;
        }
        // 1000 milliseconds by try and error, if this value is too low, increase it
        Thread.sleep(1000);
        if (current.equals(prev)) {
            break;
        }
        prev = (Long) current;
    }
}

private void checkItems(final WebDriver driver) {
    final List<WebElement> elements = driver.findElements(By.tagName("a"));
    for (final WebElement anchor : elements) {
        final String href = anchor.getAttribute("href");
        if ((href == null) || href.isBlank() || href.isEmpty()) {
            // TODO: throw Exception or whatever
        }
    }
}

编辑 - 回复您的评论

据我从您的评论中了解到，您的意图似乎是逆向工程。

我认为，创建一个“通用”链接爬虫几乎是不可能的。

因此您可能非常了解页面源代码及其所有依赖项。

您可以在单击页面上的“所有可用元素”时跟踪/捕获网络流量

我可以给你的一个(可能很奇怪的)例子是针对 Instagram 的:

import java.io.IOException;
import java.util.Iterator;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Set;

import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpUriRequest;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.openqa.selenium.By;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;

public class Analyzer {
    private final String baseUrl = "https://www.instagram.com";
    private final WebDriver driver;
    private final Set<String> hrefs = new LinkedHashSet<>();
    private final String profileName;

    private Analyzer(final String profileName) {
        super();
        this.profileName = profileName;
        this.driver = new ChromeDriver();
    }

    public String getBaseUrl() {
        return this.baseUrl;
    }

    public WebDriver getDriver() {
        return this.driver;
    }

    public String getProfileName() {
        return this.profileName;
    }

    public void run() {
        this.getDriver().get(this.getUrl());
        this.performScrollToEndOfPage();
        this.getDriver().quit();
        final Iterator<String> iterator = this.hrefs.iterator();
        while (iterator.hasNext()) {
            final String href = iterator.next();
            final boolean linkBroken = this.isLinkBroken(href);
            // remove unbroken links
            if (!linkBroken) {
                // remove unbroken URLs/hrefs
                iterator.remove();
            }
        }
        // TODO: handle broken links
        System.out.println(this.hrefs);
    }

    private void collectOperation() {
        // collect whatever you want
        // items with src-attribute (for example image-tags),
        // action-attribute (for example form-tags) or
        // href-attribute (for example anchor-tags)
        // and evaluate the attributes value(s)
        final List<WebElement> elements = this.getDriver().findElements(By.tagName("a"));
        for (final WebElement anchor : elements) {
            final String href = anchor.getAttribute("href");
            final List<WebElement> images = anchor.findElements(By.tagName("img"));
            if ((href == null) || href.isBlank() || href.isEmpty() || (images == null) || images.isEmpty()) {
                continue;
            }
            this.hrefs.add(href);
        }
    }

    private String getUrl() {
        return this.getBaseUrl() + "/" + this.getProfileName();
    }

    private boolean isLinkBroken(final String href) {
        try (CloseableHttpClient client = HttpClients.createDefault();) {
            final HttpUriRequest request = new HttpGet(href);
            try (CloseableHttpResponse response = client.execute(request)) {
                if (response.getStatusLine().getStatusCode() == 200) {
                    return false;
                }
            }
        } catch (final IOException e) {
            // TODO: logging or something else
            e.printStackTrace();
        }
        return true;
    }

    private void performScrollToEndOfPage() {
        final JavascriptExecutor js = (JavascriptExecutor) this.driver;
        // This will scroll the web page till end.
        Long prev = null;
        while (true) {
            this.collectOperation();
            // scroll by 2000px
            long val = 2000;
            if (prev != null) {
                val += prev.longValue();
            }
            final Object current = js.executeScript("window.scrollTo(0, " + Long.toString(val) + ");return window.pageYOffset;");
            try {
                // try and error, if this value is too low, increase it
                Thread.sleep(1000);
            } catch (final Exception e) {
                // TODO: logging or something else
                e.printStackTrace();
            }
            if (!(current instanceof Long)) {
                break;
            }
            if (current.equals(prev)) {
                break;
            }
            prev = (Long) current;
        }
    }

    public static void main(final String[] args) {
        // i hope Microsoft will be lenient with me
        final String profileName = "microsoft";
        final Analyzer analyzer = new Analyzer(profileName);
        analyzer.run();
    }
}

instagram 的工作方式与“我们的”页面完全相同。您可以向下滚动到某人“页面”的最后，您只会看到大约 35 个帖子。

无论使用驱动程序 findElements 或 getPageSource-方法并使用 ( 解析 getPageSources return-value例如)Jsoup