r - 使用 RSelenium 在 R 中抓取 Javascript

Question

我正在尝试抓取 Washington Post's database on police shootings 。由于它不是 html，我无法使用 rvest，所以我使用 RSelenium和 phantomjs .

library(RSelenium)

checkForServer()
startServer()

eCap <- list(phantomjs.binary.path = "C:/Program Files/Chrome Driver/phantomjs.exe")
remDr <- remoteDriver(browserName = "phantomjs", extraCapabilities = eCap)

remDr$open()

remDr$navigate("http://www.washingtonpost.com/graphics/national/police-shootings/")

在检查源代码后，很明显我感兴趣的项目具有以下 id 和 class

<div id="js-list-690" class="listWrapper cf">

或在 Chrome 中:

我可以访问单个项目的文本:

remDr$findElement("css", "#js-list-691")$getElementText()

返回

[[1]]
[1] "An unidentified person, a 47-year-old Hispanic man, was shocked with a stun gun and shot on July 30, 2015, in Whittier, Calif. Los Angeles County deputies were investigating a domestic disturbance when he threatened the officers and struck one of them with a metal rod.\nMALEDEADLY WEAPONHISPANIC45 TO 54\nCBS Los AngelesWhittier Daily News"}

但是如果我想获取所有这些项目的列表:

remDr$findElements("class name", "listWrapper cf")

导致错误。

我该怎么做

1. 获取共享此 listWrapper cf 类的所有元素的列表？
2. 返回与每个元素关联的文本列表？

Answer 1

直接使用 JSON 数据会更容易(在几乎所有现代浏览器中使用“开发人员工具”来跟踪加载的 URL...这并没有花费很长时间在该列表中找到):

library(jsonlite)

url <- "https://js.washingtonpost.com/graphics/policeshootings/policeshootings.json?d14385542"

shootings <- fromJSON(url)

dplyr::glimpse(shootings)

## Observations: 564
## Variables:
## $ id                 (int) 3, 4, 5, 8, 9, 11, 13, 15, 16, 17, 19, 21, ...
## $ date               (chr) "2015-01-02", "2015-01-02", "2015-01-03", "...
## $ description        (chr) "Elliot, who was on medication for depressi...
## $ blurb              (chr) "a 53-year-old man of Asian heritage armed ...
## $ name               (chr) "Tim Elliot", "Lewis Lee Lembke", "John Pau...
## $ age                (int) 53, 47, 23, 32, 39, 18, 22, 35, 34, 47, 25,...
## $ gender             (chr) "M", "M", "M", "M", "M", "M", "M", "M", "F"...
## $ race               (chr) "A", "W", "H", "W", "H", "W", "H", "W", "W"...
## $ armed              (chr) "gun", "gun", "unarmed", "toy weapon", "nai...
## $ city               (chr) "Shelton", "Aloha", "Wichita", "San Francis...
## $ state              (chr) "WA", "OR", "KS", "CA", "CO", "OK", "AZ", "...
## $ address            (chr) "600 block of E. Island Lake Drive", "4519 ...
## $ lat                (dbl) 47.24683, 45.48620, 37.69477, 37.76291, 40....
## $ lon                (dbl) -123.12159, -122.89128, -97.28055, -122.422...
## $ is_geocoding_exact (lgl) TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, T...
## $ mental             (lgl) TRUE, FALSE, FALSE, TRUE, FALSE, FALSE, FAL...
## $ sources            (list) http://kbkw.com/local-news/329755, http://...
## $ photos             (list) NULL, NULL, 107, , , , //img.washingtonpos...
## $ videos             (list) NULL, NULL, NULL, NULL, NULL, NULL, NULL, ...