r - 使用 R、RCurl 进行多 Web 表挖掘

Question

首先，提前感谢您的任何回复。

我需要通过在各自的网页中加入一些较小的表格来获得表格。迄今为止，我已经能够提取信息，但未能使用循环自动提取信息。迄今为止，我的命令是:

library(RCurl)
library(XML)
# index <- toupper(letters)
# EDIT:
index <- LETTERS

index[1] <- "0-A"
url <- paste("www.citefactor.org/journal-impact-factor-list-2014_", index, ".html", sep="", collapse=";")
urls <- strsplit(url, ";") [[1]]

这是我的循环尝试:

read.html.tab <- function(url){
 require(RCurl)
 require(XML)
 uri <- url
 tabs <- NULL
 for (i in uri){
  tabs <- getURL(uri)
  tabs <- readHTMLTable(tabs, stringsAsFactors = F)
  tab1 <- as.data.frame(tabs)
  }
 tab1
 }

如果我尝试使用 read.html.tab 函数:

tab0 <- read.html.tab(urls)

我收到以下错误: data.frame 错误(`Search Journal Impact Factor List 2014` = list(`0-A` = "N", : arguments imply different number of rows: 1, 1100, 447, 874, 169, 486, 201、189、172、837....

但是，如果 urls 只有一个元素，则该函数有效:

tabA <- read.html.tab(urls[1])
tabB <- read.html.tab(urls[2]) 
tab.if <- rbind(tabA,tabB)

ifacs <- tab.if[,27:ncol(tab.if)]
View(ifacs)

看来我不明白循环是如何工作的...

Answer 1

强制性 Hadleyverse 答案:

library(rvest)
library(dplyr)
library(magrittr)
library(pbapply)

urls <- sprintf("http://www.citefactor.org/journal-impact-factor-list-2014_%s.html", 
                c("0-A", LETTERS[-1]))

dat <- urls %>%
  pblapply(function(url) 
    html(url) %>% html_table(header=TRUE) %>% extract2(2)) %>%
  bind_rows()

glimpse(dat)

## Observations: 1547
## Variables:
## $ INDEX     (int) 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,...
## $ JOURNAL   (chr) "4OR-A Quarterly Journal of Operations Researc...
## $ ISSN      (chr) "1619-4500", "0891-0162", "0149-1423", "1550-7...
## $ 2013/2014 (chr) "0.918", "0.608", "1.832", "3.905", "1.776", "...
## $ 2012      (chr) "0.73", "0.856", "1.768", "4.386", "1.584", "0...
## $ 2011      (chr) "0.323", "0.509", "1.831", "5.086", "1.432", "...
## $ 2010      (chr) "0.69", "0.56", "1.964", "3.942", "1.211", "0....
## $ 2009      (chr) "0.75", "-", "1.448", "3.54", "1.19", "0.293",...
## $ 2008      (chr) "-", "-", "1.364", "-", "1.445", "0.352", "1.4...

rvest 给我们 html 和 html_table

我仅将 magrittr 用于 extract2，它只是包装 [[ 并且阅读起来更好 (IMO)。

pbapply 包包装了 *apply 函数并为您提供免费的进度条。

注意:bind_rows 在最新的 dplyr 中，所以在使用它之前先捕获它。