web-scraping - 进入每个链接，找到文件类型并下载

Question

我想知道是否有任何解决方案可以使用 VBscript 从网站下载文件？

我知道如何从网站下载单个文件，但如何使其成为循环？另外，如何在特定页面中搜索特定文件扩展名并下载文件(如果可用)？

For each pdf in website
  xhr.open "GET", pdf.src, False
  xhr.send
  set stream = CreateObject("Adodb.Stream")
  with stream
    .type = 1
    .Open
    .Write xhr.responsebody
    .SaveToFile "C:\temp\" + CStr(index) + ".pdf", 2
  end with 
  stream.Close
  set stream = nothing
  index = index + 1 
Next

假设我们有一个网站 https://website.com/productpage/然后是所有具有相同结构的链接 https://website.com/products/xx-x-xx-x/所以所有需要的链接都以 https://website.com/products/ 开头.根据源代码，这种链接似乎有 33 个。

然后在进行到某个页面后会有 PDF 文件。有时是 1 个，有时是 3 或 4 个。但是，PDF 文件的链接类似于 https://website.com/wp-content/uploads/2016/12/xxxx.pdf其中 xxxx.pdf 实际上可以是文件名。

这是我设法为一个文件获得的内容:

dim xHttp: Set xHttp = createobject("Microsoft.XMLHTTP")
dim bStrm: Set bStrm = createobject("Adodb.Stream")
xHttp.Open "GET", "https://website.com/wp-content/uploads/2016/12/xxxx.pdf", False
xHttp.Send

with bStrm
    .type = 1 '//binary
    .open
    .write xHttp.responseBody
    .savetofile "c:\temp\xxxx.pdf", 2 '//overwrite
end with

编辑:

应该是这样的:

获取所有需要的链接

进入各个链接

搜索以“.pdf”结尾的链接

下载文件到 C:\temp\

网站结构:

https://website.com/productpage/
        https://website.com/products/xx-x/
              https://website.com/wp-content/uploads/2016/12/xx-xx.pdf
        https://website.com/products/xxxxx-xsx/
              https://website.com/wp-content/uploads/2018/12/x-xx-x.pdf
              https://website.com/wp-content/uploads/2015/12/x-x-xx.pdf
              https://website.com/wp-content/uploads/2019/12/xxx-x.pdf
        https://website.com/products/x-xx-xsx/
              https://website.com/wp-content/uploads/2014/12/x-xxx.pdf
              https://website.com/wp-content/uploads/2013/12/x-x-x-x.pdf
        https://website.com/products/xx-x-xsx/
              https://website.com/wp-content/uploads/2012/12/x-xxxx.pdf

Answer 1

既然你有保存链接的代码，你可以把它包装成一个子来重复使用:

Sub GetFile(p_sRemoteFile, p_sLocalFile)

    Dim xHttp: Set xHttp = CreateObject("Microsoft.XMLHTTP")
    Dim bStrm: Set bStrm = CreateObject("Adodb.Stream")
    xHttp.open "GET", p_sRemoteFile, False
    xHttp.Send

    With bStrm
        .Type = 1 '//binary
        .open
        .write xHttp.responseBody
        .SaveToFile p_sLocalFile, 2 '//overwrite
    End With

End Sub

然后，您可以使用 InternetExplorer 对象获取页面中的链接集合:

Sub GetPageLinks(p_sURL)
    Dim objIE
    Dim objLinks
    Dim objLink
    Dim iCounter

    Set objIE = CreateObject("InternetExplorer.Application")
    objIE.Visible = True
    objIE.Navigate p_sURL

    Do Until objIE.ReadyState = 4
        Wscript.Sleep 100
    Loop

    Set objLinks = objIE.Document.All.Tags("a")
    For iCounter = 1 To objLinks.Length
        Set objLink = objLinks(iCounter - 1)
        With objLink
            If StrComp(Right(.href, 3), "pdf", 1) = 0 Then
                ' Get file
                GetFile .href, "C:\temp\downloads\" & GetFileNameFromURL(.href)
            Else
                ' Process page
                GetPageLinks .href
            End If
        End With
    Next

End Sub

这是一个从 URL 中提取文件名的函数:

Function GetFileNameFromURL(p_sURL)
    Dim arrFields

    arrFields = Split(p_sURL, "/")

    GetFileNameFromURL = arrFields(UBound(arrFields))

End Function

此函数将返回 xxxx.pdf给定 https://website.com/wp-content/uploads/2016/12/xxxx.pdf .