python - Scrapy - urlparse.urljoin 的行为与 str.join 相同吗？

Question

我正在尝试在 Scrapy 蜘蛛中使用 urlparse.urljoin 来编译要抓取的 url 列表。目前，我的蜘蛛没有返回任何内容，但没有抛出任何错误。所以我试图检查我是否正确编译了网址。

我的尝试是使用 str.join 在空闲状态下进行测试，如下所示:

>>> href = ['lphs.asp?id=598&city=london',
 'lphs.asp?id=480&city=london',
 'lphs.asp?id=1808&city=london',
 'lphs.asp?id=1662&city=london',
 'lphs.asp?id=502&city=london',]
>>> for x in href:
    base = "http:/www.url-base.com/destination/"
    final_url = str.join(base, x)
    print(final_url)

一行返回内容:

lhttp:/www.url-base.com/destination/phttp:/www.url-base.com/destination/hhttp:/www.url-base.com/destination/shttp:/www。 url-base.com/destination/.http:/www.url-base.com/destination/ahttp:/www.url-base.com/destination/shttp:/www.url-base.com/destination/phttp:/www.url-base.com/destination/?http:/www.url-base.com/destination/ihttp:/www.url-base.com/destination/dhttp:/www.url-base.com/destination/=http:/www.url-base.com/destination/5http:/www.url-base.com/destination/9http:/www.url-base.com/destination/8http:/www.url-base。 com/destination/&http:/www.url-base.com/destination/chttp:/www.url-base.com/destination/ihttp:/www.url-base.com/destination/thttp:/www.url- base.com/destination/yhttp:/www.url-base.com/destination/=http:/www.url-base.com/destination/lhttp:/www.url-base.com/destination/ohttp:/www .url-base.com/destination/nhttp:/www.url-base.com/destination/dhttp:/www.url-base.com/destination/ohttp:/www.url-base.com/destination/n

我认为从我的示例中可以明显看出 str.join 的行为方式不同 - 如果确实如此，那么这就是为什么我的蜘蛛没有跟踪这些链接! - 不过，最好能对此进行确认。

如果这不是正确的测试方法，我该如何测试这个过程？

更新尝试使用下面的 urlparse.urljoin:从 urllib.parse 导入 urlparse

    >>> from urllib.parse import urlparse
    >>> for x in href:
        base = "http:/www.url-base.com/destination/"
        final_url = urlparse.urljoin(base, x)
        print(final_url)

抛出AttributeError:'function'对象没有属性'urljoin'

更新 - 有问题的蜘蛛功能

def parse_links(self, response): 
    room_links = response.xpath('//form/table/tr/td/table//a[div]/@href').extract() # insert xpath which contains the href for the rooms 
    for link in room_links:
        base_url = "http://www.example.com/followthrough"
        final_url = urlparse.urljoin(base_url, link)
        print(final_url)
        # This is not joing the final_url right
        yield Request(final_url, callback=parse_links)

更新

我刚刚在空闲状态下再次测试:

>>> from urllib.parse import urljoin
>>> from urllib import parse
>>> room_links = ['lphs.asp?id=562&city=london',
 'lphs.asp?id=1706&city=london',
 'lphs.asp?id=1826&city=london',
 'lphs.asp?id=541&city=london',
 'lphs.asp?id=1672&city=london',
 'lphs.asp?id=509&city=london',
 'lphs.asp?id=428&city=london',
 'lphs.asp?id=614&city=london',
 'lphs.asp?id=336&city=london',
 'lphs.asp?id=412&city=london',
 'lphs.asp?id=611&city=london',]
>>> for link in room_links:
    base_url = "http:/www.url-base.com/destination/"
    final_url = urlparse.urljoin(base_url, link)
    print(final_url)

哪个抛出了这个:

Traceback (most recent call last):
  File "<pyshell#34>", line 3, in <module>
    final_url = urlparse.urljoin(base_url, link)
AttributeError: 'function' object has no attribute 'urljoin'

Answer 1

您会看到由于以下原因给出的输出:

for x in href:
    base = "http:/www.url-base.com/destination/"
    final_url = str.join(base, href)   # <-- 'x' instead of 'href' probably intended here
    print(final_url)

urljoin来自 urllib 库的行为有所不同，只需参阅文档即可。这不是简单的字符串连接。

编辑:根据您的评论，我认为您正在使用 Python 3。使用该 import 语句，您导入 urlparse功能。这就是您收到该错误的原因。要么导入并直接使用该函数:

from urllib.parse import urljoin
...
final_url = urljoin(base, x)

或导入parse模块并使用如下函数:

from urllib import parse
...
final_url = parse.urljoin(base, x)