python - 无法同时从两个不同深度刮取不同字段

Question

我用 python 编写了一个脚本，用于从网页的登陆页面中抓取不同餐厅的名称、地址和电话并解析每个餐厅内页的作者和评论。

I would like to generate results using yield within get_additional_info(link) function but print the same within get_links(link) function together with other results.

Website address

到目前为止我已经写过:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

url = "https://www.yellowpages.com/search?search_terms=restaurant&geo_location_terms=San+Francisco%2C+CA"
base = "https://www.yellowpages.com"

def get_links(link):
    res = requests.get(link,headers={'User-Agent':'Mozilla/5.0'})
    soup = BeautifulSoup(res.text,"lxml")
    for item in soup.select(".v-card"):

        inner_link = item.select_one("a.business-name")
        author,review = get_additional_info(urljoin(base,inner_link.get('href')))

        title = inner_link.text
        address = item.select_one("p.adr").get_text(strip=True)
        phone = item.select_one(".phone").text
        yield title,address,phone,author,review


def get_additional_info(link):
    res = requests.get(link,headers={'User-Agent':'Mozilla/5.0'})
    soup = BeautifulSoup(res.text,"lxml")
    for elem in soup.select("article[class='clearfix']"):
        try:
            author = elem.select_one(".review-info a.author").text
        except AttributeError: author = ""
        try:
            review = elem.select_one(".review-response > p").text
        except AttributeError: review = ""

        yield author, review

if __name__ == '__main__':
    for item in get_links(url):
        print(item)

如果我运行上面的脚本，它会抛出以下错误，指向 author,review = get_additional_info(urljoin(base,inner_link.get('href'))) 行:

Traceback (most recent call last):
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\demo.py", line 36, in <module>
    for item in get_links(url):
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\demo.py", line 14, in get_links
    author,review = get_additional_info(urljoin(base,inner_link.get('href')))
ValueError: too many values to unpack (expected 2)

我希望抓取的所有字段都已正确定义(选择器)。

这就是the output我追求的是:

PS I wish to stick to the way I've already tried, meaning I do not want to parse everything from inner pages as the data are useless to me.

Answer 1

如果我理解正确的话，您想要“加入”链接和附加信息。一种方法是这样的:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

from textwrap import shorten

url = "https://www.yellowpages.com/search?search_terms=restaurant&geo_location_terms=San+Francisco%2C+CA"
base = "https://www.yellowpages.com"

def get_links(session, link):
    res = session.get(link,headers={'User-Agent':'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0'})

    soup = BeautifulSoup(res.text,"lxml")
    for item in soup.select(".v-card"):

        inner_link = item.select_one("a.business-name")

        title = inner_link.text
        address = item.select_one("p.adr").get_text(strip=True)
        phone = item.select_one(".phone").text

        for author, review in get_additional_info(session, urljoin(base,inner_link.get('href'))):
            yield title,address,phone,author,review


def get_additional_info(session, link):
    res = session.get(link,headers={'User-Agent':'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0'})

    soup = BeautifulSoup(res.text,"lxml")
    for elem in soup.select("article[class='clearfix']"):
        try:
            author = elem.select_one(".review-info a.author").text
        except AttributeError: author = ""
        try:
            review = elem.select_one(".review-response > p").text
        except AttributeError: review = ""

        yield author, review

if __name__ == '__main__':
    with requests.session() as s:
        # this sets all cookies
        res = s.get("https://www.yellowpages.com", headers={'User-Agent':'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0'}).text

        for title,address,phone,author,review in get_links(s, url):
            print('{: <30}{: <30}{: <20}{: <20}{}'.format(shorten(title, 30), shorten(address, 30), shorten(phone, 20), shorten(author, 20), shorten(review, 60)))

打印:

El Toreador Restaurant        50 W Portal Ave, San [...]    (415) 347-3294      Mark I.             Their food is good but i think they need to improve on [...]
El Toreador Restaurant        50 W Portal Ave, San [...]    (415) 347-3294      Cathy L.            This place is pretty much my go to place is I want [...]
El Toreador Restaurant        50 W Portal Ave, San [...]    (415) 347-3294      Mary C.             They have so many things in here worth going in here [...]
El Toreador Restaurant        50 W Portal Ave, San [...]    (415) 347-3294      Claude R.           The appetizers in here are enough to make you ask for [...]
El Toreador Restaurant        50 W Portal Ave, San [...]    (415) 347-3294      Felicia M.          How can this be? This place looks like magic and their [...]
El Toreador Restaurant        50 W Portal Ave, San [...]    (415) 347-3294      Jose H.             I feel like I just got from Mexico, we went here last [...]
El Toreador Restaurant        50 W Portal Ave, San [...]    (415) 347-3294                          Authentic Mexican. Always busy and the house salsa is [...]
El Toreador Restaurant        50 W Portal Ave, San [...]    (415) 347-3294                          I'm disappointed. The decor is ecclectic and fun, the [...]
El Toreador Restaurant        50 W Portal Ave, San [...]    (415) 347-3294                          This used to be one of my favorite restaurants until I [...]
El Toreador Restaurant        50 W Portal Ave, San [...]    (415) 347-3294                          I came to this restarnt for a birthday of a friend of [...]
El Toreador Restaurant        50 W Portal Ave, San [...]    (415) 347-3294                          The reviews here, which I consulted before going, were [...]
El Toreador Restaurant        50 W Portal Ave, San [...]    (415) 347-3294                          I have been told to give it a try.Food is on [...]
El Toreador Restaurant        50 W Portal Ave, San [...]    (415) 347-3294                          Great food... love the empalmada... sort of like a [...]
El Toreador Restaurant        50 W Portal Ave, San [...]    (415) 347-3294                          Definitely the best Mexican restaurant in town!... [...]
El Toreador Restaurant        50 W Portal Ave, San [...]    (415) 347-3294                          This place has been consistenly good for a few years. [...]
El Toreador Restaurant        50 W Portal Ave, San [...]    (415) 347-3294                          So-so Mexican food served by a vaguely condescending, [...]
El Toreador Restaurant        50 W Portal Ave, San [...]    (415) 347-3294                          since the place is small, it gets crowded quickly and [...]
El Toreador Restaurant        50 W Portal Ave, San [...]    (415) 347-3294                          Go early if you don't want to wait. They don't take [...]
El Toreador Restaurant        50 W Portal Ave, San [...]    (415) 347-3294                          A great place where you belong like part of the [...]
House Of Prime Rib            1906 Van Ness Ave, San [...]  (415) 636-6476      Keith Y.            Loved this place. Food and service was amazing
House Of Prime Rib            1906 Van Ness Ave, San [...]  (415) 636-6476      Quintrell P.        Was really hungry and needed a place to get some [...]
House Of Prime Rib            1906 Van Ness Ave, San [...]  (415) 636-6476      Len K.              I'm not usually a fan of red meat, but I'm definitely [...]
House Of Prime Rib            1906 Van Ness Ave, San [...]  (415) 636-6476      Emm C.              I haven't been able to see San Francisco, one of my [...]
House Of Prime Rib            1906 Van Ness Ave, San [...]  (415) 636-6476      James O.            For me, it`s one of the best ribs in town, I give [...]
House Of Prime Rib            1906 Van Ness Ave, San [...]  (415) 636-6476      Jing H.             This is one of the best places if you are craving for [...]

...etc.