python - python 中 facebook 的网络爬虫

Question

我正在尝试使用 python 中的网络爬虫来打印 facebook 推荐者的数量。例如这篇来自天空新闻的文章(http://news.sky.com/story/1330046/are-putins-little-green-men-back-in-ukraine)大约有 60 个 facebook 推荐。我想用网络爬虫在 python 程序中打印这个数字。 我尝试这样做，但它没有打印任何内容:

import requests
from bs4 import BeautifulSoup

def get_single_item_data(item_url):
    source_code = requests.get(item_url)
    plain_text = source_code.text
    soup = BeautifulSoup(plain_text)
    # if you want to gather information from that page
    for item_name in soup.findAll('span', {'class': 'pluginCountTextDisconnected'}):
        try:
                print(item_name.string)
        except:
                print("error")

get_single_item_data("http://news.sky.com/story/1330046/are-putins-little-green-men-back-in-ukraine")

Answer 1

Facebook 推荐在 iframe 中加载。 您可以跟随 iframe src 属性到该页面，然后加载 span.pluginCountTextDisconnected 的文本:

import requests
from bs4 import BeautifulSoup

url = 'http://news.sky.com/story/1330046/are-putins-little-green-men-back-in-ukraine'
r = requests.get(url) # get the page through requests
soup = BeautifulSoup(r.text) # create a BeautifulSoup object from the page's HTML

url = soup('iframe')[0]['src'] # search for the iframe element and get its src attribute
r = requests.get('http://' + url[2:]) # get the next page from requests with the iframe URL
soup = BeautifulSoup(r.text) # create another BeautifulSoup object

print(soup.find('span', class_='pluginCountTextDisconnected').string) # get the directed information

第二个 requests.get 是这样写的，因为 src 属性返回 //www.facebook.com/plugins/like.php?href=http%3A%2F% 2Fnews.sky.com%2Fstory%2F1330046&send=false&layout=button_count&width=120&show_faces=false&action=recommend&colorscheme=light&font=arial&height=21。我添加了 http:// 并忽略了前导 //。

---

BeautifulSoup documentation
Requests documentation