python爬虫ieee论文关键词

Question

我尝试使用爬虫来获取 ieee 论文关键字，但现在出现错误如何修复我的爬虫？我的代码在这里

import requests
import json
from bs4 import BeautifulSoup
ieee_content = requests.get("http://ieeexplore.ieee.org/document/8465981", timeout=180)
soup = BeautifulSoup(ieee_content.text, 'xml')
tag = soup.find_all('script')
for i in tag[9]:
    s = json.loads(re.findall('global.document.metadata=(.*;)', i)[0].replace("'", '"').replace(";", ''))

这里有错误

Traceback (most recent call last):
  File "G:/github/爬蟲/redigg-leancloud/crawlers/sup_ieee_keywords.py", line 90, in <module>
    a.get_es_data(offset=0, size=1)
  File "G:/github/爬蟲/redigg-leancloud/crawlers/sup_ieee_keywords.py", line 53, in get_es_data
    self.get_data(link=ieee_link, esid=es_id)
  File "G:/github/爬蟲/redigg-leancloud/crawlers/sup_ieee_keywords.py", line 65, in get_data
    s = json.loads(re.findall('global.document.metadata=(.*;)', i)[0].replace(";", '').replace("'", '"'))
IndexError: list index out of range

Answer 1

这是另一个答案。我不知道在我的代码中加载(替换)后，您在代码中使用“s”做什么。

下面的代码不会抛出错误，但是你如何使用 's'

import requests
import json
from bs4 import BeautifulSoup

ieee_content = requests.get("http://ieeexplore.ieee.org/document/8465981", timeout=180)
soup = BeautifulSoup(ieee_content.text, 'xml')
tag = soup.find_all('script')

# i is a list
for i in tag[9]:
   metadata_format = re.compile(r'global.document.metadata=.*', re.MULTILINE)
   metadata = re.findall(metadata_format, i)
   if len(metadata) != 0:
      # convert the list 
      convert_to_json = json.dumps(metadata)
      x = json.loads(convert_to_json)
      s = x[0].replace("'", '"').replace(";", '')
      ###########################################
      # I don't know what you plan to do with 's'
      ###########################################
      print (s)