python - 基于 ajax 的数据加载后如何获取 Chrome webdriver 的最新内容

Question

我在 chrome webdriver selenium 上遇到一个问题，它没有给我更新的内容，它显示了以前的内容，但实际上在单击下一页链接后，新数据会附加到浏览器中，但当我通过驱动程序时，它会像以前一样给我.

网站链接为:www.abc.com

我的目标是提取所有工作链接。但我无法做到这一点，请在这方面帮助我。

job_links = []
per_page = 9
total_jobs = int(driver.find_element_by_css_selector(".search-results-count.total-jobs").text.split("(")[1].split(")")[0])
total_pages = math.ceil(total_jobs / per_page)

for x in range(1, total_pages):
    print("Page number: ", x)
    jobs_on_page = ""
    time.sleep(5)
    jobs_on_page = driver.find_elements_by_xpath("//div[@class='module job-card-wrapper col-md-4 col-xs-12 col-sm-6 corporate-regular background-white']")
    for job in jobs_on_page:
        print("job is:", job)
        job_link = job.find_element_by_xpath("./a").get_attribute('href').split("%")[0]
        job_links.append(job_link)
    # if x != (total_pages - 1):
    print("Hello Page: ", x)
    element = driver.find_element_by_xpath(
        "//div[@class='reinvent-pagination-next']//span[@class='arrow cta-arrow']")
    webdriver.ActionChains(driver).move_to_element(element).click(element).perform()
    # self.wait.until(EC.element_to_be_clickable((By.XPATH, "//div[@class='reinvent-pagination-next']//span[@class='arrow cta-arrow']"))).click()
    time.sleep(10)"

它反复为我提供首页作业链接，但是我的页面在 webdriver 中发生了变化。

Answer 1

引入 WebDriverWait() 和 visibility_of_all_elements_ located() 并遵循 css 选择器来获取所有链接。

使用无限 while 循环并使用 try.. except 检查下一个按钮是否可用

代码:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options)
driver.get("https://www.boom.com")

Alllinks=[]
while True:

      elements=WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.module > a[data-linkcomponentname='jobsearchblock']")))
      for link in elements:
         Alllinks.append(link.get_attribute('href'))
      try :
           next_btn=WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,'//a[@class="next-page-btn"]')))
           driver.execute_script("arguments[0].click();", next_btn)
      except:
           break
      time.sleep(1)
print('Total links :' + str(len(Alllinks)))
print(Alllinks)

输出:

Total links :90

['https://www.boom.com/ca-en/careers/jobdetails?id=00728259_en&title=Sales+Capture+Lead+%e2%80%93+Large-Scale+Consulting%2c+Technology+and+Operations+Sales', 'https://www.boom.com/ca-en/careers/jobdetails?id=00778020_en&title=Business+Operations+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00804572_en&title=Test+Automation+Engineer', 'https://www.boom.com/ca-en/careers/jobdetails?id=00780841_en&title=Consulting+Senior+Manager%2c+Automotive', 'https://www.boom.com/ca-en/careers/jobdetails?id=00788609_en&title=Senior+Integration+Architect', 'https://www.boom.com/ca-en/careers/jobdetails?id=00788884_en&title=E2E+Senior+Tester', 'https://www.boom.com/ca-en/careers/jobdetails?id=00739145_en&title=Oracle+Project+Portfolio+Management+Cloud+Consultant', 'https://www.boom.com/ca-en/careers/jobdetails?id=00777973_en&title=Executive+Assistant', 'https://www.boom.com/ca-en/careers/jobdetails?id=00756315_en&title=Azure+Consultant', 'https://www.boom.com/ca-en/careers/jobdetails?id=00798395_en&title=Technology+Delivery+Lead+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00783770_en&title=SAP+Customer+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00780180_en&title=Oracle+Cloud+integration+Architect', 'https://www.boom.com/ca-en/careers/jobdetails?id=00740026_en&title=Smart+Spend+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00765639_en&title=Hybris+Architect%2fDevelopment+Lead', 'https://www.boom.com/ca-en/careers/jobdetails?id=00765637_en&title=Hybris+Senior+Developer', 'https://www.boom.com/ca-en/careers/jobdetails?id=00801716_en&title=Senior+Cloud+Native+Architect', 'https://www.boom.com/ca-en/careers/jobdetails?id=00762181_en&title=Smart+Spend+Consultant', 'https://www.boom.com/ca-en/careers/jobdetails?id=00752420_en&title=Senior+Cloud+Architect', 'https://www.boom.com/ca-en/careers/jobdetails?id=00785832_en&title=Digital+Technology+Consulting+Senior+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00736712_en&title=Azure+Data+Architect+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00742724_en&title=Client+Financial+Management+Analyst', 'https://www.boom.com/ca-en/careers/jobdetails?id=00789817_en&title=SAP+Sourcing%2fProcurement+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00791760_en&title=SAP+HCM+Manager+-+H%26PS', 'https://www.boom.com/ca-en/careers/jobdetails?id=00782632_en&title=Workday+Integration+Senior+Analyst', 'https://www.boom.com/ca-en/careers/jobdetails?id=00775896_en&title=SAP+SCM+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00752413_en&title=Red+Hat+OpenShift+Cloud+Architect', 'https://www.boom.com/ca-en/careers/jobdetails?id=00759225_en&title=Cloud+Application+Architect', 'https://www.boom.com/ca-en/careers/jobdetails?id=00797835_en&title=SAP+S%2f4+HANA+EAM+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00778099_en&title=Front+Desk+Assistant%2fReception', 'https://www.boom.com/ca-en/careers/jobdetails?id=00734569_en&title=SAP+Payroll+Consultant', 'https://www.boom.com/ca-en/careers/jobdetails?id=00747056_en&title=SAP+Ariba+Delivery+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00684615_en&title=Solutions+Architect%2fManager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00740979_en&title=SAP+IBP+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00594586_en&title=Sales+Capture%2c+Senior+Manager+(Application+Services)', 'https://www.boom.com/ca-en/careers/jobdetails?id=00752409_en&title=Sr+Implementation+Specialist', 'https://www.boom.com/ca-en/careers/jobdetails?id=00784403_en&title=Senior+Technical+Consultant', 'https://www.boom.com/ca-en/careers/jobdetails?id=00778080_en&title=Marketing+Campaign+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00786043_en&title=Microservices%2fJava+Spring+Boot+Developer', 'https://www.boom.com/ca-en/careers/jobdetails?id=00774712_en&title=SAP+S%2f4+Finance+Consultant+-+SAP+Technology', 'https://www.boom.com/ca-en/careers/jobdetails?id=00756729_en&title=SAP+Delivery+Lead+-+SAP+Technology', 'https://www.boom.com/ca-en/careers/jobdetails?id=00758527_en&title=Management+Consulting+Manager+%e2%80%93+Utilities+T%26D+(Toronto)', 'https://www.boom.com/ca-en/careers/jobdetails?id=00789288_en&title=SAP+Finance+Manager+-+Health+and+Public+Services+Sector', 'https://www.boom.com/ca-en/careers/jobdetails?id=00789286_en&title=SAP+Finance+Consultant+-+Health+and+Public+Services+Sector', 'https://www.boom.com/ca-en/careers/jobdetails?id=00752355_en&title=Oracle+Cloud+SCM+Consutant', 'https://www.boom.com/ca-en/careers/jobdetails?id=00733096_en&title=Oracle+Cloud+-+Order+To+Cash+Functional+Consultant', 'https://www.boom.com/ca-en/careers/jobdetails?id=00782656_en&title=Sr+Oracle+Projects+Lead', 'https://www.boom.com/ca-en/careers/jobdetails?id=00756751_en&title=Data+Governance+Senior+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00789201_en&title=Technical+Consultant', 'https://www.boom.com/ca-en/careers/jobdetails?id=00768916_en&title=CI+Functional+Designer+-+Technology+Consultant', 'https://www.boom.com/ca-en/careers/jobdetails?id=00747893_en&title=SAP+S4+HANA+Finance+%e2%80%93+Senior+Manager+(IPT)', 'https://www.boom.com/ca-en/careers/jobdetails?id=00768965_en&title=Data+Engineering+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00721462_en&title=AEM+Architect', 'https://www.boom.com/ca-en/careers/jobdetails?id=00754980_en&title=Sales+Capture+Senior+Manager+-+Financial+Services', 'https://www.boom.com/ca-en/careers/jobdetails?id=00791449_en&title=Azure+Cloud+Operations+Lead', 'https://www.boom.com/ca-en/careers/jobdetails?id=00779191_en&title=Workday+Data+Consultant', 'https://www.boom.com/ca-en/careers/jobdetails?id=00785754_en&title=Organization+Change+Senior+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00752384_en&title=Full+Stack+Developer', 'https://www.boom.com/ca-en/careers/jobdetails?id=00766888_en&title=Oracle+Cloud+ERP+-+Business+Lead', 'https://www.boom.com/ca-en/careers/jobdetails?id=00770105_en&title=SAP+Finance+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00788292_en&title=Systems+Engineer+(Azure%2c+Cloud+%26+O365)', 'https://www.boom.com/ca-en/careers/jobdetails?id=00755903_en&title=Cloud+Engineer+Consulting+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00749401_en&title=Azure+Cloud+Architect', 'https://www.boom.com/ca-en/careers/jobdetails?id=00768544_en&title=Cloud+Native+Developer', 'https://www.boom.com/ca-en/careers/jobdetails?id=00773267_en&title=Global+Category+Management+Associate+Manager+(Canada)', 'https://www.boom.com/ca-en/careers/jobdetails?id=00752415_en&title=Pivotal+Cloud+Foundry+Developer', 'https://www.boom.com/ca-en/careers/jobdetails?id=00763409_en&title=Mulesoft+Architect', 'https://www.boom.com/ca-en/careers/jobdetails?id=00775495_en&title=Consulting+Manager+-+Contact+Center+Strategy+(Retail+Banking)', 'https://www.boom.com/ca-en/careers/jobdetails?id=00780965_en&title=SAP+Finance+Transformation+Senior+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00760167_en&title=SAP+Fieldglass+Consultant', 'https://www.boom.com/ca-en/careers/jobdetails?id=00780860_en&title=Oracle+Cloud+SCM+-+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00780864_en&title=Oracle+Cloud+Finance+-+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00751969_en&title=Innovation+and+Best+Practices+(F%26A)+Associate+Director', 'https://www.boom.com/ca-en/careers/jobdetails?id=00781338_en&title=SAP+SuccessFactors+Employee+Central+Consultant', 'https://www.boom.com/ca-en/careers/jobdetails?id=00779384_en&title=Vlocity+%2f+Salesforce+Developer', 'https://www.boom.com/ca-en/careers/jobdetails?id=00744256_en&title=SAP+S%2f4+HANA+Finance+Senior+Manager-+SAP+Technology', 'https://www.boom.com/ca-en/careers/jobdetails?id=00774716_en&title=SAP+Technical+Architect+-+Senior+Leader', 'https://www.boom.com/ca-en/careers/jobdetails?id=00756760_en&title=AWS+Cloud+Architect+Specialist', 'https://www.boom.com/ca-en/careers/jobdetails?id=00769005_en&title=SAP+SuccessFactors+Recruiting+%26+Onboarding+Consultant', 'https://www.boom.com/ca-en/careers/jobdetails?id=00735660_en&title=SAP+SuccessFactors+LMS+Consultant', 'https://www.boom.com/ca-en/careers/jobdetails?id=00736215_en&title=SAP+SuccessFactors+Technical+Solution+Architect', 'https://www.boom.com/ca-en/careers/jobdetails?id=00747061_en&title=SAP+S4+HANA+Supply+Chain+(SCM)+-+Senior+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00747058_en&title=SAP+S4+HANA+Central+Finance+Senior+Manager+-+SAP+Technologies', 'https://www.boom.com/ca-en/careers/jobdetails?id=00776370_en&title=ERP+(SAP%2c+Oracle)+Security+Senior+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00773097_en&title=Organization+Change+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00773095_en&title=Organization+Change+Consultant', 'https://www.boom.com/ca-en/careers/jobdetails?id=00773099_en&title=Organization+Change+Manager', 'https://www.boom.com/ca-en/careers/jobdetails?id=00768546_en&title=Cloud+Native+Senior+Application+Developer', 'https://www.boom.com/ca-en/careers/jobdetails?id=00766506_en&title=Mulesoft+Developer', 'https://www.boom.com/ca-en/careers/jobdetails?id=00748946_en&title=Senior+Software+Engineer%2fTeam+Lead', 'https://www.boom.com/ca-en/careers/jobdetails?id=00334756_en&title=Military+Service+Members+and+Veterans+-+Canada+%2b%2b']