个人中心
gpt4 book ai didi

python - 使用 Python Selenium 时数据/内容丢失

转载 作者：行者123 更新时间：2023-12-01 06:23:49 25 4
gpt4 key购买 nike

我正在测试如何在 python 中使用 Selenium，并在 Ubuntu 16.04 中通过以下代码成功打开页面:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from selenium import webdriver 
from selenium.webdriver.firefox.options import Options 
firefox_options = Options()
firefox_options.binary_location = '/usr/bin/firefox'  
driver= webdriver.Firefox(executable_path='/home/myname/geckodriver',firefox_options=firefox_options)
driver.get('https://www.toutiao.com')

但是，与手动打开此页面('https://www.toutiao.com')相比，缺少一些数据/内容。

enter image description here

我的 Firefox 版本是“72.0.2”，geckodriver 版本是“0.26.0”。有人可以帮我解决这个问题吗？提前致谢!

最佳答案

我采用了您的代码，简化了脚本，在执行时我遇到了类似的问题，即与打开此页面相比缺少数据/内容，如下所示:

  • 代码块:

        from selenium import webdriver

        driver = webdriver.Firefox(executable_path=r'C:\Utility\BrowserDrivers\geckodriver.exe')
        driver.get('https://www.toutiao.com')
        print(driver.page_source)

  • 控制台输出:

        <html><head><style class="vjs-styles-defaults">
              .video-js {
            width: 300px;
            height: 150px;
              }

              .vjs-fluid {
            padding-top: 56.25%
              }
            </style><meta charset="utf-8"><title>????</title><meta http-equiv="x-dns-prefetch-control" content="on"><meta name="renderer" content="webkit"><link rel="dns-prefetch" href="//s3.pstatp.com/"><link rel="dns-prefetch" href="//s3a.pstatp.com/"><link rel="dns-prefetch" href="//s3b.pstatp.com"><link rel="dns-prefetch" href="//p1.pstatp.com/"><link rel="dns-prefetch" href="//p3.pstatp.com/"><meta http-equiv="Content-Security-Policy" content="upgrade-insecure-requests"><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1,minimum-scale=1,user-scalable=no,minimal-ui"><meta name="360-site-verification" content="b96e1758dfc9156a410a4fb9520c5956"><meta name="360_ssp_verify" content="2ae4ad39552c45425bddb738efda3dbb"><meta name="google-site-verification" content="3PYTTW0s7IAfkReV8wAECfjIdKY-bQeSkVTyJNZpBKE"><meta name="shenma-site-verification" content="34c05607e2a9430ad4249ed48faaf7cb_1432711730"><meta name="baidu_union_verify" content="b88dd3920f970845bad8ad9f90d687f7"><meta name="domain_verify" content="pmrgi33nmfuw4ir2ej2g65lunfqw6ltdn5wselbcm52wszbchirdqyztge3tenrsgq3dknjume2tayrvmqytemlfmiydimddgu4gcnzcfqrhi2lnmvjwc5tfei5dcnbwhazdcobuhe2dqobrpu"><meta name="keywords" content="????,??,???,????,??????"><meta name="description" content="«????»(www.toutiao.com)????????????????,?????????????????,?????????????,??????????????????????"><link rel="alternate" media="only screen and (max-width: 640px)" href="//m.toutiao.com/"><link rel="shortcut icon" href="//s3a.pstatp.com/toutiao/resource/ntoutiao_web/static/image/favicon_5995b44.ico" type="image/x-icon"><link rel="stylesheet" href="//s3.pstatp.com/toutiao/player/dist/pc_vue2.css" media="screen" title="no title"><!--[if lt IE 9]>
          <p>?????????,?<a href="http://browsehappy.com/">?????</a></p>
        .
        .
        .
        <script>var imgUrl = '/c/9ubkblw9out4h9t6ya05r7h0uu7q2u341jhsdh7l4r4yphpuxlqgdm/';</script><script>tac='i+2gv2ch1tigds!i$1dmgs"yZl!%s"l"u&kLs#l l#vr*charCodeAtx0[!cb^i$1em7b*0d#>>>s j\uffeel  s#0,<8~z|\x7f@QGNCJF[\\^D\\KFYSk~^WSZhg,(lfi~ah`{md"inb|1d<,%Dscafgd"in,8[xtm}nLzNEGQMKAdGG^NTY\x1ckgd"inb<b|1d<g,&TboLr{m,(\x02)!jx-2n&vr$testxg,%@tug{mn ,%vrfkbm[!cb|'</script><script type="text/javascript" crossorigin="anonymous" src="//s3b.pstatp.com/toutiao/static/js/vendor.63b66d4280309ac2fb48.js"></script><script type="text/javascript" crossorigin="anonymous" src="//s3a.pstatp.com/toutiao/static/js/page/index_node/index.e6afc60a3a3f653cfdba.js"></script><script type="text/javascript" crossorigin="anonymous" src="//s3b.pstatp.com/toutiao/static/js/ttstatistics.a083f6cd9b1a9a970725.js"></script><script src="//s3.pstatp.com/inapp/lib/raven.js" crossorigin="anonymous"></script><script>;(function(window) {
            // sentry
            window.Raven && Raven.config('//key@m.toutiao.com/log/sentry/v2/96', {
              whitelistUrls: [/pstatp\.com/],
              shouldSendCallback: function(data) {
            var ua = navigator && navigator.userAgent;
            var isDeviceOK = !/Mobile|Linux/i.test(navigator.userAgent);
            return isDeviceOK;
              },
              tags: {
            bid: 'toutiao_pc',
            pid: 'index_new'
              },
              autoBreadcrumbs: {
            'xhr': false,
            'console': true,
            'dom': true,
            'location': true
              }
            }).install();
          })(window);</script><script>document.getElementsByTagName('body')[0].addEventListener('click', function(e) {
            var target = e.target,
            ga_event,
            ga_category,
            ga_label,
            ga_value;
            while(target && target.nodeName.toUpperCase() !== 'BODY') {
              ga_event = target.getAttribute('ga_event');
              ga_category = target.getAttribute('ga_category') || '/';
              ga_label = target.getAttribute('ga_label') || '';
              ga_value = target.getAttribute('ga_value') || 1;

              ga_event && window.ttAnalysis && ttAnalysis.send('event', { ev: ga_event });
              target = target.parentNode;
            }
          });</script><script src="https://xxbg.snssdk.com/websdk/v1/getInfo?q=YOsueEs6CjZquUQrQwttBa2p27c%2FmJBGcEmZKypwf%2Fh%2B%2FFzCVrIwzk9L3bo%2FZb2O8gVTNaA4L2Bk10qWfZ2s94e6qe8KRXlOEjnI%2FrONB4jQynV3bfJ9exD2E4QPsgydRGjRLlDXE9uYD7HU3IZ%2FOU2MJG2vMgfNU55%2FmsOAlVSrPQH2wo4Eor0lgghKHjRi28vVvBdKY7JT4gG7S7ThRFD2YBIc%2Fs4JYViQu1Ll1Bg5Xn5bKuD6jZRz3AzfFqzSOWguO6vUbzL0wBc4mpa22mdpmAXIvUNWtjg5MUfXh9rfWI0ti7saL%2B0r4%2BaBdN5y4lrmxAcQZq2oeAKl4WjOeJsN%2BePpYmisoxTzdBZ6TL8IGE0E7ZUUlFlPGyUWhU3E4IRbtbCCd0QdVaJajiSOIhg9cImqTZYI56kIao1yVnV%2Bxu4%2BhaC1kHu5xsk49%2BX%2FNdwGcel%2BlOUzagkE5s8X6jEswA7jzW%2ByD6%2FusfkNyyx8WOWCJmZlTGQ4SNQr%2FQHvmK2QscQ7KnTvKVqjedUd7IFcvyTyYz3iFFrmRkOMRN9042sLiQwerXsn0f%2Fc%2Bh46PNdeU1S6BsFKq%2BZhMDxw1vI2Y1C%2Fa0RBdZC%2BGZq%2BkbNaoVotfvslg05ahevHTainlZR9DHEiWawFBJbTwjMeYrmo4NZiL5eNBUvslFn%2BDPHk%2F6Oj0Nbb89Rx8Ihi2pRH04voRog9848H2o2LR9gx0N0i0o6%3D&amp;callback=_8712_1581940674310"></script></body></html>
<小时/>

分析

在检查DOM Tree时在网页中，您会发现一些<script>标签指的是具有关键字dist的JavaScript。举个例子:

  • <link rel="stylesheet" href="//s3.pstatp.com/toutiao/player/dist/pc_vue2.css" media="screen" title="no title">
  • <script src="//unpkg.pstatp.com/byted/sec_sdk_build/1.1.12/dist/captcha.js"></script>
  • //s3a.pstatp.com/toutiao/picc_mig/dist/img.min.js

这清楚地表明该网站受到机器人管理服务提供商的保护Distil Networks ChromeDriver 的导航会被检测到并随后被阻止。

<小时/>

蒸馏

根据文章There Really Is Something About Distil.it... :

Distil protects sites against automatic content scraping bots by observing site behavior and identifying patterns peculiar to scrapers. When Distil identifies a malicious bot on one site, it creates a blacklisted behavioral profile that is deployed to all its customers. Something like a bot firewall, Distil detects patterns and reacts.

此外，

"One pattern with **Selenium** was automating the theft of Web content", Distil CEO Rami Essaid said in an interview last week. "Even though they can create new bots, we figured out a way to identify Selenium the a tool they're using, so we're blocking Selenium no matter how many times they iterate on that bot. We're doing that now with Python and a lot of different technologies. Once we see a pattern emerge from one type of bot, then we work to reverse engineer the technology they use and identify it as malicious".

<小时/>

引用

您可以在以下位置找到一些详细的讨论:

关于python - 使用 Python Selenium 时数据/内容丢失，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/60261510/

25 4 0
文章推荐： jquery - 如果元素包含单词 X 则执行 Y
文章推荐： asp.net - 路由错误指的是不再存在的命名空间
文章推荐： DataFrame 上的 Python 多级索引。访问第一个索引级别的第一行以应用函数
文章推荐： sql-server - SQL USER_NAME() Windows 身份验证
行者123
个人简介

我是一名优秀的程序员,十分优秀！

滴滴打车优惠券免费领取
滴滴打车优惠券
全站热门文章
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com