I'm trying to scrape the Bing AI chat website, but my selenium isn't detecting the website or any elements on it at all. I tried the same with another website, like Wikipida and it works fine, but as soon as I try anything else it stops working. I have also tried increasing the wait time, and it has done absolutely nothing. Any idea as to what I can do?

我正在尝试抓取必应AI聊天网站，但我的Selify根本没有检测到该网站或其上的任何元素。我尝试了另一个网站，比如维基百科，它运行得很好，但一旦我尝试了其他任何东西，它就停止工作了。我也试过增加等待时间，但完全没有效果。你知道我能做什么吗？

    import time
    from selenium import webdriver
    from selenium.webdriver.common.keys import Keys
    from selenium.webdriver.chrome.options import Options
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.common.by import By
    import os
    
    
    
    
    
    chrome_options = Options()
    chrome_options.add_experimental_option("detach", True)
    
    # Adding argument to disable the AutomationControlled flag
    chrome_options.add_argument("--disable-blink-features=AutomationControlled")
    
    # Exclude the collection of enable-automation switches
    chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
    
    # Turn-off userAutomationExtension
    chrome_options.add_experimental_option("useAutomationExtension", False)
    
    # Setting the driver path and requesting a page
    
    driver = webdriver.Chrome(options=chrome_options)
    driver.get('https://www.bing.com/search?q=Bing+AI&showconv=1&FORM=hpcodx')
    time.sleep(3)
    try:
      WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="tone-options"]/li[3]/button'))).click()
    except:
        pass

It would be tricky, because Bing DOM structure is based on shadow-roots.
That means that to get access to inner elements structure, you must access to shadow-root host first.

这会很棘手，因为Bing DOM结构是基于阴影根的。这意味着要访问内部元素结构，您必须首先访问影子根主机。

What is shadow-root

什么是影子根

In your case, precise button is placed on 4th level of shadow-roots

在你的例子中，精确按钮被放置在阴影根的第四层

1. 1st host has locator .cib-serp-main


2. 2nd host has locator #cib-conversation-main


3. 3rd host has locator cib-welcome-container


4. 4th host (where your buttons are located) is cib-tone-selector

4th host is placed inside 3rd, 3rd is placed inside 2nd, 2nd places inside 1st.

第四位主机放在第三位内，第三位放在第二位内，第二位放在第一位内。

You can access to shadow-root content by getting shadowRoot property from element via JS.

您可以通过JS从元素获取shadowRoot属性来访问卷影根内容。

url = "https://www.bing.com/search?q=Bing+AI&showconv=1&FORM=hpcodx"

driver.get(url)
app_root = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'cib-serp-main')))
first_level_shadow_root = get_shadow_root(app_root)
second_level_shadow_root = get_shadow_root(first_level_shadow_root.find_element(By.ID, 'cib-conversation-main'))
third_level_shadow_root = get_shadow_root(second_level_shadow_root.find_element(By.CSS_SELECTOR,'cib-welcome-container'))
fourth_level_shadow_root = get_shadow_root(third_level_shadow_root.find_element(By.CSS_SELECTOR, 'cib-tone-selector'))
precise_button = fourth_level_shadow_root.find_element(By.CSS_SELECTOR, '#tone-options .tone-precise')

Example:

示例：

shadow_root_1 = browser.find_element(By.CLASS_NAME, 'cib-serp-main').shadow_root
shadow_root_2 = shadow_root_1.find_element(By.CSS_SELECTOR, 'cib-action-bar').shadow_root
shadow_root_3 = shadow_root_2.find_element(By.CSS_SELECTOR, 'cib-text-input').shadow_root

textarea = shadow_root_3.find_element(By.ID, 'searchbox')
textarea.send_keys("Your text here")
time.sleep(1)
textarea.send_keys(Keys.RETURN)

Full code(tested):

完整代码(已测试)：

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
import time
import json
import os
import sys
import pickle
from requests import Session
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from Screenshot import Screenshot
from selenium.webdriver.common.action_chains import ActionChains
import pyperclip
import re
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

chromeOptions = Options()
# chromeOptions.add_argument('--headless') # show or hide...
chromeOptions.add_argument("--no-sandbox")
chromeOptions.add_argument("--window-size=1280,720")
chromeOptions.add_argument('--disable-gpu')
chromeOptions.add_argument('--disable-dev-shm-usage')
chromeOptions.add_argument('--disable-web-security')
chromeOptions.add_argument('--disable-notifications')
chromeOptions.add_argument('-–allow-file-access-from-files')
chromeOptions.add_argument('--disable-site-isolation-trials')

caps = DesiredCapabilities.CHROME
caps['goog:loggingPrefs'] = {'performance': 'ALL'}

browser = webdriver.Chrome(options=chromeOptions)

try:
    browser.get("https://www.bing.com/search?q=Bing+AI&showconv=1&FORM=hpcodx")
    time.sleep(10)

    # Find the element by link text and click it
    try:
        element = browser.find_element(By.XPATH, "//a[contains(text(), 'Accept')]")
        element.click()
    except Exception as e:
        print("An error occurred:", str(e))

    time.sleep(10)

    shadow_root_1 = browser.find_element(By.CLASS_NAME, 'cib-serp-main').shadow_root
    shadow_root_2 = shadow_root_1.find_element(By.CSS_SELECTOR, 'cib-action-bar').shadow_root
    shadow_root_3 = shadow_root_2.find_element(By.CSS_SELECTOR, 'cib-text-input').shadow_root

    textarea = shadow_root_3.find_element(By.ID, 'searchbox')
    textarea.send_keys("Your text here")
    time.sleep(1)
    textarea.send_keys(Keys.RETURN)

    time.sleep(10)

#     ob = Screenshot.Screenshot()
#     ob.full_screenshot(browser, save_path=r'.', image_name='page_source.png', is_load_at_runtime=True, load_wait_time=1)
    print("DONE")
except Exception as e:
    print("An error occurred:", str(e))
finally:
    browser.quit()