gpt4 book ai didi

encoding - 如何解码十进制的 cp1252 “ 而不是\x93?

转载 作者:行者123 更新时间:2023-12-03 06:26:01 29 4
gpt4 key购买 nike

我正在获取一个网页的源代码,编码是cp1252。Chrome 正确显示页面。

这是我的代码:

import sys
from urllib.request import urlopen
from bs4 import BeautifulSoup, UnicodeDammit
import re
import codecs

url = "http://www.sec.gov/Archives/edgar/data/1400810/000119312513211026/d515005d10q.htm"
page = urlopen(url).read()
print(page)
# A little preview :
# b'...Regulation S-T (&#167;232.405 of this chapter) during the preceding 12 months (or for such shorter period that the\nregistrant was required to submit and post such files).&nbsp;&nbsp;&nbsp;&nbsp;Yes&nbsp;&nbsp;<FONT STYLE="FONT-FAMILY:WINGDINGS">&#120;</FONT>...'

soup = BeautifulSoup(page, from_encoding="cp1252")
print(str(soup).encode('utf-8'))
# Same preview section as above
# b'...Regulation S-T (\xc2\xa7232.405 of this chapter) during the preceding 12 months (or for such shorter period that the\nregistrant was required to submit and post such files).\xc2\xa0\xc2\xa0\xc2\xa0\xc2\xa0Yes\xc2\xa0\xc2\xa0<font style="FONT-FAMILY:WINGDINGS">x</font>'

从预览部分我们可以看到
 \; =\xc2\xa0
§ =\xc2\xa7
x = x

对于cp1252编码标准,我指的是 http://en.wikipedia.org/wiki/Windows-1252#Code_page_layout和/Lib/encodings/cp1252.py

当我使用 BeautifulSoup(page, from_encoding="cp1252") 时,某些字符编码正确,但其他一些字符编码不正确。

角色|十进制编码 | cp1252->utf-8编码
“| “ |\xc2\x93(错误)
” | ” |\xc2\x94(错误)
X | x |\xc2\x92(错误)
§| § |\xc2\xa7(确定)
þ | þ
¡ ¨
' | ’ |\xc2\x92(错误)
– | –

我使用此代码来获得等效项:

characters = "’ “ ” X § þ ¨ ' –"
list = characters.split()

for ch in list:
print(ch)
cp1252 = ch.encode('cp1252')
print(cp1252)

decimal = cp1252[0]

special = "&#" + str(decimal)
print(special)
print(ch.encode('utf-8'))
print()

offenders = [120, 146]

for n in offenders:
toHex = hex(n)
print(toHex)
print()

#120
off = b'\x78'
print(off)
buff = off.decode('cp1252')
print(buff)
uni = buff.encode('utf-8')
print(uni)
print()

#146
off = b'\x92'
print(off)
buff = off.decode('cp1252')
print(buff)
uni = buff.encode('utf-8')
print(uni)
print()

输出


b'\x92'
&#146
b'\xe2\x80\x99'


b'\x93'
&#147
b'\xe2\x80\x9c'


b'\x94'
&#148
b'\xe2\x80\x9d'

X
b'X'
&#88
b'X'

§
b'\xa7'
&#167
b'\xc2\xa7'

þ
b'\xfe'
&#254
b'\xc3\xbe'

¨
b'\xa8'
&#168
b'\xc2\xa8'

'
b"'"
&#39
b"'"


b'\x96'
&#150
b'\xe2\x80\x93'

0x78
0x92

b'x'
x
b'x'

b'\x92'

b'\xe2\x80\x99'

某些字符无法复制粘贴到编辑器,例如奇怪的 X 和奇怪的 ',因此我添加了一些代码来处理该问题。

我可以做什么来获取\xe2\x80\x9d 而不是\xc2\x94 for ” (”)?

我的设置:
Windows 7
终端:chcp 1252 + Lucida Console 字体
Python 3.3
美丽汤4

期待您的答复

最佳答案

HTML 中的数字字符引用指的是 Unicode 代码点,即,它不依赖于文档的字符编码,例如 ”U+0094 CANCEL CHARACTER*

b"\xe2\x80\x9d" 字节解释为 utf-8 为 U+201D RIGHT DOUBLE QUOTATION MARK :

u'\u201d'.encode('utf-8') == b'\xe2\x80\x9d'
u'\u201d'.encode('cp1252') == b'\x94'
u'\u201d'.encode('ascii', 'xmlcharrefreplace') == b'&#8221;'

要修复代码,请删除不必要的位:

from urllib.request import urlopen
from bs4 import BeautifulSoup

url = "http://www.sec.gov/path/to.htm"
soup = BeautifulSoup(urlopen(url))
print(soup)

如果失败;尝试 sys.stdout.buffer.write(soup.encode('cp1252')) 或将 PYTHONIOENCODING 环境变量设置为 cp1252:xmlcharrefreplace

关于encoding - 如何解码十进制的 cp1252 &#147 而不是\x93?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/17392422/

29 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com