Perl UTF8 编码错误。 LWP::UserAgent->decoded_content 或 Encode::decode 都不起作用。其他想法？

Question

在尝试使用 LWP::Useragent 和 Encode 进行字符编码从网页中提取全局地址时，我在 perl 中遇到了编码问题。我试过谷歌搜索解决方案，但似乎没有任何效果。我正在使用 Strawberry Perl 5.12.3。

以美国驻捷克共和国大使馆的地址页面为例 (http://prague.usembassy.gov/contact.html)。我想要的只是撤回地址:

Address: Tržiště 15 118 01 Praha 1 - Malá Strana Czech Republic

哪个 firefox 使用与网页标题字符集相同的字符编码 UTF-8 正确显示。但是当我尝试使用 perl 将其拉回并将其写入文件时，尽管在 Useragent 或 Encode::decode 中使用了 decoded_content，但编码看起来很困惑。

我尝试在数据上使用正则表达式来检查错误不是在打印数据时(即在 perl 中内部正确)，但错误似乎在于 perl 处理编码的方式。

这是我的代码:

#!/usr/bin/perl

require Encode;
require LWP::UserAgent;
use utf8;

my $ua = LWP::UserAgent->new;
$ua->timeout(30);
$ua->env_proxy;

my $output_file;
$output_file = "C:/Documents and Settings/ian/Desktop/utf8test.txt";
open (OUTPUTFILE, ">$output_file") or die("Could not open output file $output_file: $!" );
binmode OUTPUTFILE, ":utf8";
binmode STDOUT, ":utf8";

# US embassy in Czech Republic webpage
$url = "http://prague.usembassy.gov/contact.html";

$ua_response = $ua->get($url);
if (!$ua_response->is_success) { die "Couldn't get data from $url";}

print 'CONTENT TYPE: '.$ua_response->content_charset."\n";
print OUTPUTFILE 'CONTENT TYPE: '.$ua_response->content_charset."\n";

my $content_not_decoded;
my $content_ua_decoded;
my $content_Endode_decoded;
my $content_double_decoded;

$ua_response->content =~ /<p><b>Address(.*?)<\/p>/;
$content_not_decoded = $1;
$ua_response->decoded_content =~ /<p><b>Address(.*?)<\/p>/;
$content_ua_decoded = $1;
Encode::decode_utf8($ua_response->content) =~ /<p><b>Address(.*?)<\/p>/;
$content_Endode_decoded = $1;
Encode::decode_utf8($ua_response->content) =~ /<p><b>Address(.*?)<\/p>/;
$content_double_decoded = $1;

# get the content without decoding
print 'UNDECODED CONTENT:'.$content_not_decoded."\n";
print OUTPUTFILE 'UNDECODED CONTENT:'.$content_not_decoded."\n";

# print the decoded content
print 'DECODED CONTENT:'.$content_ua_decoded."\n";
print OUTPUTFILE 'DECODED CONTENT:'.$content_ua_decoded."\n";

# use Encode to decode the content
print 'ENCODE::DECODED CONTENT:'.$content_Endode_decoded."\n";
print OUTPUTFILE 'ENCODE::DECODED CONTENT:'.$content_Endode_decoded."\n";

# try both!
print 'DOUBLE-DECODED CONTENT:'.$content_double_decoded."\n";
print OUTPUTFILE 'DOUBLE-DECODED CONTENT:'.$content_double_decoded."\n";

# check for #-digit character in the strings (to guard against the error coming in the print statement) 
if ($content_not_decoded =~ /\&/) {
    print "AMPERSAND FOUND IN UNDECODED CONTENT- LIKELY ENCODING ERROR\n";
    print OUTPUTFILE "AMPERSAND FOUND IN UNDECODED CONTENT- LIKELY ENCODING ERROR\n";
}
if ($content_ua_decoded =~ /\&/) {
    print "AMPERSAND FOUND IN DECODED CONTENT- LIKELY ENCODING ERROR\n"; 
    print OUTPUTFILE "AMPERSAND FOUND IN DECODED CONTENT- LIKELY ENCODING ERROR\n"; 
}
if ($content_Endode_decoded =~ /\&/) {
    print "AMPERSAND FOUND IN ENCODE::DECODED CONTENT- LIKELY ENCODING ERROR\n";
    print OUTPUTFILE "AMPERSAND FOUND IN ENCODE::DECODED CONTENT- LIKELY ENCODING ERROR\n";
}
if ($content_double_decoded =~ /\&/) {
    print "AMPERSAND FOUND IN DOUBLE-DECODED CONTENT- LIKELY ENCODING ERROR\n";
    print OUTPUTFILE "AMPERSAND FOUND IN DOUBLE-DECODED CONTENT- LIKELY ENCODING ERROR\n";
}

close (OUTPUTFILE);
exit;

这是终端的输出:

CONTENT TYPE: UTF-8 UNDECODED CONTENT::
Tr├à┬╛išt├ä┬¢ 15
118 01 Praha 1 - Malá Strana
Czech Republic DECODED CONTENT::
Tr┼╛išt─¢ 15
118 01 Praha 1 - Malá Strana
Czech Republic ENCODE::DECODED CONTENT::
Tr┼╛išt─¢ 15
118 01 Praha 1 - Malá Strana
Czech Republic DOUBLE-DECODED CONTENT::Tr┼╛išt─¢ 15
118 01 Praha 1 - Malá StranaCzech Republic AMPERSAND FOUND IN UNDECODED CONTENT- LIKELY ENCODING ERROR AMPERSAND FOUND IN DECODED CONTENT- LIKELY ENCODING ERROR AMPERSAND FOUND IN ENCODE::DECODED CONTENT- LIKELY ENCODING ERROR AMPERSAND FOUND IN DOUBLE-DECODED CONTENT- LIKELY ENCODING ERROR

和文件(注意这与终端略有不同，但不正确)。 OK WOW - 这在堆栈溢出中显示为正确，但在 Bluefish、LibreOffice、Excel、Word 或我计算机上的任何其他内容中显示为正确。所以数据只是错误编码。我真的不明白发生了什么。

CONTENT TYPE: UTF-8 UNDECODED CONTENT::
TržištÄ 15
118 01 Praha 1 - Malá Strana
Czech Republic DECODED CONTENT::
Tržiště 15
118 01 Praha 1 - Malá Strana
Czech Republic ENCODE::DECODED CONTENT::
Tržiště 15
118 01 Praha 1 - Malá Strana
Czech Republic DOUBLE-DECODED CONTENT::Tržiště 15
118 01 Praha 1 - Malá StranaCzech Republic AMPERSAND FOUND IN UNDECODED CONTENT- LIKELY ENCODING ERROR AMPERSAND FOUND IN DECODED CONTENT- LIKELY ENCODING ERROR AMPERSAND FOUND IN ENCODE::DECODED CONTENT- LIKELY ENCODING ERROR AMPERSAND FOUND IN DOUBLE-DECODED CONTENT- LIKELY ENCODING ERROR

任何指示如何做到这一点真的很感激。

谢谢，
伊恩/蒙特克里斯托

Answer 1

错误是使用正则表达式来解析 HTML。你缺decoding of HTML entities ，至少。您可以手动执行此操作，也可以将其留给强大的解析器:

use strictures;
use Web::Query 'wq';
use autodie qw(:all);

open my $output, '>:encoding(UTF-8)', '/tmp/embassy-prague.txt';
print {$output} wq('http://prague.usembassy.gov/contact.html')->find('p')->first->html; # or perhaps ->text