Java:读者和编码

Question

Java 的默认编码是ASCII。是的？ (请参阅下面我的编辑)

当文本文件以 UTF-8 编码时？读者如何知道他必须使用 UTF-8？

我所说的读者是:

• FileReader
• BufferedReader 来自 Socket
• 来自 System.in 的 Scanner
• ...

编辑

事实证明我们的编码取决于操作系统，这意味着以下情况并非适用于每个操作系统:

'a'== 97

Answer 1

How does a Reader know that he have to use UTF-8?

您通常在 InputStreamReader 中指定您自己 .它有一个采用字符编码的构造函数。例如

Reader reader = new InputStreamReader(new FileInputStream("c:/foo.txt"), "UTF-8");

所有其他阅读器(据我所知)使用平台默认字符编码，这本身可能确实不是正确的编码(例如 -cough- CP-1252 )。

理论上你也可以根据 byte order mark 自动检测字符编码.这将几种 unicode 编码与其他编码区分开来。不幸的是，Java SE 没有为此提供任何 API，但您可以自制一个可用于替换 InputStreamReader 的 API，如上例所示:

public class UnicodeReader extends Reader {
    private static final int BOM_SIZE = 4;
    private final InputStreamReader reader;

    /**
     * Construct UnicodeReader
     * @param in Input stream.
     * @param defaultEncoding Default encoding to be used if BOM is not found,
     * or <code>null</code> to use system default encoding.
     * @throws IOException If an I/O error occurs.
     */
    public UnicodeReader(InputStream in, String defaultEncoding) throws IOException {
        byte bom[] = new byte[BOM_SIZE];
        String encoding;
        int unread;
        PushbackInputStream pushbackStream = new PushbackInputStream(in, BOM_SIZE);
        int n = pushbackStream.read(bom, 0, bom.length);

        // Read ahead four bytes and check for BOM marks.
        if ((bom[0] == (byte) 0xEF) && (bom[1] == (byte) 0xBB) && (bom[2] == (byte) 0xBF)) {
            encoding = "UTF-8";
            unread = n - 3;
        } else if ((bom[0] == (byte) 0xFE) && (bom[1] == (byte) 0xFF)) {
            encoding = "UTF-16BE";
            unread = n - 2;
        } else if ((bom[0] == (byte) 0xFF) && (bom[1] == (byte) 0xFE)) {
            encoding = "UTF-16LE";
            unread = n - 2;
        } else if ((bom[0] == (byte) 0x00) && (bom[1] == (byte) 0x00) && (bom[2] == (byte) 0xFE) && (bom[3] == (byte) 0xFF)) {
            encoding = "UTF-32BE";
            unread = n - 4;
        } else if ((bom[0] == (byte) 0xFF) && (bom[1] == (byte) 0xFE) && (bom[2] == (byte) 0x00) && (bom[3] == (byte) 0x00)) {
            encoding = "UTF-32LE";
            unread = n - 4;
        } else {
            encoding = defaultEncoding;
            unread = n;
        }

        // Unread bytes if necessary and skip BOM marks.
        if (unread > 0) {
            pushbackStream.unread(bom, (n - unread), unread);
        } else if (unread < -1) {
            pushbackStream.unread(bom, 0, 0);
        }

        // Use given encoding.
        if (encoding == null) {
            reader = new InputStreamReader(pushbackStream);
        } else {
            reader = new InputStreamReader(pushbackStream, encoding);
        }
    }

    public String getEncoding() {
        return reader.getEncoding();
    }

    public int read(char[] cbuf, int off, int len) throws IOException {
        return reader.read(cbuf, off, len);
    }

    public void close() throws IOException {
        reader.close();
    }
}

编辑作为对您编辑的回复:

So the encoding is depends on the OS. So that means that not on every OS this is true:

'a'== 97

不，这不是真的。 ASCII编码(包含 128 个字符，0x00 直到 0x7F)是所有其他字符编码的基础。只有 ASCII 字符集之外的字符可能会在另一种编码中显示不同。 ISO-8859 encodings 覆盖 ASCII 范围内具有相同代码点的字符。 Unicode encodings 涵盖 ISO-8859-1 范围内具有相同代码点的字符。

您可能会发现这些博客中的每一个都很有趣:

1. The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) (两者的理论性更强)
2. Unicode - How to get the characters right? (两者比较实用)