python - 如何将带有变音符号 ɔ̃、ɛ̃ 和 ɑ̃ 的字符与 python 中的无重音符号(从 utf-8 编码的文本文件导入)进行比较？

Question

---

我写了一个脚本，它同时移动两个单词中的字符，比较它们以找到不同的字符对。单词长度相等(除了变音符号问题会引入一个额外的字符)，并表示仅相隔一个音素的两个法语单词的国际音标发音。

最终目标是过滤 anki 卡片列表，以便仅包含某些音素对，因为其他音素对太容易识别了。每对单词代表一个 anki 注释。

为此，我需要区分鼻音 ɔ̃、ɛ̃ 和 ɑ̃ 形成其他声音，因为它们只会真正与它们自己混淆。

如所写，代码将重音字符视为字符加 ~，以此类推作为两个字符。因此，如果一个单词的唯一区别是最后一个重音字符和重音字符之间的区别，脚本在最后一个字母上没有发现差异，并且按照所写的那样会找到一个比另一个短的单词(另一个仍然有 ~ 左)和尝试比较一个字符时抛出错误。这本身就是一个完整的“问题”，但如果我能让重音字符作为单个单元读取，那么单词将具有相同的长度，并且它会消失。

我不想像某些人那样用非重音字符代替重音字符进行比较，因为它们是不同的声音。

我已经尝试将 unicode ‘标准化’为‘组合’形式，例如unicodedata.normalize('NFKC', line)，但它没有改变任何东西。

---

这是一些输出，包括它刚刚抛出错误的行；打印输出显示代码正在比较的每个单词的单词和字符；该数字是单词中该字符的索引。因此，最后一个字母是脚本“认为”这两个字符的内容，它对 ɛ̃ 和 ɛ 的看法是一样的。当它报告差异时，它也会选择错误的字母对，这对是正确的很重要，因为我与允许对的主列表进行比较。

0 alyʁ alɔʁ a a # this first word is done well
1 alyʁ alɔʁ l l
2 alyʁ alɔʁ y ɔ # it doesn't continue to compare the ʁ because it found the difference
...
0 ɑ̃bisjø ɑ̃bisjɔ̃ ɑ ɑ
1 ɑ̃bisjø ɑ̃bisjɔ̃ ̃ ̃  # the tildes are compared / treated  separately
2 ɑ̃bisjø ɑ̃bisjɔ̃ b b
3 ɑ̃bisjø ɑ̃bisjɔ̃ i i
4 ɑ̃bisjø ɑ̃bisjɔ̃ s s
5 ɑ̃bisjø ɑ̃bisjɔ̃ j j
6 ɑ̃bisjø ɑ̃bisjɔ̃ ø ɔ # luckily that wasn't where the difference was, this is
...
0 osi ɛ̃si o ɛ # here it should report (o, ɛ̃), not (o, ɛ)
...
0 bɛ̃ bɔ̃ b b
1 bɛ̃ bɔ̃ ɛ ɔ # an error of this type
...
0 bo ba b b
1 bo ba o a # this is working correctly 
...
0 bjɛ bjɛ̃ b b
1 bjɛ bjɛ̃ j j
2 bjɛ bjɛ̃ ɛ ɛ # AND here's the money, it thinks these are the same letter, but it has also run out of characters to compare from the first word, so it throws the error below
Traceback (most recent call last):

  File "C:\Users\tchak\OneDrive\Desktop\French.py", line 42, in <module>
    letter1 = line[0][index]

IndexError: string index out of range

代码如下:

def lens(word):
    return len(word)

# open file, and new file to write to
input_file = "./phonetics_input.txt"
output_file = "./phonetics_output.txt"
set1 = ["e", "ɛ", "œ", "ø", "ə"]
set2 = ["ø", "o", "œ", "ɔ", "ə"]
set3 = ["ə", "i", "y"]
set4 = ["u", "y", "ə"]
set5 = ["ɑ̃", "ɔ̃", "ɛ̃", "ə"]
set6 = ["a", "ə"]
vowelsets = [set1, set2, set3, set4, set5, set6]
with open(input_file, encoding="utf8") as ipf, open(output_file, encoding="utf8") as opf:
    # for line in file; 
    vowelpairs= []
    acceptedvowelpairs = []
    input_lines = ipf.readlines()
    print(len(input_lines))
    for line in input_lines:
        #find word ipa transctipts
        unicodedata.normalize('NFKC', line)
        line = line.split("/")
        line.sort(key = lens)
        line = line[0:2] # the shortest two strings after splitting are the ipa words
        index = 0
        letter1 = line[0][index]
        letter2 = line[1][index]
        print(index, line[0], line[1], letter1, letter2)
            
        linelen = max(len(line[0]), len(line[1]))
        while letter1 == letter2:
            index += 1
            letter1 = line[0][index] # throws the error here, technically, after printing the last characters and incrementing the index one more
            letter2 = line[1][index]
            print(index, line[0], line[1], letter1, letter2)
            
        vowelpairs.append((letter1, letter2))   
        
    for i in vowelpairs:
        for vowelset in vowelsets:
            if set(i).issubset(vowelset):
                acceptedvowelpairs.append(i)
    print(len(vowelpairs))
    print(len(acceptedvowelpairs))

Answer 1

Unicode 规范化对描述的特定字符组合没有帮助，因为摘自 Unicode database UnicodeData.Txt使用简单的正则表达式 "Latin.*Letter.*with tilde$" 给出 ÃÑÕãñõĨĩŨũṼṽẼẽỸỹ(没有拉丁字母 Open O，Open E 或 Alpha)。因此，您需要按如下方式分别遍历两个比较的字符串(省略了上面 a Minimal, Reproducible Example 的大部分代码):

import unicodedata

def lens(word):
    return len(word)

input_lines = ['alyʁ/alɔʁ', 'ɑ̃bisjø/ɑ̃bisjɔ̃ ', 'osi/ɛ̃si', 'bɛ̃ /bɔ̃ ', 'bo/ba', 'bjɛ/bjɛ̃ ']
print(len(input_lines))
for line in input_lines:
    print('')
    #find word ipa transctipts
    line = unicodedata.normalize('NFKC', line.rstrip('\n'))
    line = line.split("/")
    line.sort(key = lens)
    word1, word2 = line[0:2] # the shortest two strings after splitting are the ipa words
    index = i1 = i2 = 0
    while i1 < len(word1) and i2 < len(word2):
        letter1 = word1[i1]
        i1 += 1
        if i1 < len(word1) and unicodedata.category(word1[i1]) == 'Mn':
            letter1 += word1[i1]
            i1 += 1
        letter2 = word2[i2]
        i2 += 1
        if i2 < len(word2) and unicodedata.category(word2[i2]) == 'Mn':
            letter2 += word2[i2]
            i2 += 1
        same = chr(0xA0) if letter1 == letter2 else '#' 
        print(index, same, word1, word2, letter1, letter2)
        index += 1
        #if same != chr(0xA0):
        #    break

输出:.\SO\67335977.py

6

0   alyʁ alɔʁ a a
1   alyʁ alɔʁ l l
2 # alyʁ alɔʁ y ɔ
3   alyʁ alɔʁ ʁ ʁ

0   ɑ̃bisjø ɑ̃bisjɔ̃  ɑ̃ ɑ̃
1   ɑ̃bisjø ɑ̃bisjɔ̃  b b
2   ɑ̃bisjø ɑ̃bisjɔ̃  i i
3   ɑ̃bisjø ɑ̃bisjɔ̃  s s
4   ɑ̃bisjø ɑ̃bisjɔ̃  j j
5 # ɑ̃bisjø ɑ̃bisjɔ̃  ø ɔ̃

0 # osi ɛ̃si o ɛ̃
1   osi ɛ̃si s s
2   osi ɛ̃si i i

0   bɛ̃  bɔ̃  b b
1 # bɛ̃  bɔ̃  ɛ̃ ɔ̃
2   bɛ̃  bɔ̃

0   bo ba b b
1 # bo ba o a

0   bjɛ bjɛ̃  b b
1   bjɛ bjɛ̃  j j
2 # bjɛ bjɛ̃  ɛ ɛ̃

注意:变音符号测试为 Unicode 类别 Mn；您可以针对另一个条件进行测试(例如来自以下列表):

• Mn Nonspacing_Mark:非间距组合标记(零提前宽度)
• Mc Spacing_Mark :一个间距组合标记(正向前进宽度)
• Me Enclosing_Mark : 一个封闭的组合标记
• M 标记: Mn |麦克 |我