python - 如何标记波斯语字符串并将其保存到 txt 文件中

Question

我正在尝试标记 A.txt 并将其保存到 B.txt 文件中我试图处理的字符串是波斯语，我想用波斯语逐字保存该字符串，这是我的代码

这是main.py

import LevelOne
import save_file
import nltk

original_data = " ".join(open("A.txt"))print('Processing')
save_file.saving(LevelOne.spliter(original_data))
print('Done')

这里是 LevelOne

import re 
import persian
import stop_word


def spliter(text):
    data = re.split(r'\W+',text)
    tokenized = [word for word in data if word not in 
    stop_word.stop_words]
    return tokenized

这是保存部分

# -*- coding: utf-8 -*-

def saving(infile):
    outfile = open('B.txt', 'w')
    replacements = {'پ':'\u067e',
          'چ':'\u0686','ج':'\u062c', 'ح':'\u062d','خ':'\u062e', 
          'ه':'\u0647','ع':'\u0639', 'غ':'\u063a','ف':'\u0641',                                           
          'ق':'\u0642','ث':'\u062b', 'ص':'\u0635','ض':'\u0636', 
          'گ':'\u06af','ک':'\u06a9', 'م':'\u0645','ن':'\u0646', 
          'ت':'\u062a','ا':'\u0627', 'ل':'\u0644','ب':'\u0628', 
          'ي':'\u06cc','س':'\u0633', 'ش':'\u0634','و':'\u0648', 
          'ئ':'\u0626','د':'\u062f', 'ذ':'\u0630','ر':'\u0631', 
          'ز':'\u0632','ط':'\u0637', 'ظ':'\u0638','ژ':'\u0698', 
          'آ':'\u0622','ی':'\u064a', '؟':'\u061f'}
    data = " ".join(infile)
    print(data)
    for line in data:
        for src, target in replacements.items() :
            line = line.replace(src, target)
            outfile.write(line)
    outfile.close()

但是当我打开 B.text 文件时，我看到了这个

Ú Ù Ù¾Ø³Ø Ø³Ù Ø Ù Ø ÙˆØ ÛŒ Ú Ù Ø Ø Ø ØŸ

原始文件如下所示

گل پسر
سلام خوبی چه خبر؟

Answer 1

您不需要替换 Unicode 字符。在Python3中，文件默认以utf-8保存。但是，为了标记句子和单词，由于 NLTK 不支持波斯语，您可能需要指定标点符号。

以下是不带标点符号的虚拟标记化示例:

tokenized = [
    line.strip().split(' ')
    for line in open("A.txt")
]

with open('B.txt', "w") as f:
    for line in tokenized:
        for word in line:
            f.write(word+"\n")
        f.write("\n")
    f.close()

A.txt:

گل پسر.
سلام، خوبی چه خبر؟

B.txt:

گل
پسر.

سلام،
خوبی
چه
خبر؟

但是如果您想根据标点符号进行标记，这是一个简单的解决方案(无需 NLTK 或任何额外的库):

import re

original_text = " ".join([
    line.strip()
    for line in open("A.txt")
])

def sent_tokenizer(text):
    # sentence separators: ؟ . ! ?
    seperators = "؟.!?"
    pattern = r"[^{0}]+[{0}]".format(seperators)
    return re.findall(pattern, text)

def word_tokenizer(sent):
    seperators = "؟!.?,:;، \t"
    pattern = r"([^{0}]+)([{0}])".format(seperators)

    words_sep = re.findall(pattern, sent)
    words = [
        w
        for word, sep in words_sep
        for w in [word, sep]
        if w != " "
    ]
    return words

tokenized = [
    word_tokenizer(sent)
    for sent in sent_tokenizer(original_text)
]

with open('B.txt', "w") as f:
    for line in tokenized:
        for word in line:
            f.write(word + "\n")
        f.write("\n")
    f.close()

B.txt:

گل
پسر
.

سلام
،
خوبی
چه
خبر
؟