python - 完全标记化句子，包括标点符号、缩略语和带连字符的单词

Question

我想把一句话完全标记化:“半衰期最长的元素是铀234”教授说。

我想要这个输出:

['"', 'The', 'element', 'with', 'the', 'longests', 'half-life', 'isn't', 'Uranium-234', '"', 'said', 'the', 'professor', '.']

这里所有的标点符号都是分开的，但是像“isn't”和“doesn't”这样的词是一个标记。带连字符的词也被视为一个标记，这就是我想要的。

目前我正在使用它来标记它:

p = re.compile(r"\w+(?:'\w+)?|[^\w\s]")
p.findall(s)

这给了我输出:

['"', 'The', 'element', 'with', 'the', 'longest', 'half', '-', 'life', 'isn't', 'Uranium', '-', '234', '"', 'said', 'the', 'professor', "."]

有了这个，我无法将带连字符的单词标记为一个标记。

Answer 1

使用 ['-] 字符类，你忘记了下划线:

\w+(?:['-]\w+)?|[^\w\s]|_

参见 proof .

解释

--------------------------------------------------------------------------------
  \w+                      word characters (a-z, A-Z, 0-9, _) (1 or
                           more times (matching the most amount
                           possible))
--------------------------------------------------------------------------------
  (?:                      group, but do not capture (optional
                           (matching the most amount possible)):
--------------------------------------------------------------------------------
    ['-]                     any character of: ''', '-'
--------------------------------------------------------------------------------
    \w+                      word characters (a-z, A-Z, 0-9, _) (1 or
                             more times (matching the most amount
                             possible))
--------------------------------------------------------------------------------
  )?                       end of grouping
--------------------------------------------------------------------------------
 |                        OR
--------------------------------------------------------------------------------
  [^\w\s]                  any character except: word characters (a-
                           z, A-Z, 0-9, _), whitespace (\n, \r, \t,
                           \f, and " ")
--------------------------------------------------------------------------------
 |                        OR
--------------------------------------------------------------------------------
  _                        '_'

Python code :

import re
regex = r"\w+(?:['-]\w+)?|[^\w\s]|_"
test_str = "\"The element with the longest half-life is Uranium-234\" said the professor."
print(re.findall(regex, test_str))

结果:['"', 'The', 'element', 'with', 'the', 'longest', 'half-life', 'is', '铀 234'、'"'、'说'、'那个'、'教授'、'.']