python - 何时使用 re.compile

Question

请耐心等待，我不能包括我的 1,000 多行程序，并且描述中有几个问题。

所以我正在寻找几种类型的模式:

#literally just a regular word
re.search("Word", arg)

#Varying complex pattern
re.search("[0-9]{2,6}-[0-9]{2}-[0-9]{1}", arg)

#Words with varying cases and the possibility of ending special characters 
re.search("Supplier [Aa]ddress:?|Supplier [Ii]dentification:?|Supplier [Nn]ame:?", arg)

#I also use re.findall for the above patterns as well
re.findall("uses patterns above", arg

我总共有大约 75 个这样的函数，有些需要移动到深层嵌套的函数中

我应该何时何地编译模式？

现在我正在尝试通过编译 main 中的所有内容来改进我的程序，然后将已编译的 RegexObjects 的正确列表传递给使用它的函数。 这会提高我的表现吗？

像下面这样的操作会提高我的程序速度吗？

re.compile("pattern").search(arg)

编译后的模式是否保留在内存中，所以如果一个函数被调用多次，它会跳过编译部分吗？所以我不必将数据从一个函数移动到另一个函数。

如果我移动这么多数据，是否值得编译所有模式？

有没有更好的方法来匹配没有正则表达式的常规单词？

我的代码的简短示例:

import re

def foo(arg, allWords):
   #Does some things with arg, then puts the result into a variable, 
   # this function does not use allWords

   data = arg #This is the manipulated version of arg

   return(bar(data, allWords))


def bar(data, allWords):
   if allWords[0].search(data) != None:
      temp = data.split("word1", 1)[1]
      return(temp)

   elif allWords[1].search(data) != None:
      temp = data.split("word2", 1)[1]
      return(temp)


def main():

   allWords = [re.compile(m) for m in ["word1", "word2", "word3"]]

   arg = "This is a very long string from a text document input, the provided patterns might not be word1 in this string but I need to check for them, and if they are there do some cool things word3"

   #This loop runs a couple million times 
   # because it loops through a couple million text documents
   while True:
      data = foo(arg, allWords)

Answer 1

这是一个棘手的主题:许多答案，甚至一些合法来源，例如 David Beazley 的 Python Cookbook , 会告诉你类似这样的事情:

[Use compile()] when you’re going to perform a lot of matches using the same pattern. This lets you compile the regex only once versus at each match. [see p. 45 of that book]

但是，自从 Python 2.5 出现以来，情况并非如此。这是直接来自 re 文档的注释:

Note The compiled versions of the most recent patterns passed to re.compile() and the module-level matching functions are cached, so programs that use only a few regular expressions at a time needn’t worry about compiling regular expressions.

有两个小论点反对这一点，但(有趣的是)这些在大多数情况下不会导致明显的时间差异:

• 缓存的大小是有限的。
• 直接使用编译表达式可以避免缓存查找开销。

这是使用 20 newsgroups text dataset 对上述内容进行的初步测试.相对而言，编译速度提高了大约 1.6%，大概主要是由于缓存查找。

import re
from sklearn.datasets import fetch_20newsgroups

# A list of length ~20,000, paragraphs of text
news = fetch_20newsgroups(subset='all', random_state=444).data

# The tokenizer used by most text-processing vectorizers such as TF-IDF
regex = r'(?u)\b\w\w+\b'
regex_comp = re.compile(regex)


def no_compile():
    for text in news:
        re.findall(regex, text)


def with_compile():
    for text in news:
        regex_comp.findall(text)

%timeit -r 3 -n 5 no_compile()
1.78 s ± 16.2 ms per loop (mean ± std. dev. of 3 runs, 5 loops each)

%timeit -r 3 -n 5 with_compile()
1.75 s ± 12.2 ms per loop (mean ± std. dev. of 3 runs, 5 loops each)

这真的只剩下一个非常合理的理由来使用 re.compile():

By precompiling all expressions when the module is loaded, the compilation work is shifted to application start time, instead of to a point when the program may be responding to a user action. [source; p. 15]. It's not uncommon to see constants declared at the top of a module with compile. For example, in smtplib you'll find OLDSTYLE_AUTH = re.compile(r"auth=(.*)", re.I).

请注意，无论您是否使用 re.compile()，编译都会(最终)发生。当您使用 compile() 时，您正在编译传递的正则表达式。如果您使用像 re.search() 这样的模块级函数，您将在这一次调用中进行编译和搜索。以下两个过程在这方面是等效的:

# with re.compile - gets you a regular expression object (class)
#     and then call its method, `.search()`.
a = re.compile('regex[es|p]')  # compiling happens now
a.search('regexp')             # searching happens now

# with module-level function
re.search('regex[es|p]', 'regexp')  # compiling and searching both happen here

---

最后你问，

Is there a better way to match regular words without regex?

是的；这被称为 "common problem"在 HOWTO 中:

Sometimes using the re module is a mistake. If you’re matching a fixedstring, or a single character class, and you’re not using any refeatures such as the IGNORECASE flag, then the full power of regularexpressions may not be required. Strings have several methods forperforming operations with fixed strings and they’re usually muchfaster, because the implementation is a single small C loop that’sbeen optimized for the purpose, instead of the large, more generalizedregular expression engine. [emphasis added]

...

In short, before turning to the re module, consider whether yourproblem can be solved with a faster and simpler string method.