python - 在python中进行正则表达式匹配后获取 token 序列号

Question

我想在列表中找到与正则表达式匹配的所有元素。为了减少正则表达式匹配的次数，我通过连接以空格分隔的元素创建了一个字符串，如下所示:

list_a = ["4123", "7648", "afjsdn", "ujaf", "huh23"]
regex_num = r"\d+"
string_a = " ".join(list_a)
num_matches = re.findall(regex_num, string_a)

名单和比赛如下:

list_a:  ['4123', '7648', 'afjsdn', 'ujaf', 'huh23']
matches:  ['4123', '7648', '23']

现在我有了所有匹配项，我想知道匹配项是元素/标记的一部分还是整个标记。我可以做到这一点的一种方法是将匹配项与实际标记/元素进行比较:

"23" == "huh23"
False

但为此，我需要 token 序列号。哪个不能直接使用。正则表达式匹配唯一可以提供的位置信息是字符级别的匹配范围。

我可以采用的另一条路径是通过循环遍历列表并比较字符串与匹配项(如果存在匹配项)来对所有元素应用正则表达式匹配。

我想尽可能降低此操作的时间复杂度。

是否有更 pythonic 的方法来确定匹配项是否只是 token 的一部分，或者是否有更 pythonic 的方法来查找匹配单词的序列号，以便可以利用初始列表进行字符串比较？

如有任何帮助，我们将不胜感激。提前致谢!

编辑 1:

如果我的列表是这样的:

list_a = ["4123", "7648", "afjsdn", "ujaf", "huh23", "n23kl3l24"] 就像@Artyom Vancyan 在评论中建议的那样

我想要的输出是:

matches_with_slno = [[0,'4123'], [1,'7648'], [4, '23'], [5, '23'], [5,'3'], [5, '24']

Answer 1

使用 itertools 的函数式编程方法:

from itertools import chain, zip_longest

list_a = ["4123", "7648", "afjsdn", "ujaf", "huh23", "n23kl3l24"]

regex = r'\d+'
# pre-compilation of the regex
pattern = re.compile(regex)

# group the matches
grps = ((i, m) for i, m in enumerate(chain(map(pattern.findall, list_a))) if m)

# coupling the groups + flatting
out = list(chain.from_iterable(tuple(zip_longest((i,), matches, fillvalue=i)) for i, matches in grps))
print(out)
#[(0, '4123'), (1, '7648'), (4, '23'), (5, '23'), (5, '3'), (5, '24')]

---

性能测试

我试图做一个公平的测试:每个答案都有两个实现:

• v1:使用正则表达式预编译，pattern.findall
• v2:没有预编译，re.findall

测试的参数是重复次数(以获得统计上稳定的结果)和扩展数据列表大小的长度因子。

在所有情况下，预编译都会使代码的性能更高。列表理解似乎是更好的解决方案，然后是生成器，最后是 itertool。

1 位有效数字，对于小数据，它们的工作原理都是一样的。

使用 timeit 完成的时间测量.

from timeit import timeit
from itertools import chain, zip_longest
import re

def with_itertools_v1(lst):
    grps = ((i, m) for i, m in enumerate(chain(map(lambda x: re.findall(r'\d+', x), lst))) if m)
    return list(chain.from_iterable(list(zip_longest((i,), matches, fillvalue=i)) for i, matches in grps))
    
def with_itertools_v2(lst):
    pattern = re.compile(r'\d+')
    grps = ((i, m) for i, m in enumerate(chain(map(pattern.findall, lst))) if m)
    return list(chain.from_iterable(list(zip_longest((i,), matches, fillvalue=i)) for i, matches in grps))


def with_generators_v1(lst):
    def gen():
        for index, item in enumerate(lst):
            yield from [[index, match] for match in re.findall(r"\d+", item)]
    return list(gen())


def with_generators_v2(lst):
    def gen():
        pattern = re.compile(r'\d+')
        for index, item in enumerate(lst):
            yield from [[index, match] for match in pattern.findall(item)]
    return list(gen())

def with_list_comprehension_v1(lst):
    return [[index, match] for index, item in enumerate(lst) for match in re.findall(r"\d+", item)]


def with_list_comprehension_v2(lst):
    pattern = re.compile(r'\d+')
    return [[index, match] for index, item in enumerate(lst) for match in pattern.findall(item)]


funcs = (with_itertools_v1, with_itertools_v2, with_generators_v1, with_generators_v2, with_list_comprehension_v1, with_list_comprehension_v2)


def tester(repetition=5, length_factor=1):
    data = ["4123", "7648", "afjsdn", "ujaf", "huh23", "n23kl3l24"] * length_factor
    print(f'Test with repeated {repetition}-times with data with {len(data)*length_factor} terms')
    for func in funcs:
        t = timeit(lambda: func(data), number=repetition)
        print(func.__name__, f'ms: {t / repetition * 1e3:3f}')
    print()


tester(5, 1)
tester(5, 10)
tester(5, 100)
tester(5, 1000)

结果

Test with repeated 5-times with data with 6 terms
with_itertools_v1 ms: 0.060872
with_itertools_v2 ms: 0.019029
with_generators_v1 ms: 0.023321
with_generators_v2 ms: 0.015797
with_list_comprehension_v1 ms: 0.017898
with_list_comprehension_v2 ms: 0.010917

Test with repeated 5-times with data with 600 terms
with_itertools_v1 ms: 0.248608
with_itertools_v2 ms: 0.142064
with_generators_v1 ms: 0.201945
with_generators_v2 ms: 0.116305
with_list_comprehension_v1 ms: 0.160765
with_list_comprehension_v2 ms: 0.076942

Test with repeated 5-times with data with 60000 terms
with_itertools_v1 ms: 2.165053
with_itertools_v2 ms: 1.612787
with_generators_v1 ms: 1.484747
with_generators_v2 ms: 0.639824
with_list_comprehension_v1 ms: 11.219491
with_list_comprehension_v2 ms: 0.809310

Test with repeated 5-times with data with 6000000 terms
with_itertools_v1 ms: 46.531748
with_itertools_v2 ms: 35.922768
with_generators_v1 ms: 77.271880
with_generators_v2 ms: 20.042708
with_list_comprehension_v1 ms: 22.927475
with_list_comprehension_v2 ms: 16.760901