python - 如何批量循环遍历 python 列表？

Question

一个文件包含 10000 行，每行一个条目。我需要分批处理文件(小块)。

file = open("data.txt", "r")
data = file.readlines()
file.close()

total_count = len(data) # equals to ~10000 or less
max_batch = 50 # loop through 'data' with 50 entries at max in each loop.

for i in range(total_count):
     batch = data[i:i+50] # first 50 entries
     result = process_data(batch) # some time consuming processing on 50 entries
     if result == True:
           # add to DB that 50 entries are processed successfully!
     else:
           return 0 # quit the operation
           # later start again from the point it failed.
           # say 51st or 2560th or 9950th entry

这里要做什么才能使下一个循环从第 51 项到第 100 项中选择条目，依此类推？

如果操作以某种方式不成功并在中间中断，则只需要从失败的批处理(基于数据库条目)再次开始循环。

我无法编写正确的逻辑代码。我应该保留两个列表吗？还是别的？

Answer 1

l = [1,2,3,4,5,6,7,8,9,10]
batch_size = 3    

for i in range(0, len(l), batch_size):
    print(l[i:i+batch_size])
    # more logic here

>>> [1,2,3]
>>> [4,5,6]
>>> [7,8,9]
>>> [10}

我认为这是最直接、可读性最强的方法。如果您需要重试某个批处理，您可以在循环内重试(串行)或者您可以为每个批处理打开一个线程 - 取决于应用程序...