python - 获取文件的最后n行，类似于tail

Question

我正在为 Web 应用程序编写日志文件查看器，为此我想通过日志文件的行进行分页。文件中的项目是基于行的，最新的项目位于底部。

所以我需要一个 tail() 方法，它可以从底部读取 n 行并支持偏移量。这是我想出的帽子:

def tail(f, n, offset=0):
    """Reads a n lines from f with an offset of offset lines."""
    avg_line_length = 74
    to_read = n + offset
    while 1:
        try:
            f.seek(-(avg_line_length * to_read), 2)
        except IOError:
            # woops.  apparently file is smaller than what we want
            # to step back, go to the beginning instead
            f.seek(0)
        pos = f.tell()
        lines = f.read().splitlines()
        if len(lines) >= to_read or pos == 0:
            return lines[-to_read:offset and -offset or None]
        avg_line_length *= 1.3

这是一个合理的方法吗？使用偏移量尾随日志文件的推荐方法是什么？

Answer 1

这可能比你的更快。不对行长做任何假设。一次一个 block 地返回文件，直到找到正确数量的 '\n' 字符。

def tail( f, lines=20 ):
    total_lines_wanted = lines

    BLOCK_SIZE = 1024
    f.seek(0, 2)
    block_end_byte = f.tell()
    lines_to_go = total_lines_wanted
    block_number = -1
    blocks = [] # blocks of size BLOCK_SIZE, in reverse order starting
                # from the end of the file
    while lines_to_go > 0 and block_end_byte > 0:
        if (block_end_byte - BLOCK_SIZE > 0):
            # read the last block we haven't yet read
            f.seek(block_number*BLOCK_SIZE, 2)
            blocks.append(f.read(BLOCK_SIZE))
        else:
            # file too small, start from begining
            f.seek(0,0)
            # only read what was not read
            blocks.append(f.read(block_end_byte))
        lines_found = blocks[-1].count('\n')
        lines_to_go -= lines_found
        block_end_byte -= BLOCK_SIZE
        block_number -= 1
    all_read_text = ''.join(reversed(blocks))
    return '\n'.join(all_read_text.splitlines()[-total_lines_wanted:])

我不喜欢关于行长的棘手假设——实际上——你永远无法知道这样的事情。

通常，这将在循环的第一次或第二次通过时定位最后 20 行。如果您的 74 个字符的内容实际上是准确的，那么您将 block 大小设置为 2048，您将几乎立即拖尾 20 行。

此外，我不会消耗大量大脑卡路里来尝试与物理操作系统 block 对齐。使用这些高级 I/O 包，我怀疑您会看到尝试在 OS block 边界上对齐的任何性能后果。如果您使用较低级别的 I/O，那么您可能会看到加速。

---

更新

对于 Python 3.2 及更高版本，按照在文本文件中处理字节(那些在模式字符串中没有 "b" 的情况下打开)，仅相对于开头查找文件是允许的(异常(exception)是用 seek(0, 2) 搜索到文件末尾)。:

例如:f = open('C:/.../../apache_logs.txt', 'rb')

 def tail(f, lines=20):
    total_lines_wanted = lines

    BLOCK_SIZE = 1024
    f.seek(0, 2)
    block_end_byte = f.tell()
    lines_to_go = total_lines_wanted
    block_number = -1
    blocks = []
    while lines_to_go > 0 and block_end_byte > 0:
        if (block_end_byte - BLOCK_SIZE > 0):
            f.seek(block_number*BLOCK_SIZE, 2)
            blocks.append(f.read(BLOCK_SIZE))
        else:
            f.seek(0,0)
            blocks.append(f.read(block_end_byte))
        lines_found = blocks[-1].count(b'\n')
        lines_to_go -= lines_found
        block_end_byte -= BLOCK_SIZE
        block_number -= 1
    all_read_text = b''.join(reversed(blocks))
    return b'\n'.join(all_read_text.splitlines()[-total_lines_wanted:])