lisp - With-open-file 读取额外的字符

Question

我正在尝试将文件读入 Common Lisp 中的 string(而不是 list)，但是我最终在字符串。只有当文件包含换行符或制表符等字符时才会发生这种情况；空格似乎工作得很好。这是我的代码:

(defun load-file (filename)
  (with-open-file (stream filename 
                          :direction :input 
                          :if-does-not-exist :error)
    (let ((contents (make-string (file-length stream))))
      (read-sequence contents stream)
       contents)))

请注意:很遗憾，我不允许在此程序中使用循环或外部库。

Answer 1

这是一个老问题，答案是“不要那样做”。这样做的原因是 file-length 在许多有趣的情况下不能做你想做的事。特别是，file-length 的一个版本以您期望的方式工作，返回文件中的字符数，仅当以下一个或两个为真时才易于实现:

• 文件中的字符数是文件中字节数的某个固定倍数；
• 您使用的操作系统会为您记录文件中的字符数。

遗憾的是，对于我所知道的任何现代平台，这些都不是真的:

• 文件中的字符数不是其中字节数的固定倍数，原因至少有两个:

  • 行尾编码意味着文件可能在行尾包含两个字符 (#\Return #\Newline)，它们将被视为一个字符；
  • 文件可以使用不以任何简单方式将字节映射到字符的编码，例如 UTF-8，与行结束序列完全不同；
• 但操作系统只告诉您文件中的字节数。

对于像这样的平台，file-length 告诉你你想知道你正在读取的文件的内容的唯一方法是读取和解码整个文件文件，这显然是不可取的。实际上，file-length 仅告诉您文件的字节长度。

所以这种“计算出文件的长度并将其压缩成一大块”的技巧一般来说是行不通的，因为无法知道文件的长度以字符为单位没有阅读它。

有点烦人(我认为是 CL 的轻微缺陷)它不包含一个函数，其约定是“读取此文件并返回包含它的字符串”。

我相信，至少对于常见的编码，文件的字符长度永远不会长于字节长度。所以如果你愿意冒险一点，你可以做的一件事是分配一个数组，它是文件的字节长度，读取文件，然后记下你填充了多少数组(为了更聪明，使用可调阵列，读取后调整为合适的长度)。

---

请注意 Alexandria包含一个函数，read-file-into-string ，它可以满足您的需求，并且便携且速度可能很快。

---

这是一个相当简单的版本，我认为它适用于大多数情况(它根本不考虑字符串的元素类型):

(defun file->string (f &key (buffer-size 1024))
  (with-open-file (in f :direction :input)
    (with-output-to-string (out)
      (loop with buffer = (make-string buffer-size)
            for nchars = (read-sequence buffer in)
            do (write-sequence buffer out :start 0 :end nchars)
            while (= nchars buffer-size)))))

这是一个经过部分测试的、毛茸茸的函数，它试图变得更聪明，并处理文件的字节数比字符数短的情况(即使在正常的平台上，如果文件在读取时被附加到)。处理此问题的代码分支尚未经过测试:caveat emptor。

在大多数情况下，这也会减少数据的复制，但它返回的字符串通常会有一些浪费的空间。它假设填充指针很便宜(它们应该是)并且调整数组大小仅作为最后的手段才可以接受:因此当它需要使字符串更短时，它通过设置填充指针而不是调整它的大小来实现，只调整大小当它需要让它更长的时候。

它还温和地假设尾调用是优化的。

(defun file->string (f &key (element-type ':default)
                       (external-format ':default)
                       (growth-factor 0.1))
  "Read a file into a string, dealing with character encoding issues"
  ;; This attempts to be efficient: it allocates a string which, if
  ;; there are slightly fewer characters than bytes in the file (which
  ;; is the case for common encodings, will be a little too large,
  ;; then reads the file into it in one fell swoop, setting the
  ;; fill-pointer correctly after doing so if needed.  It also
  ;; attempts to deal with the case where the file is *shorter* in
  ;; bytes than it is in characters (this might be true if the file
  ;; was being appended to as the read is happening, or on some
  ;; platform which compresses files and reports the compressed
  ;; length), although this part of the code is untested.
  ;;
  ;; I am not sure if the use of LISTEN here is really right.
  ;;
  (with-open-file (in f :direction :input
                      :element-type element-type
                      :external-format external-format)
    (let* ((l (file-length in))
           (buf (make-array (list l)
                            :element-type (stream-element-type in)
                            :adjustable t :fill-pointer t))
           (n (read-sequence buf in)))
      (cond ((< n l)
             ;; Just make the array seem a bit shorter: this is the
             ;; common case for things like UTF-8 and DOS line endings
             (adjust-array buf (list l) :fill-pointer n))
            ((and (= n l) (not (listen in)))
             ;; We got the exact length of the string and the stream
             ;; is at EOF.  So the string is fine as is: this will be
             ;; true for traditional Unix encodings where a character
             ;; is a byte and line endings are a single character.
             buf)
            (t
             ;; This is unexpected: the file is longer in characters
             ;; than it is in bytes.  This code is UNTESTED since the
             ;; only case I can engineer for it involves a race
             ;; between something which is appending to the file and
             ;; this code, and that test is too hard to set up.
             (labels ((get-more (start chunk-size)
                        (let ((size (+ start chunk-size)))
                          (adjust-array buf (list size) :fill-pointer size)
                          (let ((n (read-sequence buf in :start start)))
                            (cond ((< n chunk-size)
                                   ;; we're done: set the fill pointer
                                   ;; right and return
                                   (adjust-array buf (list size)
                                                 :fill-pointer (+ start n)))
                                  ((and (= n chunk-size) (not (listen in)))
                                   ;; We're also done: we got the
                                   ;; exact number of characters we
                                   ;; had allocated fortuitously
                                   buf)
                                  (t
                                   ;; there is more to get
                                   (get-more (+ start chunk-size) chunk-size)))))))
               (get-more l (ceiling (* l growth-factor)))))))))