linux - 为什么 Apache 不建议在 Linux 上将 sendfile() 与 NFS 一起使用

Question

Apache 文档包含 EnableSendfile 的这条语句:

With a network-mounted DocumentRoot (e.g., NFS, SMB, CIFS, FUSE), the kernel may be unable to serve the network file through its own cache.[1]

Apache 2.4 和 Nginx 的默认配置禁用 sendfile()。

我试图找到一些具体的东西来描述在 Linux 上将 sendfile() 与 NFS 文件系统一起使用时的确切问题。在内核 3.10.0-327.36.3 (CentOS 7) 上运行一个最小的测试程序验证当源在 NFS 上时 sendfile() 确实工作，并且它确实从页面缓存读取(第一次运行很慢，随后很快， drop_caches 使其再次变慢，即从源代码重新读取)。我尝试使用最大 1G 的文件，一切似乎都正常。我假设一定有一些情况会揭示错误行为，但我想确切地知道那是什么。

为了比较，有一些关于 VirtualBox 卷在 sendfile()[2] 中存在的问题的文档，但我找不到类似的内容涵盖 Apache，或者如何复制有问题的配置。

• [1] https://httpd.apache.org/docs/2.4/mod/core.html#enablesendfile
• [2] https://www.virtualbox.org/ticket/12597

Answer 1

Nginx 的默认配置打开 sendfile - https://github.com/nginx/nginx/blob/release-1.13.8/conf/nginx.conf#L27所以我对你在那里的陈述感到困惑。

回到 2000 年代初期，您可以看到 Apache dev introducing the option to disable SendFile (这里是 mailing list post for the patch )。还有old bugs that might have been related to sendfile over in the Apache bug tracker .来自 Apache bug #12893我们了解到，其中一个失败是因为 Linux 内核中的 NTFS 实现根本不支持 sendfile 系统调用:

[...] apparently there is some characteristic of your NTFS filesystem that prevents sendfile() from working.

sendfile(8, 9, [0], 9804)               = -1 EINVAL (Invalid argument)

A blog post titled "The Mysterious Case of Sendfile and Apache"引用您正在阅读的 stackoverflow 问题提出了以下理论:

sendfile() will transfer at most 0x7ffff000 (2,147,479,552) bytes, returning the number of bytes actually transferred. (This is true on both 32-bit and 64-bit systems.)

有 2GB 的限制。现在这里是假设，apache 文档说:

With a network-mounted DocumentRoot (e.g., NFS, SMB, CIFS, FUSE), the kernel may be unable to serve the network file through its own cache[2]

所以当它说“内核可能无法提供文件”时，我想我们可能指的是 sendfile 对文件大小的固有限制。

有趣的理论，但我怀疑这就是答案，因为您可以简单地选择不对太大的文件使用 sendfile 代码路径。 更新:在四处挖掘时，我发现那篇文章的作者创建了一个标题为 That Time I Was Wrong About Sendfile() and Apache 的后续文章。其中提到了您正在阅读的答案!

还有warnings about sendfile problems in the ProFTPD documentation :

There have been cases where it was the filesystems, rather than the kernels, which appeared to have been the culprits in sendfile(2) problems:

• Network filesystems (e.g NFS, SMBFS/Samba, CIFS)
• Virtualized filesystems (OpenVZ, VMware, and even Veritas)
• Other filesystems (e.g. NTFS and tmpfs on Linux)

Again, if you encounter issues with downloading files from ProFTPD when those files reside on a networked or virtualized filesystem, try using "UseSendfile off" in your proftpd.conf.

很多“这里有龙”的警告。其中一些是因为文件系统根本不支持 sendfile (例如 until 2.4.22-pre3 Linux's tmpfs didn't support sendfile )。基于 FUSE 的文件系统(例如 NTFS-3g)在过去也会因 FUSE 和 sendfile 错误(已解决)而出现问题。虚拟化文件系统列表是一个有趣的补充......

然而 OrangeFS FAQ似乎有最合理的解释:

5.16 Can we run the Apache webserver to serve files off a orangefs volume?

Sure you can! However, we recommend that you turn off the EnableSendfile option in httpd.conf before starting the web server. Alternatively, you could configure orangefs with the option -enable-kernel-sendfile. Passing this option to configure results in a orangefs kernel module that supports the sendfile callback. But we recommend that unless the files that are being served are large enough this may not be a good idea in terms of performance. Apache 2.x+ uses the sendfile system call that normally stages the file-data through the page-cache. On recent 2.6 kernels, this can be averted by providing a sendfile callback routine at the file-system. Consequently, this ensures that we don't end up with stale or inconsistent cached data on such kernels. However, on older 2.4 kernels the sendfile system call streams the data through the page-cache and thus there is a real possibility of the data being served stale. Therefore users of the sendfile system call are warned to be wary of this detail.

可以在 Linux guest readv system call returns stale (cached) shared folder file data Virtualbox bug 中阅读类似的解释。 :

I have discovered that programs that read files using the read system call return the correct data, but those using the readv system call (such as my version of gas) read stale cached data.

[...]

the use of kernel function generic_file_read_iter as the .read_iter member of the file_operations structure (.read_iter is used when doing a readv system call). This function WILL write to and read from the file cache. However, vbox function sf_reg_read, as used for the generic .read member and read system call, appears to always bypass Linux's FS cache.

[...]

Further I believe that a similar long-lived issue is reported as ticket #819, only for the sendfile system call. It seems that all of these generic_file_* functions have the expectation that the host controls all access to the drive.

以上也可以解释 ProFTPD 的问题虚拟化文件系统列表。

总结(最佳猜测)

Apache 建议不要将 sendfile() 与 Linux NFS 一起使用，因为他们的软件很受欢迎，并且会触发许多与旧版 Linux NFS 客户端调试 sendfile 相关的错误。警告是旧的，保持原样可能比使用所有警告更新它更容易。

如果您有一个 Linux 文件系统，可以在不使 Linux 页面缓存失效的情况下更改底层数据，那么在旧的 Linux 内核上使用 sendfile 是不明智的(这解释了旧的 Linux NFS 客户端问题)。对于较新的内核，如果上述文件系统未实现其自己的 sendfile Hook ，再次使用 sendfile 是不明智的(Virtualbox 共享文件夹问题证明了这一点)。

最近(2.6.31 及以上)的 Linux 内核为可能面临这种失效问题的文件系统提供了使用它们自己的 sendfile 实现的工具，并且假设文件系统可以使用 sendfile 禁止错误但买者自负!