c++ - linux下可以分配大容量的虚拟内存吗？

Question

对于某些目的来说，分配大量虚拟空间并仅在访问的页面中进行分页会很有效。分配大量内存是瞬时的，并不实际抓取页面:

char* p = new char[1024*1024*1024*256];

好吧，上面指出的是错误的，因为它是一个 32 位数字。

我预计 new 会调用 malloc，而 malloc 会调用 sbrk，并且当我访问超出起始位置 4Gb 的位置时，它会尝试将任务内存扩展那么多？

这是完整的程序:

#include <cstdint>
int main() {
  constexpr uint64_t GB = 1ULL << 30;
  char* p = new char[256*GB]; // allocate large block of virtual space
  p[0] = 1;
  p[1000000000] = 1;
  p[2000000000] = 1;
}

现在，当我尝试分配大量内存时，我得到了 bad_alloc，所以显然 malloc 不起作用。

我的印象是 mmap 会映射到文件，但由于这是建议我正在研究它。

好吧，mmap 似乎支持分配大面积的虚拟内存，但它需要一个文件描述符。创建巨大的内存数据结构可能是一个胜利，但如果它们必须由文件支持则不然:

以下代码使用 mmap，尽管我不喜欢附加到文件的想法。我不知道要在虚拟内存中请求什么数字，所以选择了 0x800000000。 mmap 返回 -1，所以显然我做错了什么:

#include <cstdint>
#include <unistd.h>
#include <fcntl.h>
#include <sys/mman.h>

int main() {
  constexpr uint64_t GB = 1ULL << 30;
  void *addr = (void*)0x8000000000ULL;
  int fd = creat("garbagefile.dat", 0660);
  char* p = (char*)mmap(addr, 256*GB, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);
  p[0] = 1;
  p[1000000000] = 1;
  p[2000000000] = 1;
  close(fd);
}

有没有办法分配大块虚拟内存并稀疏地访问页面，或者这是不可行的？

Answer 1

Is it possible to allocate large amount of virtual memory in linux?

有可能。但您可能需要将其配置为允许:

The Linux kernel supports the following overcommit handling modes

0 - Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slightly more memory in this mode. This is the default.

1 - Always overcommit. Appropriate for some scientific applications. Classic example is code using sparse arrays and just relying on the virtual memory consisting almost entirely of zero pages.

2 - Don't overcommit. The total address space commit for the system is not permitted to exceed swap + a configurable amount (default is 50%) of physical RAM. Depending on the amount you use, in most situations this means a process will not be killed while accessing pages but will receive errors on memory allocation as appropriate.

Useful for applications that want to guarantee their memory allocations will be available in the future without having to initialize every page.

The overcommit policy is set via the sysctl `vm.overcommit_memory'.

因此，如果您想分配比物理内存更多的虚拟内存，那么您需要:

# in shell
sysctl -w vm.overcommit_memory=1

RLIMIT_AS The maximum size of the process's virtual memory (address space) in bytes. This limit affects calls to brk(2), mmap(2) and mremap(2), which fail with the error ENOMEM upon exceeding this limit. Also automatic stack expansion will fail (and generate a SIGSEGV that kills the process if no alternate stack has been made available via sigaltstack(2)). Since the value is a long, on machines with a 32-bit long either this limit is at most 2 GiB, or this resource is unlimited.

所以，你想要:

setrlimit(RLIMIT_AS, {
    .rlim_cur = RLIM_INFINITY,
    .rlim_max = RLIM_INFINITY,
});

或者，如果您无法授予进程执行此操作的权限，那么您可以在/etc/security/limits.conf 中永久配置此设置，这将影响(用户/组的)所有进程。

<小时/>

Ok, so mmap seems to support ... but it requires a file descriptor. ... could be a win but not if they have to be backed by a file ... I don't like the idea of attaching to a file

您不需要使用文件支持的 mmap。为此，有 MAP_ANONYMOUS。

I did not know what number to put in to request

然后使用 null。示例:

mmap(nullptr, 256*GB, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)

也就是说，如果您按照描述配置了系统，那么 new 应该和 mmap 一样工作。它可能会使用 malloc ，而对于像这样的大型分配，它可能会使用 mmap 。

<小时/>

额外提示:您可能会受益于使用 HugeTLB Pages .