c++ - 如何同时使用推力和 valgrind 来检测内存泄漏？

Question

有没有办法将 CUDA 推力库与 Valgrind 内存泄漏检查器一起使用？
我问的原因是因为这个简单的程序:

#include <thrust/device_vector.h>

int main(){
    thrust::device_vector<int> D(5);
    assert( D.size() == 5 );
}

编译:

$ /usr/local/cuda-11.1/bin/nvcc device_vector.cu -o device_vector.cu.x

让 Valgrind 相信存在多种可能的内存泄漏。
我知道它们一定是误报，并且 valgrind 不是用来检测 GPU 内存泄漏的，但我想知道是否有标志或标准方法可以让这两种工具协同工作(例如检测 CPU 内存泄漏)。
如果周围有一套标准的 Valgrind 异常(exception)，我会很乐意使用它们，但我想在玩 wack-a-mole 之前问一下。

$ valgrind ./device_vector.cu.x 
==765561== Memcheck, a memory error detector
==765561== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==765561== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==765561== Command: ./device_vector.cu.x
==765561== 
==765561== Warning: noted but unhandled ioctl 0x30000001 with no size/direction hints.
==765561==    This could cause spurious value errors to appear.
==765561==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==765561== Warning: noted but unhandled ioctl 0x27 with no size/direction hints.
==765561==    This could cause spurious value errors to appear.
==765561==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==765561== Warning: noted but unhandled ioctl 0x25 with no size/direction hints.
==765561==    This could cause spurious value errors to appear.
==765561==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==765561== Warning: noted but unhandled ioctl 0x37 with no size/direction hints.
==765561==    This could cause spurious value errors to appear.
==765561==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==765561== Warning: noted but unhandled ioctl 0x17 with no size/direction hints.
==765561==    This could cause spurious value errors to appear.
==765561==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==765561== Warning: set address range perms: large range [0x200000000, 0x300200000) (noaccess)
==765561== Warning: set address range perms: large range [0x681f000, 0x2681e000) (noaccess)
==765561== Warning: noted but unhandled ioctl 0x19 with no size/direction hints.
==765561==    This could cause spurious value errors to appear.
==765561==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==765561== Warning: set address range perms: large range [0x10006000000, 0x10106000000) (noaccess)
==765561== Warning: noted but unhandled ioctl 0x49 with no size/direction hints.
==765561==    This could cause spurious value errors to appear.
==765561==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==765561== Warning: noted but unhandled ioctl 0x21 with no size/direction hints.
==765561==    This could cause spurious value errors to appear.
==765561==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==765561== Warning: noted but unhandled ioctl 0x1b with no size/direction hints.
==765561==    This could cause spurious value errors to appear.
==765561==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==765561== Warning: noted but unhandled ioctl 0x44 with no size/direction hints.
==765561==    This could cause spurious value errors to appear.
==765561==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==765561== 
==765561== HEAP SUMMARY:
==765561==     in use at exit: 6,678,624 bytes in 8,647 blocks
==765561==   total heap usage: 11,448 allocs, 2,801 frees, 40,718,174 bytes allocated
==765561== 
==765561== LEAK SUMMARY:
==765561==    definitely lost: 0 bytes in 0 blocks
==765561==    indirectly lost: 0 bytes in 0 blocks
==765561==      possibly lost: 22,216 bytes in 187 blocks
==765561==    still reachable: 6,656,408 bytes in 8,460 blocks
==765561==         suppressed: 0 bytes in 0 blocks
==765561== Rerun with --leak-check=full to see details of leaked memory
==765561== 
==765561== For lists of detected and suppressed errors, rerun with: -s
==765561== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

提到的自述文件 README_MISSING_SYSCALL_OR_IOCTL对我帮助不大。

已添加注释 : CUDA 带有一个名为 cuda-memcheck 的 memchecker它没有报告上面程序中的内存泄漏，但是它似乎不能替代 valgrind，因为它没有检测到简单 cpu 程序中的实际内存泄漏:

#include <thrust/device_vector.h>

int main(){
//  thrust::device_vector<int> D(5);
//  assert( D.size() == 5 );
    
//  cudaDeviceSynchronize();
    std::allocator<int> alloc;
    int* p = alloc.allocate(10);
    p[0] = 2;
    return p[0];
}

Answer 1

目前我正在使用这个抑制文件 .valgrind-supressions在我的项目的根:

{
   <suppression_for_thrust_allocations>
   Memcheck:Leak
   match-leak-kinds: possible
   fun:*alloc
   ...
   obj:*libcuda.so.*
   ...
   obj:*libcuda.so.*
   fun:__cudart*
   ...
   fun:__cudart*
   fun:cudaMalloc
   fun:_ZN6thrust6system4cuda6detail20cuda_memory_resourceIXadL_Z10cudaMallocEEXadL_Z8cudaFreeEENS_8cuda_cub7pointerIvEEE11do_allocateEmm
   ...
}

(三个点是实际代码)
通过删除 _ZN6thrust行也许它可以更笼统，但我不想过早地概括抑制。
重要的是要注意，这不是检查 GPU 中的泄漏，因为 cuda-memcheck需要。

更新:我将抑制范围扩大到 1) 包括从 cudaMallocManaged 生成的案例以及 2) 在没有推力分配器参与的情况下由 CUDA 运行时引起(如@RobertCrovella 所述)。

{
   <suppression_for_cudaMalloc_and_cudaMallocManaged_allocations>
   Memcheck:Leak
   match-leak-kinds: possible
   fun:*alloc
   ...
   obj:*libcuda.so.*
   ...
   obj:*libcuda.so.*
   fun:__cudart*
   ...
   fun:__cudart*
   fun:cudaMalloc*
   ...
}

在 CMakeLists.txt我正在使用这些选项来实际使用上面列出的抑制文件

  ...
set(MEMORYCHECK_COMMAND_OPTIONS "-q --tool=memcheck --leak-check=yes --num-callers=52 --trace-children=yes --leak-check=full --track-origins=yes --gen-suppressions=all") # must go before `include(CTest)`
set(MEMORYCHECK_SUPPRESSIONS_FILE "${PROJECT_SOURCE_DIR}/.valgrind-suppressions") # must go before `include(CTest)`

include(CTest)
  ...

(这里的三个点代表文件的其余部分)
欢迎对此抑制模式进行改进。

供引用 valgrind 完全自动生成的抑制看起来像这样:

{
   <insert_a_suppression_name_here>
   Memcheck:Leak
   match-leak-kinds: possible
   fun:calloc
   obj:/usr/lib/x86_64-linux-gnu/libcuda.so.470.63.01
   obj:/usr/lib/x86_64-linux-gnu/libcuda.so.470.63.01
   obj:/usr/lib/x86_64-linux-gnu/libcuda.so.470.63.01
   obj:/usr/lib/x86_64-linux-gnu/libcuda.so.470.63.01
   obj:/usr/lib/x86_64-linux-gnu/libcuda.so.470.63.01
   obj:/usr/lib/x86_64-linux-gnu/libcuda.so.470.63.01
   obj:/usr/lib/x86_64-linux-gnu/libcuda.so.470.63.01
   obj:/usr/lib/x86_64-linux-gnu/libcuda.so.470.63.01
   obj:/usr/lib/x86_64-linux-gnu/libcuda.so.470.63.01
   obj:/usr/lib/x86_64-linux-gnu/libcuda.so.470.63.01
   obj:/usr/lib/x86_64-linux-gnu/libcuda.so.470.63.01
   obj:/usr/lib/x86_64-linux-gnu/libcuda.so.470.63.01
   obj:/usr/lib/x86_64-linux-gnu/libcuda.so.470.63.01
   obj:/usr/lib/x86_64-linux-gnu/libcuda.so.470.63.01
   fun:__cudart764
   fun:__cudart763
   fun:__cudart768
   fun:__cudart941
   fun:__cudart607
   fun:cudaMalloc
   fun:_ZN6thrust6system4cuda6detail20cuda_memory_resourceIXadL_Z10cudaMallocEEXadL_Z8cudaFreeEENS_8cuda_cub7pointerIvEEE11do_allocateEmm
   fun:_ZN6thrust26device_ptr_memory_resourceINS_6system4cuda6detail20cuda_memory_resourceIXadL_Z10cudaMallocEEXadL_Z8cudaFreeEENS_8cuda_cub7pointerIvEEEEE11do_allocateEmm
   fun:_ZN6thrust2mr9allocatorIiNS_26device_ptr_memory_resourceINS_6system4cuda6detail20cuda_memory_resourceIXadL_Z10cudaMallocEEXadL_Z8cudaFreeEENS_8cuda_cub7pointerIvEEEEEEE8allocateEm
   fun:_ZZN6thrust6detail16allocator_traitsINS_16device_allocatorIiEEE8allocateERS3_mEN19workaround_warnings8allocateES5_m
   fun:_ZN6thrust6detail16allocator_traitsINS_16device_allocatorIiEEE8allocateERS3_m
   fun:_ZN6thrust6detail18contiguous_storageIiNS_16device_allocatorIiEEE8allocateEm
   fun:_ZN6thrust6detail11vector_baseIiNS_16device_allocatorIiEEE17allocate_and_copyINS0_15normal_iteratorIPKiEEEEvmT_SA_RNS0_18contiguous_storageIiS3_EE
   fun:_ZN6thrust6detail11vector_baseIiNS_16device_allocatorIiEEE10range_initINS0_15normal_iteratorIPKiEEEEvT_SA_NS_27random_access_traversal_tagE
   fun:_ZN6thrust6detail11vector_baseIiNS_16device_allocatorIiEEE10range_initINS0_15normal_iteratorIPKiEEEEvT_SA_
   fun:_ZN6thrust6detail11vector_baseIiNS_16device_allocatorIiEEEC1IiSaIiEEERKNS1_IT_T0_EE
   fun:_ZN6thrust13device_vectorIiNS_16device_allocatorIiEEEC1IiSaIiEEERKNS_11host_vectorIT_T0_EE
   fun:_ZN6vector11test_methodEv
   fun:_ZL14vector_invokerv
   fun:_ZN5boost6detail8function22void_function_invoker0IPFvvEvE6invokeERNS1_15function_bufferE
   obj:/usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.74.0
   fun:_ZN5boost17execution_monitor13catch_signalsERKNS_8functionIFivEEE
   fun:_ZN5boost17execution_monitor7executeERKNS_8functionIFivEEE
   fun:_ZN5boost17execution_monitor8vexecuteERKNS_8functionIFvvEEE
   fun:_ZN5boost9unit_test19unit_test_monitor_t21execute_and_translateERKNS_8functionIFvvEEEm
   obj:/usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.74.0
   obj:/usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.74.0
   fun:_ZN5boost9unit_test9framework3runEmb
   fun:_ZN5boost9unit_test14unit_test_mainEPFbvEiPPc
   fun:main
}