python - 调试(过早？)OOM-killer 输出

Question

我正在运行一个相当占用内存的 Python 脚本，但似乎我的机器正在提前终止进程。我安装了 16GB(并通过 lshw -class memory 确认)，但我的进程似乎在使用量达到 4GB 左右时被终止。我不确定如何读取 oom-killer 的输出以准确了解哪些内存耗尽以及原因。

dmesg 输出是:

[252291.326393] Xorg invoked oom-killer: gfp_mask=0x24040c0, order=3, oom_score_adj=0
[252291.326396] Xorg cpuset=/ mems_allowed=0
[252291.326400] CPU: 4 PID: 17915 Comm: Xorg Tainted: P           OE   4.4.0-59-generic #80-Ubuntu
[252291.326401] Hardware name: Dell Inc. Precision T3600/08HPGT, BIOS A07 11/08/2012
[252291.326403]  0000000000000286 0000000046b873b6 ffff880229dd38e8 ffffffff813f7583
[252291.326404]  ffff880229dd3ac0 ffff88000cd58e00 ffff880229dd3958 ffffffff8120ad5e
[252291.326406]  0000000000000015 0000000000000000 ffff8804347b8540 ffff88043c545400
[252291.326408] Call Trace:
[252291.326413]  [<ffffffff813f7583>] dump_stack+0x63/0x90
[252291.326416]  [<ffffffff8120ad5e>] dump_header+0x5a/0x1c5
[252291.326418]  [<ffffffff81390584>] ? apparmor_capable+0xc4/0x1b0
[252291.326421]  [<ffffffff81192722>] oom_kill_process+0x202/0x3c0
[252291.326422]  [<ffffffff81192b49>] out_of_memory+0x219/0x460
[252291.326424]  [<ffffffff81198abd>] __alloc_pages_slowpath.constprop.88+0x8fd/0xa70
[252291.326426]  [<ffffffff81198eb6>] __alloc_pages_nodemask+0x286/0x2a0
[252291.326429]  [<ffffffff811e271c>] alloc_pages_current+0x8c/0x110
[252291.326431]  [<ffffffff81196b49>] alloc_kmem_pages+0x19/0x90
[252291.326433]  [<ffffffff811b449e>] kmalloc_order_trace+0x2e/0xe0
[252291.326435]  [<ffffffff811ee49e>] __kmalloc+0x22e/0x250
[252291.326448]  [<ffffffffc193a3b7>] nvkms_alloc+0x27/0x60 [nvidia_modeset]
[252291.326454]  [<ffffffffc193b700>] ? _nv000313kms+0x40/0x40 [nvidia_modeset]
[252291.326463]  [<ffffffffc1974cda>] _nv001920kms+0x1a/0x30 [nvidia_modeset]
[252291.326471]  [<ffffffffc19610d2>] ? _nv001869kms+0x32/0xcf0 [nvidia_modeset]
[252291.326472]  [<ffffffff81196b49>] ? alloc_kmem_pages+0x19/0x90
[252291.326474]  [<ffffffff811b449e>] ? kmalloc_order_trace+0x2e/0xe0
[252291.326475]  [<ffffffff811ee49e>] ? __kmalloc+0x22e/0x250
[252291.326481]  [<ffffffffc193b700>] ? _nv000313kms+0x40/0x40 [nvidia_modeset]
[252291.326487]  [<ffffffffc193b72e>] ? _nv000167kms+0x2e/0x40 [nvidia_modeset]
[252291.326493]  [<ffffffffc193c111>] ? nvKmsIoctl+0x161/0x1e0 [nvidia_modeset]
[252291.326499]  [<ffffffffc193adb5>] ? nvkms_ioctl_common+0x45/0x80 [nvidia_modeset]
[252291.326505]  [<ffffffffc193ae61>] ? nvkms_ioctl+0x71/0xa0 [nvidia_modeset]
[252291.326597]  [<ffffffffc0050080>] ? nvidia_frontend_compat_ioctl+0x40/0x50 [nvidia]
[252291.326657]  [<ffffffffc005009e>] ? nvidia_frontend_unlocked_ioctl+0xe/0x10 [nvidia]
[252291.326659]  [<ffffffff8122272f>] ? do_vfs_ioctl+0x29f/0x490
[252291.326661]  [<ffffffff8108e5c1>] ? __set_task_blocked+0x41/0xa0
[252291.326663]  [<ffffffff81090f56>] ? __set_current_blocked+0x36/0x60
[252291.326664]  [<ffffffff81222999>] ? SyS_ioctl+0x79/0x90
[252291.326666]  [<ffffffff8109120e>] ? SyS_rt_sigprocmask+0x8e/0xc0
[252291.326668]  [<ffffffff818384f2>] ? entry_SYSCALL_64_fastpath+0x16/0x71
[252291.326669] Mem-Info:
[252291.326673] active_anon:976710 inactive_anon:321641 isolated_anon:0
                 active_file:429942 inactive_file:2132562 isolated_file:0
                 unevictable:8 dirty:112 writeback:0 unstable:0
                 slab_reclaimable:115928 slab_unreclaimable:19811
                 mapped:96472 shmem:14643 pagetables:11631 bounce:0
                 free:44761 free_pcp:188 free_cma:0
[252291.326675] Node 0 DMA free:15868kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15968kB managed:15884kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[252291.326679] lowmem_reserve[]: 0 2601 15909 15909 15909
[252291.326681] Node 0 DMA32 free:73192kB min:11040kB low:13800kB high:16560kB active_anon:439156kB inactive_anon:369572kB active_file:322044kB inactive_file:1378888kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:2791568kB managed:2710948kB mlocked:32kB dirty:60kB writeback:0kB mapped:67168kB shmem:11964kB slab_reclaimable:88956kB slab_unreclaimable:12792kB kernel_stack:2000kB pagetables:7696kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[252291.326685] lowmem_reserve[]: 0 0 13307 13307 13307
[252291.326687] Node 0 Normal free:89984kB min:56472kB low:70588kB high:84708kB active_anon:3467684kB inactive_anon:916992kB active_file:1397724kB inactive_file:7151360kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13893632kB managed:13626496kB mlocked:0kB dirty:388kB writeback:0kB mapped:318720kB shmem:46608kB slab_reclaimable:374756kB slab_unreclaimable:66436kB kernel_stack:10272kB pagetables:38828kB unstable:0kB bounce:0kB free_pcp:752kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[252291.326690] lowmem_reserve[]: 0 0 0 0 0
[252291.326703] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15868kB
[252291.326711] Node 0 DMA32: 392*4kB (UME) 9040*8kB (UM) 40*16kB (UM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 74528kB
[252291.326716] Node 0 Normal: 2254*4kB (UME) 651*8kB (UME) 4807*16kB (UM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 91136kB
[252291.326722] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[252291.326723] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[252291.326724] 2578955 total pagecache pages
[252291.326725] 1919 pages in swap cache
[252291.326726] Swap cache stats: add 621173, delete 619254, find 76600/102438
[252291.326726] Free swap  = 16626272kB
[252291.326727] Total swap = 16699388kB
[252291.326728] 4175292 pages RAM
[252291.326728] 0 pages HighMem/MovableOnly
[252291.326729] 86960 pages reserved
[252291.326730] 0 pages cma reserved
[252291.326730] 0 pages hwpoisoned
[252291.326731] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[252291.326735] [  826]     0   826    11000     2173      24       4      543             0 systemd-journal
[252291.326737] [  868]     0   868    10005      335      21       3      560         -1000 systemd-udevd
[252291.326739] [ 1594]     0  1594    45512    40563      94       3     1001             0 mount.ntfs
[252291.326740] [ 1627]     0  1627     5742      192      16       3       41             0 cron
[252291.326742] [ 1631]   106  1631    11015      752      26       4       63          -900 dbus-daemon
[252291.326744] [ 1637]   113  1637    90988      270      78       3      838             0 whoopsie
[252291.326745] [ 1643]     0  1643    84378      268      66       3      389             0 ModemManager
[252291.326746] [ 1645]     0  1645     5574      491      15       3       27             0 systemd-logind
[252291.326748] [ 1648]     0  1648   129217     1797      70       3      376             0 NetworkManager
[252291.326749] [ 1649]     0  1649    72390      942      31       5     1224             0 snapd
[252291.326750] [ 1671]   104  1671    64077      606      28       4      274             0 rsyslogd
[252291.326752] [ 1676]   115  1676    11237      467      27       3       87             0 avahi-daemon
[252291.326753] [ 1682]     0  1682    69002      855      37       3       71             0 accounts-daemon
[252291.326754] [ 1701]     0  1701     1100      156       7       3       30             0 acpid
[252291.326756] [ 1793]   115  1793    11171        7      23       3      201             0 avahi-daemon
[252291.326757] [ 1794]     0  1794    70117     1900      40       3      221             0 polkitd
[252291.326758] [ 1850]   109  1850    73809     1354      47       3      405             0 colord
[252291.326760] [ 1881]     0  1881   177783       52      43       4       60             0 nscd
[252291.326761] [ 1895]     0  1895     3522       84      12       3      211             0 dhclient
[252291.326763] [ 1899]     0  1899     4879      208      14       3       41             0 irqbalance
[252291.326764] [ 1911] 65534  1911    11708      289      28       3       70             0 dnsmasq
[252291.326765] [ 2305]     0  2305    36341     8787      70       3      181             0 puppet
[252291.326767] [ 2314]     0  2314    85649      340     165       3      668             0 smbd
[252291.326768] [ 2328]     0  2328    82522       23     151       3      590             0 smbd
[252291.326769] [ 2432]     0  2432     6032        1      15       3      228             0 dhclient
[252291.326771] [ 2575]     0  2575    15388        0      33       4      176         -1000 sshd
[252291.326772] [ 2579]     0  2579    39500    19242      79       3      167             0 vncserver-x11-c
[252291.326773] [ 2622]   108  2622    26535       46      22       3      228             0 ntpd
[252291.326774] [ 2680]     0  2680    85649       56     155       3      667             0 smbd
[252291.326776] [ 2847]     0  2847    14275       70      20       3      106             0 automount
[252291.326777] [ 2850]     0  2850    60183      691     114       3      385             0 nmbd
[252291.326779] [ 2904]     0  2904    82625       47      51       4      490             0 vpnagentd
[252291.326780] [ 3574]   121  3574    44304      107      22       3       51             0 rtkit-daemon
[252291.326781] [ 3598]     0  3598    86905      225      55       4      322             0 upowerd
[252291.326783] [ 3694] 521549  3694     9763      130      25       3      127             0 systemd
[252291.326784] [ 3701] 521549  3701    43047       37      46       3      613             0 (sd-pam)
[252291.326785] [ 4197]     0  4197    90424     1251      41       3      364             0 udisksd
[252291.326787] [ 4359]     0  4359   155602     6012      85       3      853          -900 fwupd
[252291.326788] [ 5776] 521549  5776   158062    23461     193       3      928             0 emacs24
[252291.326790] [ 5864] 521549  5864     7909        0      20       3      677             0 python
[252291.326791] [ 2697]     0  2697    27005       97      56       3      371             0 sshd
[252291.326792] [ 2747] 521549  2747    26996       93      55       3      336             0 sshd
[252291.326794] [ 2750] 521549  2750     7989       47      21       3      233             0 zsh
[252291.326795] [ 2763] 521549  2763     4225       84      13       3      548             0 bash
[252291.326796] [ 2897] 521549  2897     4431      496      14       3      480             0 bash
[252291.326797] [ 3229]     0  3229    27005       97      57       3      371             0 sshd
[252291.326799] [ 3256] 521549  3256    26990       31      56       3      332             0 sshd
[252291.326800] [ 3257] 521549  3257     6911        0      18       3      227             0 zsh
[252291.326801] [ 3261] 521549  3261     4197       84      14       3      517             0 bash
[252291.326803] [ 3271] 521549  3271     1048        0       8       3       18             0 emacsclient
[252291.326804] [15649]     0 15649    67156      442      63       3      255             0 cups-browsed
[252291.326806] [16854]     0 16854    23787     1732      54       4        0             0 login
[252291.326807] [16908] 521549 16908     7982     1100      20       4        0             0 zsh
[252291.326808] [16917] 521549 16917     7414     2894      18       3        0             0 bash
[252291.326810] [17023]     0 17023    23787     1755      53       4        0             0 login
[252291.326811] [17053] 521549 17053     7982     1114      20       4        0             0 zsh
[252291.326812] [17083]     0 17083    23794     1770      52       3        0             0 login
[252291.326814] [17105] 521549 17105     4426     1547      13       3        0             0 bash
[252291.326815] [17809] 521549 17809     7982     1095      20       3        0             0 zsh
[252291.326817] [17909]     0 17909    67586     1532      37       3        0             0 lightdm
[252291.326818] [17915]     0 17915   169029    73444     203       4        0             0 Xorg
[252291.326819] [17924]   125 17924     2693      390      11       3        0             0 nvidia-persiste
[252291.326821] [18006]     0 18006    62480     2315      58       3        0             0 lightdm
[252291.326822] [18180] 521549 18180    51387     1561      33       3        0             0 gnome-keyring-d
[252291.326823] [18182] 521549 18182    11569     1483      28       3        0             0 upstart
[252291.326825] [18268] 521549 18268     8288       70      19       3        0             0 upstart-udev-br
[252291.326826] [18276] 521549 18276    10863     1082      24       3        0             0 dbus-daemon
[252291.326827] [18288] 521549 18288    21659     2173      33       4        0             0 window-stack-br
[252291.326829] [18312] 521549 18312    41710      180      17       3        0             0 gpg-agent
[252291.326830] [18321] 521549 18321     8272       76      18       3        0             0 upstart-dbus-br
[252291.326832] [18322] 521549 18322     8272       76      18       3        0             0 upstart-dbus-br
[252291.326833] [18324] 521549 18324   103303     2284      37       3        0             0 ibus-daemon
[252291.326834] [18339] 521549 18339     8813       97      19       3        0             0 upstart-file-br
[252291.326836] [18350] 521549 18350    68686     2112      34       3        0             0 gvfsd
[252291.326837] [18359] 521549 18359   101715     1836      34       3        0             0 gvfsd-fuse
[252291.326838] [18363] 521549 18363    66138     2040      32       3        0             0 ibus-dconf
[252291.326840] [18364] 521549 18364   117631     7737     118       4        0             0 ibus-ui-gtk3
[252291.326841] [18375] 521549 18375   129093     6534     117       3        0             0 bamfdaemon
[252291.326842] [18380] 521549 18380   105261     5606     105       3        0             0 ibus-x11
[252291.326844] [18393] 521549 18393    84504     1401      33       3        0             0 at-spi-bus-laun
[252291.326845] [18398] 521549 18398    10613     1320      25       3        0             0 dbus-daemon
[252291.326846] [18410] 521549 18410    51744     1836      36       3        0             0 at-spi2-registr
[252291.326848] [18414] 521549 18414    47205     1323      28       3        0             0 ibus-engine-sim
[252291.326849] [18426] 521549 18426   176461    10773     163       4        0             0 hud-service
[252291.326850] [18428] 521549 18428   210855     8200     164       4        0             0 unity-settings-
[252291.326852] [18439] 521549 18439   156240     3651      98       3        0             0 gnome-session-b
[252291.326853] [18441] 521549 18441   159475     9762     130       4        0             0 unity-panel-ser
[252291.326854] [18477] 521549 18477    87239     1957      39       4        0             0 indicator-messa
[252291.326856] [18478] 521549 18478   101866     1150      33       3        0             0 indicator-bluet
[252291.326857] [18479] 521549 18479   123697     2493      42       4        0             0 indicator-power
[252291.326858] [18483] 521549 18483   306779     4524      94       4        0             0 indicator-datet
[252291.326860] [18486] 521549 18486   175316     7033     125       4        0             0 indicator-keybo
[252291.326861] [18490] 521549 18490   164979     2698      66       4        0             0 indicator-sound
[252291.326862] [18492] 521549 18492   133579     5799     126       4        0             0 indicator-print
[252291.326864] [18494] 521549 18494   238336     2196      49       4        0             0 indicator-sessi
[252291.326865] [18501] 521549 18501   117418     3077      90       3        0             0 indicator-appli
[252291.326866] [18537] 521549 18537   155428     5525     146       3        0             0 evolution-sourc
[252291.326868] [18561] 521549 18561   127303     2954      90       3        0             0 pulseaudio
[252291.326869] [18569] 521549 18569    44693     1256      23       3        0             0 dconf-service
[252291.326870] [18579] 521549 18579   212457    14611     189       3        0             0 evolution-calen
[252291.326872] [18581] 521549 18581   383682    73309     353       5        0             0 compiz
[252291.326873] [18615] 521549 18615   218897    12285     107       3        0             0 evolution-calen
[252291.326874] [18625] 521549 18625   171156     4450     113       3        0             0 evolution-addre
[252291.326876] [18627] 521549 18627   264112    12748     107       4        0             0 evolution-calen
[252291.326877] [18647] 521549 18647   224845     4211     100       4        0             0 evolution-addre
[252291.326878] [18661] 521549 18661   125158     1936      44       4        0             0 gvfs-udisks2-vo
[252291.326880] [18669] 521549 18669   100983     2268      50       3        0             0 gvfs-afc-volume
[252291.326881] [18675] 521549 18675    68033     1386      34       4        0             0 gvfs-gphoto2-vo
[252291.326882] [18696] 521549 18696    64984     1191      30       4        0             0 gvfs-mtp-volume
[252291.326884] [18701] 521549 18701    64464     1224      28       3        0             0 gvfs-goa-volume
[252291.326885] [18710] 521549 18710   281738    16416     182       4        0             0 nautilus
[252291.326886] [18717] 521549 18717    87724     1512      39       4        0             0 gvfsd-trash
[252291.326888] [18733] 521549 18733    46586     1818      27       3        0             0 gvfsd-metadata
[252291.326889] [18742] 521549 18742   325759    21904     224       4        0             0 gnome-software
[252291.326890] [18748] 521549 18748   105628     5367     104       4        0             0 polkit-gnome-au
[252291.326892] [18749] 521549 18749   142502     5189     111       4        0             0 unity-fallback-
[252291.326893] [18761] 521549 18761   145552     8397     137       3        0             0 nm-applet
[252291.326894] [18764] 521549 18764   132772     8236     126       4        0             0 ssa.py
[252291.326896] [18843] 521549 18843   144191     4502     107       3        0             0 zeitgeist-datah
[252291.326897] [18850] 521549 18850     1127      187       7       3        0             0 sh
[252291.326898] [18854] 521549 18854   102541     2249      36       3        0             0 zeitgeist-daemo
[252291.326899] [18861] 521549 18861    78154     3479      44       3        0             0 zeitgeist-fts
[252291.326901] [18934] 521549 18934   129225     7119     119       3        0             0 update-notifier
[252291.326902] [18987] 521549 18987   344436    69292     424       5        0             0 thunderbird
[252291.326904] [18998] 521549 18998    13824     1279      32       3        0             0 gconfd-2
[252291.326905] [19112] 521549 19112   107793     4884     112       3        0             0 notify-osd
[252291.326906] [19122] 521549 19122   108946     1738      47       3        0             0 deja-dup-monito
[252291.326908] [19217] 521549 19217    81658     2240      45       3        0             0 gvfsd-http
[252291.326909] [19222] 521549 19222   370048   103262     524       5        0             0 firefox
[252291.326910] [19315] 521549 19315   300942    79933     445       5        0             0 Web Content
[252291.326912] [19467]     0 19467    22146     2166      47       3        0             0 cupsd
[252291.326913] [19471]     7 19471    18762     1385      40       3        0             0 dbus
[252291.326914] [19472]     7 19472    18729     1363      41       3        0             0 dbus
[252291.326916] [19473]     7 19473    18762     1374      39       3        0             0 dbus
[252291.326917] [19474]     7 19474    18762     1321      40       3        0             0 dbus
[252291.326918] [19717]     0 19717    12234     1271      28       3        0             0 vncagent
[252291.326919] [19731] 521549 19731     1127      178       8       3        0             0 vncserverui
[252291.326921] [19754] 521549 19754    32985     3375      67       3        0             0 vncserverui.ori
[252291.326922] [19755] 521549 19755   113027     4323      85       3        0             0 vncserverui.ori
[252291.326923] [19778] 521549 19778   162444     9231     137       4        0             0 gnome-terminal-
[252291.326925] [19785] 521549 19785     8536     1125      21       3        0             0 zsh
[252291.326926] [19826] 521549 19826     8506     1138      21       3        0             0 zsh
[252291.326927] [19830] 521549 19830     4417     1572      14       3        0             0 bash
[252291.326929] [20319] 521549 20319   936657   797514    1629       6        0             0 python
[252291.326930] [20385] 521549 20385     7361      878      18       3        0             0 zsh
[252291.326931] Out of memory: Kill process 20319 (python) score 96 or sacrifice child
[252291.326935] Killed process 20319 (python) total-vm:3746628kB, anon-rss:3177076kB, file-rss:12980kB

Nvidia 列在调用跟踪中。我可能安装了不正确的驱动程序吗？

更一般地说，这里的各种类型的信息是什么意思？

提前感谢您的任何建议。

Answer 1

你的操作系统是什么？Ubuntu 16.04 中的最新内核存在 oom-killer 的错误。请参阅以下问题:https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842

您可能想要降级到 4.4.0-57-generic。