Patchwork DMA: atmel_serial: Opening and closing the serial device repeatedly causes kmalloc-32 slab leak

login
register
mail settings
Submitter Richard Genoud
Date Nov. 27, 2018, 9:51 a.m.
Message ID <83753d21-f3c8-c8dd-75d7-741cb597d1a3@sorico.fr>
Download mbox | patch
Permalink /patch/665751/
State New
Headers show

Comments

Richard Genoud - Nov. 27, 2018, 9:51 a.m.
[re-sending the bug report to the lists]
Le 16/11/2018 à 17:04, Mario Forner a écrit :
> Problem:
> When I open and close the serial device /dev/ttyS4 in a loop
> the amount of kmalloc-32 slabs increases slowly but steadily without limit.
> 
> The serial device is configured in acme-aria.dts to use DMA.
> 
> If DMA is disabled, the amount of kmalloc32-slabs remains constant
> over several hours, just fluctuating slightly.
> 
> The serial device is accessed by atmel_serial.c which is evident from
> the drivers kernel log output.
> 
> The bug was noticed on a device which had been running over several weeks and
> has accumulated ~86MB of unrelaimable kmalloc-32 slabs by now. Example:
> 
> root@master1083:~# slabtop -o | head -10
>  Active / Total Objects (% used)    : 2704880 / 2716124 (99.6%)
>  Active / Total Slabs (% used)      : 23150 / 23150 (100.0%)
>  Active / Total Caches (% used)     : 57 / 76 (75.0%)
>  Active / Total Size (% used)       : 88893.62K / 89964.32K (98.8%)
>  Minimum / Average / Maximum Object : 0.02K / 0.03K / 4096.00K
> 
>   OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
> 2674804 2673840  99%    0.03K  21571      124     86284K kmalloc-32
>   7200   7104  98%    0.08K    144       50       576K kernfs_node_cache
>   6930   5370  77%    0.13K    231       30       924K dentry     
> 
> root@master1083:~# uptime
>  15:12:55 up 93 days, 16:54,  1 user,  load average: 1.59, 2.31, 2.42 
> 
> Keywords:  atmel, serial, kernel, leak, memory, dma, slab, kmalloc
> 
> Kernel information:
> 
> Kernel version:
> Linux version 4.2.6 (mario@linux-7rm0) (gcc version 4.9.3 (crosstool-NG ) ) #115 Fri Nov 16 11:05:22 CET 2018
> Linux version 4.9.124 (mario@linux-7rm0) (gcc version 4.9.3 (crosstool-NG ) ) #16 Fri Nov 16 13:54:25 CET 2018   
> 
> Most recent Kernel version which did not have the bug:  unknown         
> 
> How to reproduce the bug:
> 
> The kmalloc-32 slab count should be monitored every few minutes with "slabtop -o".
> The leak can be triggered by running the following script, given DMA has
> been enabled for ttyS4:
> 
> #!/usr/bin/env python
> import serial
> import time
> 
> while True:
>     try:
>         with serial.Serial(port = '/dev/ttyS4'):
>             pass
>     except Exception as e:
>         print e
>     finally:
>         time.sleep(0.5)
> # end script
> 
> Environment:
> Software:
> Linux master 4.2.6 #114 Fri Nov 16 10:14:30 CET 2018 armv5tejl GNU/Linux
> 
> Binutils                2.25
> Util-linux              2.25.2
> Mount                   2.25.2
> Module-init-tools       18
> E2fsprogs               1.42.12
> Linux C Library         2.19
> Dynamic linker (ldd)    2.19
> Linux C++ Library       6.0.20
> Procps                  3.3.9
> Net-tools               1.60
> Sh-utils                8.23
> Udev                    215
> Modules Loaded          iptable_nat option usb_wwan usbserial  
> 
> Processor information:
> # cat /proc/cpuinfo
> processor       : 0
> model name      : ARM926EJ-S rev 5 (v5l)
> BogoMIPS        : 198.76
> Features        : swp half thumb fastmult edsp java
> CPU implementer : 0x41
> CPU architecture: 5TEJ
> CPU variant     : 0x0
> CPU part        : 0x926
> CPU revision    : 5
> 
> Hardware        : Atmel AT91SAM9
> Revision        : 0000
> Serial          : 0000000000000000  
> 
> Module information:
> # cat /proc/modules
> iptable_nat 1720 0 - Live 0xbf0a5000
> option 28780 0 - Live 0xbf03c000
> usb_wwan 6876 1 option, Live 0xbf024000
> usbserial 23392 2 option,usb_wwan, Live 0xbf000000
> 
> Loaded drivers:
> # cat /proc/ioports
> # cat /proc/iomem
> 00300000-00307fff : 300000.sram
> 00600000-006fffff : /ahb/ohci@00600000
> 00700000-007fffff : /ahb/ehci@00700000
> 20000000-2fffffff : System RAM
>   20008000-20577287 : Kernel code
>   205a8000-205f02b7 : Kernel data
> f0000000-f00000ff : /ahb/apb/spi@f0000000
> f8008000-f80080ff : /ahb/apb/timer@f8008000
> f800c000-f800c0ff : /ahb/apb/timer@f800c000
> f8010000-f80100ff : /ahb/apb/i2c@f8010000
> f8014000-f80140ff : /ahb/apb/i2c@f8014000
> f801c000-f801c1ff : atmel_serial
> f8020000-f80201ff : atmel_serial
> f8024000-f80241ff : atmel_serial
> f8028000-f80281ff : atmel_serial
> f802c000-f802c0ff : /ahb/apb/ethernet@f802c000
> f8034000-f80342ff : /ahb/apb/pwm@f8034000
> f804c000-f804c0ff : /ahb/apb/adc@f804c000
> ffffec00-ffffedff : at_hdmac
> ffffee00-ffffefff : at_hdmac
> fffff200-fffff3ff : atmel_serial
> fffff400-fffff5ff : /ahb/apb/pinctrl@fffff400/gpio@fffff400
> fffff600-fffff7ff : /ahb/apb/pinctrl@fffff400/gpio@fffff600
> fffff800-fffff9ff : /ahb/apb/pinctrl@fffff400/gpio@fffff800
> fffffa00-fffffbff : /ahb/apb/pinctrl@fffff400/gpio@fffffa00
> fffffe10-fffffe1f : /ahb/apb/shdwc@fffffe10
> 
> Other Information:
> The serial device ttyS4 is configuered inside .dts file by
> 
> usart3: serial@f8028000 {
>     status = "okay";
>     compatible = "atmel,at91sam9260-usart";
>     reg = <0xf8028000 0x200>;
>     interrupts = <8 IRQ_TYPE_LEVEL_HIGH 5>;
>     pinctrl-names = "default";
>     atmel,use-dma-tx;
>     atmel,use-dma-rx;
>     pinctrl-0 = <&pinctrl_usart3
>     &pinctrl_usart3_rts~
>     &pinctrl_usart3_cts>;
>     linux,rs485-enabled-at-boot-time;
>     rs485-rts-delay = <0 0>;
> };
> 
> Other notes:
> I tried the SLUB allocator with slub_debug=U but opening serial device then
> resulted in a Kernel-Oops:
>     at BUGON(ents < 0);
>     in function: dma_map_sg_attrs
>     in file: /include/asm-generic/dma-mapping-common.h
> So I could not gather information that way. I don't know if this is related.
> SLUB works if User Tracking is disabled.
> 
> Enabling kmemleak did not detect any leaks.
> 
> I also tried kernel version 4.19.1 and it exhibits the same behaviour, but I
> could not configure it to run reliably on the target system yet. 
> 

Hi all,

I reproduced the memory leak on my board (at91sam9g35-cm) with a 4.20-rc3.

It triggered an OOM after a couple of hours running a code like this:
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>


int main(int argc, char **argv)
{
	int fd;
	do {
		fd = open("/dev/ttyS1", O_RDONLY);
		close(fd);
	} while (true);
	return 0;
}

As Mario pointed out, this only happens when atmel,use-dma-{r,t}x are
used in the device-tree.

Adding:
CONFIG_DEBUG_SLAB=y
CONFIG_DEBUG_SLAB_LEAK=y
Doesn't show anything suspect in /proc/slab_allocators

From what I found until now, it's something done in :
dma_request_slave_channel();
that leaks kmalloc-32
Mabe I missed something, but it seems that everything DMA related is
deallocated in atmel_release_{tx,rx}_dma().

Is this ringing a bell ?



The oom-killer, slabinfo, meminfo and slab_allocator traces follow.

NB: the -dirty flag is there beacause I removed some log from atmel-serial:



OOM-killer trace :
[ 5220.560000] test_open invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
[ 5220.580000] CPU: 0 PID: 1062 Comm: test_open Not tainted 4.20.0-rc3-dirty #31
[ 5220.580000] Hardware name: Atmel AT91SAM9
[ 5220.590000] Backtrace: 
[ 5220.590000] [<c000e020>] (dump_backtrace) from [<c000e2b8>] (show_stack+0x20/0x24)
[ 5220.600000]  r6:00000009 r5:c0649bb5 r4:c72a1e18 r3:02b88502
[ 5220.600000] [<c000e298>] (show_stack) from [<c0521114>] (dump_stack+0x20/0x28)
[ 5220.610000] [<c05210f4>] (dump_stack) from [<c0097280>] (dump_header+0x6c/0x1c4)
[ 5220.620000] [<c0097214>] (dump_header) from [<c0096454>] (oom_kill_process+0x6c/0x3cc)
[ 5220.620000]  r10:00000001 r8:c070e788 r7:c72a1e18 r6:00000009 r5:c0649bb5 r4:c7b64ea0
[ 5220.630000] [<c00963e8>] (oom_kill_process) from [<c009712c>] (out_of_memory+0x388/0x41c)
[ 5220.640000]  r10:00000001 r9:00000000 r8:c070e788 r7:0000000b r6:c070e8e4 r5:c070e788
[ 5220.650000]  r4:c72a1e18
[ 5220.650000] [<c0096da4>] (out_of_memory) from [<c009b1d8>] (__alloc_pages_nodemask+0x8b4/0xc48)
[ 5220.660000]  r8:00000011 r7:00000000 r6:00000000 r5:00000000 r4:006200ca
[ 5220.670000] [<c009a924>] (__alloc_pages_nodemask) from [<c0092bd4>] (filemap_fault+0x384/0x54c)
[ 5220.680000]  r10:c609b480 r9:00000000 r8:00000004 r7:00000000 r6:c72a1ed0 r5:c77dbb5c
[ 5220.680000]  r4:006200ca
[ 5220.690000] [<c0092850>] (filemap_fault) from [<c00babe8>] (__do_fault+0x28/0x9c)
[ 5220.690000]  r10:c7375a20 r9:c7375a58 r8:80000005 r7:000000b1 r6:b6efe000 r5:c72a1ed0
[ 5220.700000]  r4:00000000
[ 5220.700000] [<c00babc0>] (__do_fault) from [<c00bd9b0>] (handle_mm_fault+0x3e4/0x97c)
[ 5220.710000]  r5:000000c0 r4:00000000
[ 5220.720000] [<c00bd5cc>] (handle_mm_fault) from [<c000f774>] (do_page_fault+0x138/0x2ac)
[ 5220.720000]  r7:00000054 r6:b6efef9c r5:c72a1fb0 r4:c7a300e0
[ 5220.730000] [<c000f63c>] (do_page_fault) from [<c000f99c>] (do_translation_fault+0x2c/0xb4)
[ 5220.740000]  r10:b6f9e000 r9:00000000 r8:00053177 r7:80000005 r6:b6efef9c r5:c070f91c
[ 5220.750000]  r4:00000005
[ 5220.750000] [<c000f970>] (do_translation_fault) from [<c000fb38>] (do_PrefetchAbort+0x44/0x94)
[ 5220.760000]  r7:c72a1fb0 r6:b6efef9c r5:c070f91c r4:00000005
[ 5220.760000] [<c000faf4>] (do_PrefetchAbort) from [<c0009f08>] (ret_from_exception+0x0/0x18)
[ 5220.770000] Exception stack(0xc72a1fb0 to 0xc72a1ff8)
[ 5220.780000] 1fa0:                                     00000000 00000000 beb25d54 beb25e71
[ 5220.780000] 1fc0: 00000000 00000000 000082f4 00000006 00000000 00000000 b6f9e000 beb25bf4
[ 5220.790000] 1fe0: 00000000 beb25bdc 0000847c b6efef9c 40000010 ffffffff
[ 5220.800000]  r7:0005317f r6:ffffffff r5:40000010 r4:b6efef9c
[ 5220.800000] Mem-Info:
[ 5220.810000] active_anon:1659 inactive_anon:47 isolated_anon:0
[ 5220.810000]  active_file:2 inactive_file:4 isolated_file:0
[ 5220.810000]  unevictable:0 dirty:0 writeback:0 unstable:0
[ 5220.810000]  slab_reclaimable:1098 slab_unreclaimable:26200
[ 5220.810000]  mapped:3 shmem:60 pagetables:125 bounce:0
[ 5220.810000]  free:344 free_pcp:45 free_cma:0
[ 5220.840000] Node 0 active_anon:6636kB inactive_anon:188kB active_file:8kB inactive_file:16kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:12kB dirty:0kB writeback:0kB shmem:240kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[ 5220.860000] Normal free:1376kB min:1396kB low:1744kB high:2092kB active_anon:6636kB inactive_anon:188kB active_file:8kB inactive_file:16kB unevictable:0kB writepending:0kB present:131072kB managed:122316kB mlocked:0kB kernel_stack:448kB pagetables:500kB bounce:0kB free_pcp:180kB local_pcp:180kB free_cma:0kB
[ 5220.890000] lowmem_reserve[]: 0 0
[ 5220.890000] Normal: 0*4kB 54*8kB (U) 29*16kB (U) 11*32kB (U) 2*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1376kB
[ 5220.900000] 66 total pagecache pages
[ 5220.910000] 32768 pages RAM
[ 5220.910000] 0 pages HighMem/MovableOnly
[ 5220.910000] 2189 pages reserved
[ 5220.920000] Unreclaimable slab info:
[ 5220.920000] Name                      Used          Total
[ 5220.930000] ubi_wl_entry_slab         92KB         93KB
[ 5220.930000] bridge_fdb_cache           0KB          3KB
[ 5220.940000] fib6_nodes                 0KB          3KB
[ 5220.940000] ip6_dst_cache              0KB          3KB
[ 5220.950000] RAWv6                      4KB          7KB
[ 5220.950000] UDPv6                      1KB          7KB
[ 5220.960000] TCPv6                      3KB          6KB
[ 5220.960000] sd_ext_cdb                 0KB          3KB
[ 5220.970000] sgpool-128                 4KB          4KB
[ 5220.970000] sgpool-64                  2KB          4KB
[ 5220.980000] sgpool-32                  1KB          4KB
[ 5220.980000] sgpool-16                  0KB          4KB
[ 5220.990000] sgpool-8                   0KB          4KB
[ 5220.990000] nfs_commit_data            1KB          3KB
[ 5221.000000] nfs_write_data            18KB         19KB
[ 5221.000000] bio-1                      0KB          3KB
[ 5221.010000] rpc_buffers               16KB         16KB
[ 5221.010000] rpc_tasks                  1KB          4KB
[ 5221.020000] UNIX                      14KB         15KB
[ 5221.030000] tcp_bind_bucket            0KB          3KB
[ 5221.030000] ip_fib_trie                0KB          3KB
[ 5221.040000] ip_fib_alias               0KB          3KB
[ 5221.040000] ip_dst_cache               1KB          4KB
[ 5221.050000] RAW                        1KB          3KB
[ 5221.050000] UDP                        2KB          7KB
[ 5221.060000] TCP                        6KB          6KB
[ 5221.060000] eventpoll_pwq              0KB          3KB
[ 5221.070000] eventpoll_epi              1KB          3KB
[ 5221.070000] inotify_inode_mark          1KB          3KB
[ 5221.080000] request_queue             18KB         22KB
[ 5221.080000] blkdev_ioc                 0KB          3KB
[ 5221.090000] bio-0                      4KB          7KB
[ 5221.090000] biovec-max                90KB         90KB
[ 5221.100000] uid_cache                  0KB          3KB
[ 5221.100000] dmaengine-unmap-2          0KB          3KB
[ 5221.110000] skbuff_head_cache          5KB          7KB
[ 5221.110000] configfs_dir_cache          0KB          3KB
[ 5221.120000] file_lock_ctx              0KB          3KB
[ 5221.130000] fsnotify_mark_connector          0KB          3KB
[ 5221.130000] shmem_inode_cache        205KB        210KB
[ 5221.140000] proc_dir_entry            30KB         30KB
[ 5221.140000] pde_opener                 0KB          3KB
[ 5221.150000] seq_file                   0KB          3KB
[ 5221.150000] kernfs_node_cache        672KB        673KB
[ 5221.160000] mnt_cache                  5KB          7KB
[ 5221.160000] filp                      52KB         58KB
[ 5221.170000] names_cache               24KB         24KB
[ 5221.170000] key_jar                    0KB          4KB
[ 5221.180000] nsproxy                    0KB          3KB
[ 5221.180000] vm_area_struct            92KB        103KB
[ 5221.190000] mm_struct                 12KB         14KB
[ 5221.190000] fs_cache                   0KB          3KB
[ 5221.200000] files_cache                5KB         11KB
[ 5221.200000] signal_cache              33KB         46KB
[ 5221.210000] sighand_cache             71KB         80KB
[ 5221.210000] task_struct               57KB         86KB
[ 5221.220000] cred_jar                  13KB         20KB
[ 5221.220000] anon_vma_chain            32KB         46KB
[ 5221.230000] anon_vma                  29KB         39KB
[ 5221.240000] pid                        4KB          7KB
[ 5221.240000] debug_objects_cache         48KB         50KB
[ 5221.250000] trace_event_file          85KB         86KB
[ 5221.250000] ftrace_event_field        161KB        163KB
[ 5221.260000] pool_workqueue             1KB          4KB
[ 5221.260000] kmalloc-256k             256KB        256KB
[ 5221.270000] kmalloc-64k               64KB         64KB
[ 5221.270000] kmalloc-32k               32KB         32KB
[ 5221.280000] kmalloc-16k               32KB         32KB
[ 5221.280000] kmalloc-8k               128KB        128KB
[ 5221.290000] kmalloc-4k               104KB        104KB
[ 5221.290000] kmalloc-2k               102KB        108KB
[ 5221.300000] kmalloc-1k               191KB        192KB
[ 5221.300000] kmalloc-512              154KB        168KB
[ 5221.310000] kmalloc-256               64KB         80KB
[ 5221.310000] kmalloc-192               78KB         78KB
[ 5221.320000] kmalloc-128               22KB         24KB
[ 5221.320000] kmalloc-96                60KB         98KB
[ 5221.330000] kmalloc-32             97766KB      97770KB
[ 5221.340000] kmalloc-64               173KB        176KB
[ 5221.340000] kmem_cache                31KB         35KB
[ 5221.350000] Tasks state (memory values in pages):
[ 5221.350000] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[ 5221.360000] [    617]     0   617      458       30     8192        0             0 bootlogd
[ 5221.370000] [    653]     0   653      776       11     8192        0             0 syslogd
[ 5221.380000] [    656]     0   656      776       12     6144        0             0 klogd
[ 5221.380000] [    676]     0   676     1578      299    10240        0             0 conf-manager
[ 5221.390000] [    679]     0   679      484       27     6144        0             0 conf-dispatcher
[ 5221.400000] [    688]     0   688      432       17     8192        0             0 keypad
[ 5221.410000] [    700]     0   700      904       79     8192        0         -1000 udevd
[ 5221.420000] [    717]    81   717     1074       49     8192        0             0 dbus-daemon
[ 5221.430000] [    747]     0   747      776       11     8192        0             0 respawn.sh
[ 5221.440000] [    749]     0   749     1493       69    10240        0             0 connmand
[ 5221.440000] [    751]    99   751      581       27     6144        0             0 dnsmasq
[ 5221.450000] [    755]     0   755      572       21     6144        0             0 dropbear
[ 5221.460000] [    759]     0   759     1083       75    10240        0             0 lighttpd
[ 5221.470000] [    761]     0   761     4393      291    20480        0             0 php-cgi
[ 5221.480000] [    763]     0   763     4393      291    22528        0             0 php-cgi
[ 5221.490000] [    768]     0   768      777       39     6144        0             0 lns_networking.
[ 5221.500000] [    839]     0   839      455       23     6144        0             0 inotifywait
[ 5221.500000] [    897]     0   897      776        9     6144        0             0 ntpd
[ 5221.510000] [    905]     0   905     1404       22     8192        0             0 crond
[ 5221.520000] [    922]     0   922      484       21     8192        0             0 event-monitor
[ 5221.530000] [    941]     0   941     1011       29     8192        0             0 sh
[ 5221.540000] [   1062]     0  1062      383       11     6144        0             0 test_open
[ 5221.550000] [   1063]     0  1063      592       31     8192        0             0 dropbear
[ 5221.550000] [   1064]     0  1064     1011       30     8192        0             0 sh
[ 5221.560000] [   5315]     0  5315      776       15     6144        0             0 check_online.sh
[ 5221.570000] [   5316]     0  5316      776       15     4096        0             0 check_online.sh
[ 5221.580000] [   5317]     0  5317     1159       53    10240        0             0 socat
[ 5221.590000] [   5318]     0  5318     1016       37    10240        0             0 xsltproc
[ 5221.600000] [   5319]     0  5319      777        9     8192        0             0 uniq
[ 5221.610000] Out of memory: Kill process 676 (conf-manager) score 9 or sacrifice child
[ 5221.610000] Killed process 676 (conf-manager) total-vm:6312kB, anon-rss:1196kB, file-rss:0kB, shmem-rss:0kB


full stabinfo:
slabinfo - version: 2.1 (statistics)
# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail> : globalstat <listallocs> <maxobjs> <grown> <reaped> <error> <maxfreeable> <nodeallocs> <remotefrees> <alienoverflow> : cpustat <allochit> <allocmiss> <freehit> <freemiss>
ubi_wl_entry_slab   1964   1992     48   83    1 : tunables   32   16    0 : slabdata     24     24      0 : globalstat    1973   1973    24    0    0    0    0    0    0 : cpustat   1822    142      0      0
ubifs_inode_slab      66    189    432    9    1 : tunables   32   16    0 : slabdata     21     21      0 : globalstat   92965    477    53    0    0    0    0    0    0 : cpustat 354826   5836 355569   5041
bridge_fdb_cache       2     63     64   63    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      48     17     1    0    0    0    0    0    0 : cpustat      1      3      2      0
nf-frags               0      0    152   26    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
xfrm6_tunnel_spi       0      0     64   63    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
ip6-frags              0      0    152   26    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
fib6_nodes             6     63     64   63    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      32     17     1    0    0    0    0    0    0 : cpustat      4      2      0      0
ip6_dst_cache          3     25    160   25    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat     640     19     1    0    0    0    0    0    0 : cpustat     32     40     69      0
PINGv6                 0      0    832    9    2 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
RAWv6                  5      9    832    9    2 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      13      9     1    0    0    0    0    0    0 : cpustat      4      2      1      0
UDPLITEv6              0      0    832    9    2 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
UDPv6                  1      9    832    9    2 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      17      9     1    0    0    0    0    0    0 : cpustat      4      2      5      0
tw_sock_TCPv6          0      0    200   20    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
request_sock_TCPv6      0      0    256   16    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
TCPv6                  1      4   1760    4    2 : tunables   24   12    0 : slabdata      1      1      0 : globalstat       4      4     1    0    0    0    0    0    0 : cpustat      1      1      1      0
nf_conntrack_expect      0      0    152   26    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
nf_conntrack           0      0    192   21    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
sd_ext_cdb             2     71     56   71    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      16     16     1    0    0    0    0    0    0 : cpustat      1      1      0      0
sgpool-128             2      2   2048    2    1 : tunables   24   12    0 : slabdata      1      1      0 : globalstat       2      2     1    0    0    0    0    0    0 : cpustat      1      1      0      0
sgpool-64              2      4   1024    4    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat       4      4     1    0    0    0    0    0    0 : cpustat      1      1      0      0
sgpool-32              2      8    512    8    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat       8      8     1    0    0    0    0    0    0 : cpustat      1      1      0      0
sgpool-16              2     16    256   16    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      16     16     1    0    0    0    0    0    0 : cpustat      1      1      0      0
sgpool-8               2     32    128   32    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      16     16     1    0    0    0    0    0    0 : cpustat      1      1      0      0
nfs_direct_cache       0      0    208   19    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
nfs_commit_data        4      9    448    9    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat       9      9     1    0    0    0    0    0    0 : cpustat      3      1      0      0
nfs_write_data        32     35    576    7    1 : tunables   32   16    0 : slabdata      5      5      0 : globalstat      35     35     5    0    0    0    0    0    0 : cpustat     27      5      0      0
nfs_read_data          0      0    576    7    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
nfs_inode_cache        0      0    608    6    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
nfs_page               0      0     64   63    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
fat_inode_cache        0      0    424    9    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
fat_cache              0      0     48   83    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
jbd2_transaction_s      0      0    160   25    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
jbd2_inode             0      0     48   83    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
jbd2_journal_handle      0      0     64   63    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
jbd2_journal_head      0      0     88   46    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
jbd2_revoke_table_s      0      0     40   99    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
jbd2_revoke_record_s      0      0     32  124    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
ext4_inode_cache       0      0    624    6    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
ext4_free_data         0      0     64   63    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
ext4_allocation_context      0      0    128   32    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
ext4_prealloc_space      0      0     88   46    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
ext4_system_zone       0      0     56   71    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
ext4_io_end            0      0     72   56    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
ext4_pending_reservation      0      0     40   99    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
ext4_extent_status      0      0     56   71    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
mbcache                0      0     64   63    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kioctx                 0      0    224   18    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
aio_kiocb              0      0     96   42    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
fanotify_event_info      0      0     56   71    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
fsnotify_mark          0      0     64   63    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
dnotify_mark           0      0     72   56    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
dnotify_struct         0      0     48   83    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
dio                    0      0    360   11    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
bio-1                  4     25    160   25    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      16     16     1    0    0    0    0    0    0 : cpustat      3      1      0      0
fasync_cache           0      0     48   83    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
posix_timers_cache      0      0    192   21    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
rpc_inode_cache        0      0    352   11    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
rpc_buffers            8      8   2048    2    1 : tunables   24   12    0 : slabdata      4      4      0 : globalstat       8      8     4    0    0    0    0    0    0 : cpustat      4      4      0      0
rpc_tasks              8     32    128   32    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      16     16     1    0    0    0    0    0    0 : cpustat      7      1      0      0
UNIX                   7     28    576    7    1 : tunables   32   16    0 : slabdata      4      4      0 : globalstat   65532     35    88   84    0    0    0    0    0 : cpustat  20071   4224  24288      0
ip4-frags              0      0    160   25    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
UDP-Lite               0      0    704   11    2 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
tcp_bind_bucket        1     63     64   63    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      16     16     1    0    0    0    0    0    0 : cpustat      4      1      4      0
inet_peer_cache        0      0    128   32    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
secpath_cache          0      0     64   63    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
xfrm_dst_cache         0      0    224   18    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
xfrm_state             0      0    576    7    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
ip_fib_trie           12     71     56   71    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      32     19     1    0    0    0    0    0    0 : cpustat     11      2      1      0
ip_fib_alias          15     71     56   71    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      32     20     1    0    0    0    0    0    0 : cpustat     14      2      1      0
ip_dst_cache          10     32    128   32    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat    1824     24     1    0    0    0    0    0    0 : cpustat    116    114    222      0
PING                   0      0    672    6    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
RAW                    2      6    672    6    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      10      6     1    0    0    0    0    0    0 : cpustat     12      2     12      0
UDP                    0      0    704   11    2 : tunables   32   16    0 : slabdata      0      0      0 : globalstat     994     11     2    2    0    0    0    0    0 : cpustat    580    123    703      0
tw_sock_TCP            0      0    200   20    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
request_sock_TCP       0      0    256   16    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat      16     16     1    1    0    0    0    0    0 : cpustat      0      1      1      0
TCP                    1      4   1664    4    2 : tunables   24   12    0 : slabdata      1      1      0 : globalstat       5      4     1    0    0    0    0    0    0 : cpustat     11      2     12      0
eventpoll_pwq          5     63     64   63    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat    1392     48     1    0    0    0    0    0    0 : cpustat     61     87    142      1
eventpoll_epi          5     42     96   42    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat    1386     42     1    0    0    0    0    0    0 : cpustat     61     87    143      0
inotify_inode_mark     15     56     72   56    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      48     39     1    0    0    0    0    0    0 : cpustat     21      3      9      0
scsi_data_buffer       0      0     48   83    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
request_queue         12     15   1536    5    2 : tunables   24   12    0 : slabdata      3      3      0 : globalstat      15     15     3    0    0    0    0    0    0 : cpustat      9      3      0      0
blkdev_requests        0      0    216   18    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
blkdev_ioc             1     51     80   51    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      16     16     1    0    0    0    0    0    0 : cpustat      0      1      0      0
bio-0                 26     50    160   25    1 : tunables   32   16    0 : slabdata      2      2      0 : globalstat      41     41     2    0    0    0    0    0    0 : cpustat     27      3      4      0
biovec-max            30     30   3072    2    2 : tunables   24   12    0 : slabdata     15     15      0 : globalstat      30     30    15    0    0    0    0    0    0 : cpustat     15     15      0      0
biovec-128             0      0   1536    5    2 : tunables   24   12    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
biovec-64              0      0    768    5    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
biovec-16              0      0    192   21    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
uid_cache              0      0     96   42    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat      16     16     1    1    0    0    0    0    0 : cpustat      1      1      2      0
dmaengine-unmap-2      1    124     32  124    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      16     16     1    0    0    0    0    0    0 : cpustat      0      1      0      0
sock_inode_cache      26     66    352   11    1 : tunables   32   16    0 : slabdata      6      6      0 : globalstat   55763     77     7    0    0    0    0    0    0 : cpustat   7743   3489  11206      0
skbuff_fclone_cache      0      0    384   10    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat    5630     10   552  552    0    0    0    0    0 : cpustat    122    563    685      0
skbuff_head_cache     12     42    192   21    1 : tunables   32   16    0 : slabdata      2      2      0 : globalstat   56541    121     6    1    0    0    0    0    0 : cpustat  24027   3924  27932      9
configfs_dir_cache      1     51     80   51    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      16     16     1    0    0    0    0    0    0 : cpustat      0      1      0      0
file_lock_cache        0      0    144   28    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat    1696     18   103  103    0    0    0    0    0 : cpustat   1338    106   1444      0
file_lock_ctx          1     83     48   83    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      16     16     1    0    0    0    0    0    0 : cpustat      0      1      0      0
fsnotify_mark_connector     15     99     40   99    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      32     32     1    0    0    0    0    0    0 : cpustat     20      2      7      0
shmem_inode_cache    545    560    384   10    1 : tunables   32   16    0 : slabdata     56     56      0 : globalstat     573    560    56    0    0    0    0    0    0 : cpustat    533     58     46      0
proc_dir_entry       203    208    152   26    1 : tunables   32   16    0 : slabdata      8      8      0 : globalstat     232    208     8    0    0    0    0    0    0 : cpustat 1655734     19 1655550      0
pde_opener            16     83     48   83    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat    3392     16   208  207    0    0    0    0    0 : cpustat    379    212    590      0
proc_inode_cache      19     20    384   10    1 : tunables   32   16    0 : slabdata      2      2      0 : globalstat   65673    330    58   24    0    2    0    0    0 : cpustat  17629   9256  26850     29
seq_file              16     39    104   39    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat    3440     19     4    3    0    0    0    0    0 : cpustat   1822    215   2036      0
sigqueue               0      0     72   56    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat     496     25    28   28    0    0    0    0    0 : cpustat     22     31     53      0
bdev_cache            13     18    448    9    1 : tunables   32   16    0 : slabdata      2      2      0 : globalstat      26     18     2    0    0    0    0    0    0 : cpustat     14      3      4      0
kernfs_node_cache   6152   6156    112   36    1 : tunables   32   16    0 : slabdata    171    171      0 : globalstat    6164   6152   171    0    0    0    0    0    0 : cpustat   6046    512    406      0
mnt_cache             23     36    224   18    1 : tunables   32   16    0 : slabdata      2      2      0 : globalstat      34     34     2    0    0    0    0    0    0 : cpustat     20      3      0      0
filp                  94    325    160   25    1 : tunables   32   16    0 : slabdata     13     13      0 : globalstat   56563    366    16    1    0    1    0    0    0 : cpustat 2140672   3542 2141060   3073
inode_cache         7258   7260    344   11    1 : tunables   32   16    0 : slabdata    660    660      0 : globalstat    9668   8316   828   77    0    0    0    0    0 : cpustat   9994    928   3600     64
dentry              7885   8800    160   25    1 : tunables   32   16    0 : slabdata    352    352      0 : globalstat  209061   9941   398    0    0    1    0    0    0 : cpustat 579132  13240 572829  11669
names_cache            1      1   4096    1    1 : tunables   24   12    0 : slabdata      1      1      0 : globalstat    1216     11   965  964    0    0    0    0    0 : cpustat 2313916   1211 2315127      0
key_jar                3     32    128   32    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      16     16     1    0    0    0    0    0    0 : cpustat      2      1      0      0
buffer_head            0      0     88   46    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat      16     16     1    1    0    0    0    0    0 : cpustat      3      1      4      0
nsproxy                1     71     56   71    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      16     16     1    0    0    0    0    0    0 : cpustat      0      1      0      0
vm_area_struct       196    646    120   34    1 : tunables   32   16    0 : slabdata     19     19      0 : globalstat  493665   1326    41    1    0    1    0    0    0 : cpustat 1030662  30889 1031088  30293
mm_struct             25     36    416    9    1 : tunables   32   16    0 : slabdata      4      4      0 : globalstat   19979     45     5    1    0    0    0    0    0 : cpustat  44416   1297  45703      0
fs_cache              18    124     32  124    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat    1904    112     1    0    0    0    0    0    0 : cpustat  23929    119  24029      9
files_cache           18     63    192   21    1 : tunables   32   16    0 : slabdata      3      3      0 : globalstat    1913    105     5    0    0    0    0    0    0 : cpustat  23925    123  24028     10
signal_cache          45     72    608    6    1 : tunables   32   16    0 : slabdata     12     12      0 : globalstat   15230    120    20    1    0    0    0    0    0 : cpustat  23152    965  24070     10
sighand_cache         43     57   1312    3    1 : tunables   24   12    0 : slabdata     19     19      0 : globalstat    1401    117    48    8    0    6    0    0    0 : cpustat  23947    170  24066     14
task_struct           43     84   1056    7    2 : tunables   24   12    0 : slabdata     12     12      0 : globalstat   11531    119    18    0    0    1    0    0    0 : cpustat  23150    969  24064     18
cred_jar              70    160    128   32    1 : tunables   32   16    0 : slabdata      5      5      0 : globalstat   20529    144     5    0    0    0    0    0    0 : cpustat  56391   1284  57604     15
anon_vma_chain       161    781     56   71    1 : tunables   32   16    0 : slabdata     11     11      0 : globalstat  139671   1381    21    1    0    1    0    0    0 : cpustat 472282   8741 473000   7891
anon_vma             125    567     64   63    1 : tunables   32   16    0 : slabdata      9      9      0 : globalstat   68139    741    12    0    0    0    0    0    0 : cpustat 329676   4260 330198   3626
pid                   50    126     64   63    1 : tunables   32   16    0 : slabdata      2      2      0 : globalstat   15054    126     2    0    0    0    0    0    0 : cpustat  23178    941  24066     11
debug_objects_cache   1024   1079     48   83    1 : tunables   32   16    0 : slabdata     13     13      0 : globalstat    1028   1028    13    0    0    0    0    0    0 : cpustat    950     74      0      0
trace_event_file    1219   1232     72   56    1 : tunables   32   16    0 : slabdata     22     22      0 : globalstat    1219   1219    22    0    0    0    0    0    0 : cpustat      0   1219      0      0
ftrace_event_field   2953   2982     56   71    1 : tunables   32   16    0 : slabdata     42     42      0 : globalstat    2959   2959    42    0    0    0    0    0    0 : cpustat   2745    208      0      0
pool_workqueue         4     16    256   16    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      16     16     1    0    0    0    0    0    0 : cpustat      2      2      0      0
radix_tree_node      193    300    328   12    1 : tunables   32   16    0 : slabdata     25     25      0 : globalstat   60882    396    33    0    0    0    0    0    0 : cpustat 234553   3841 234990   3211
kmalloc-rcl-4M         0      0 4194304    1 1024 : tunables    1    1    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-rcl-2M         0      0 2097152    1  512 : tunables    1    1    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-rcl-1M         0      0 1048576    1  256 : tunables    1    1    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-rcl-512k       0      0 524288    1  128 : tunables    1    1    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-rcl-256k       0      0 262144    1   64 : tunables    1    1    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-rcl-128k       0      0 131072    1   32 : tunables    8    4    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-rcl-64k        0      0  65536    1   16 : tunables    8    4    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-rcl-32k        0      0  32768    1    8 : tunables    8    4    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-rcl-16k        0      0  16384    1    4 : tunables    8    4    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-rcl-8k         0      0   8192    1    2 : tunables    8    4    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-rcl-4k         0      0   4096    1    1 : tunables   24   12    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-rcl-2k         0      0   2048    2    1 : tunables   24   12    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-rcl-1k         0      0   1024    4    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-rcl-512        0      0    512    8    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-rcl-256        0      0    256   16    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-rcl-192        0      0    192   21    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-rcl-128        0      0    128   32    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-rcl-96         0      0     96   42    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-rcl-64         1     64     64   64    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      16     16     1    0    0    0    0    0    0 : cpustat      0      1      0      0
kmalloc-rcl-32         0      0     32  124    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-4M             0      0 4194304    1 1024 : tunables    1    1    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-2M             0      0 2097152    1  512 : tunables    1    1    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-1M             0      0 1048576    1  256 : tunables    1    1    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-512k           0      0 524288    1  128 : tunables    1    1    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-256k           1      1 262144    1   64 : tunables    1    1    0 : slabdata      1      1      0 : globalstat       1      1     1    0    0    0    0    0    0 : cpustat      0      1      0      0
kmalloc-128k           0      0 131072    1   32 : tunables    8    4    0 : slabdata      0      0      0 : globalstat       0      0     0    0    0    0    0    0    0 : cpustat      0      0      0      0
kmalloc-64k            1      1  65536    1   16 : tunables    8    4    0 : slabdata      1      1      0 : globalstat       1      1     1    0    0    0    0    0    0 : cpustat      0      1      0      0
kmalloc-32k            1      1  32768    1    8 : tunables    8    4    0 : slabdata      1      1      0 : globalstat       1      1     1    0    0    0    0    0    0 : cpustat      0      1      0      0
kmalloc-16k            2      2  16384    1    4 : tunables    8    4    0 : slabdata      2      2      0 : globalstat     222      4   212  210    0    0    0    0    0 : cpustat   7107    220   7325      0
kmalloc-8k             9      9   8192    1    2 : tunables    8    4    0 : slabdata      9      9      0 : globalstat     503     20   155  146    0    4    0    0    0 : cpustat 22043876    253 22044039     82
kmalloc-4k            27     27   4096    1    1 : tunables   24   12    0 : slabdata     27     27      0 : globalstat     504    135   399  255    0   25    0    0    0 : cpustat   3415    419   3789     18
kmalloc-2k            54     54   2048    2    1 : tunables   24   12    0 : slabdata     27     27      0 : globalstat    2022     68    36    9    0    0    0    0    0 : cpustat    858    667   1474      0
kmalloc-1k           151    172   1024    4    1 : tunables   32   16    0 : slabdata     43     43      0 : globalstat    1056    196    54   11    0    3    0    0    0 : cpustat    453    153    453      2
kmalloc-512          302    320    512    8    1 : tunables   32   16    0 : slabdata     40     40      0 : globalstat    7933    400    51    3    0    2    0    0    0 : cpustat 2220216    547 2220469      8
kmalloc-256          293    304    256   16    1 : tunables   32   16    0 : slabdata     19     19      0 : globalstat 12361098   1616 53263  284    0    2    0    0    0 : cpustat 19771823 772713 19771803 772442
kmalloc-192          410    420    192   21    1 : tunables   32   16    0 : slabdata     20     20      0 : globalstat    3763    417    20    0    0    0    0    0    0 : cpustat 20526116    416 20526138      0
kmalloc-128          288    288    128   32    1 : tunables   32   16    0 : slabdata      9      9      0 : globalstat   11689    288    13    4    0    0    0    0    0 : cpustat  21015   1676  22409      0
kmalloc-96           666    966     96   42    1 : tunables   32   16    0 : slabdata     23     23      0 : globalstat  138190   1241    30    0    0    0    0    0    0 : cpustat 264861   8696 269873   3026
kmalloc-32        3317614 3317620     32  124    1 : tunables   32   16    0 : slabdata  26755  26755      0 : globalstat 3329656 3317661 26779   17    0    1    0    0    0 : cpustat 11621275 215178 8518750    104
kmalloc-64          3094   3136     64   64    1 : tunables   32   16    0 : slabdata     49     49      0 : globalstat 1643930   3440 20618    0    0    1    0    0    0 : cpustat 46206626 102866 46203878 102535
kmem_cache           169    189    192   21    1 : tunables   32   16    0 : slabdata      9      9      0 : globalstat     184    184     9    0    0    0    0    0    0 : cpustat    114     57      2      0

# cat /proc/meminfo
MemTotal:         122316 kB
MemFree:            1804 kB
MemAvailable:       2468 kB
Buffers:               0 kB
Cached:              716 kB
SwapCached:            0 kB
Active:             1148 kB
Inactive:            404 kB
Active(anon):        932 kB
Inactive(anon):      188 kB
Active(file):        216 kB
Inactive(file):      216 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:           884 kB
Mapped:              472 kB
Shmem:               236 kB
KReclaimable:       4284 kB
Slab:             115048 kB
SReclaimable:       4284 kB
SUnreclaim:       110764 kB
KernelStack:         296 kB
PageTables:          136 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:       61156 kB
Committed_AS:       5640 kB
VmallocTotal:     901120 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:               64 kB

# cat /proc/slab_allocators 
ubi_wl_entry_slab: 3 erase_aeb+0x34/0xe0
ubi_wl_entry_slab: 805 ubi_wl_init+0x148/0x4e4
ubi_wl_entry_slab: 1155 ubi_wl_init+0x218/0x4e4
ubi_wl_entry_slab: 1 ubi_scan_fastmap+0x6f8/0x8fc
ubifs_inode_slab: 50 ubifs_alloc_inode+0x24/0x60
sd_ext_cdb: 2 mempool_alloc_slab+0x24/0x28
ip_fib_trie: 12 fib_insert_alias+0x40/0x298
ip_fib_alias: 15 fib_table_insert+0x33c/0x49c
eventpoll_pwq: 5 ep_ptable_queue_proc+0x38/0xb4
inotify_inode_mark: 15 sys_inotify_add_watch+0x12c/0x2bc
request_queue: 12 blk_alloc_queue_node+0x34/0x280
blkdev_ioc: 1 create_task_io_context+0x28/0xe0
configfs_dir_cache: 1 configfs_new_dirent+0x30/0xb4
file_lock_ctx: 1 locks_get_lock_context+0x48/0x10c
fsnotify_mark_connector: 15 fsnotify_add_mark_locked+0xbc/0x2b0
shmem_inode_cache: 545 shmem_alloc_inode+0x24/0x38
proc_dir_entry: 202 __proc_create+0x160/0x22c
proc_dir_entry: 1 proc_net_ns_init+0x28/0xd0
pde_opener: 1 proc_reg_open+0x7c/0x140
proc_inode_cache: 6 proc_alloc_inode+0x24/0x5c
seq_file: 1 seq_open+0x44/0xa4
kernfs_node_cache: 6152 __kernfs_new_node+0x48/0x174
inode_cache: 7258 alloc_inode+0x40/0xa8
dentry: 7872 __d_alloc+0x2c/0x1b4
nsproxy: 1 create_new_namespaces+0x34/0x144
vm_area_struct: 70 vm_area_alloc+0x28/0x68
vm_area_struct: 100 vm_area_dup+0x28/0x58
anon_vma_chain: 41 __anon_vma_prepare+0x28/0x124
anon_vma_chain: 51 anon_vma_clone+0x3c/0x168
anon_vma_chain: 40 anon_vma_fork+0x88/0x144
anon_vma: 72 __anon_vma_prepare+0x50/0x124
anon_vma: 40 anon_vma_fork+0x60/0x144
debug_objects_cache: 1024 debug_objects_mem_init+0x8c/0x248
trace_event_file: 1219 trace_create_new_event+0x24/0x6c
ftrace_event_field: 2953 __trace_define_field+0x30/0x98
radix_tree_node: 11 __radix_tree_preload.constprop.4+0x28/0x80
radix_tree_node: 125 radix_tree_node_alloc.constprop.5+0x54/0xd8
radix_tree_node: 11 radix_tree_node_alloc.constprop.5+0x8c/0xd8
radix_tree_node: 45 xas_alloc+0x40/0xb0
Richard Genoud - Nov. 27, 2018, 10:46 a.m.
Le 27/11/2018 à 10:58, Alexandre Belloni a écrit :
> Hello Richard,
> 
> On 27/11/2018 10:51:13+0100, richard.genoud@gmail.com wrote:
>> Hi all,
>>
>> I reproduced the memory leak on my board (at91sam9g35-cm) with a 4.20-rc3.
>>
>> It triggered an OOM after a couple of hours running a code like this:
>> #include <sys/types.h>
>> #include <sys/stat.h>
>> #include <fcntl.h>
>> #include <unistd.h>
>>
>>
>> int main(int argc, char **argv)
>> {
>> 	int fd;
>> 	do {
>> 		fd = open("/dev/ttyS1", O_RDONLY);
>> 		close(fd);
>> 	} while (true);
>> 	return 0;
>> }
>>
>> As Mario pointed out, this only happens when atmel,use-dma-{r,t}x are
>> used in the device-tree.
>>
>> Adding:
>> CONFIG_DEBUG_SLAB=y
>> CONFIG_DEBUG_SLAB_LEAK=y
>> Doesn't show anything suspect in /proc/slab_allocators
>>
>> From what I found until now, it's something done in :
>> dma_request_slave_channel();
>> that leaks kmalloc-32
>> Mabe I missed something, but it seems that everything DMA related is
>> deallocated in atmel_release_{tx,rx}_dma().
>>
>> Is this ringing a bell ?
>>
> 
> Yes, this is known issue and it has yet to be worked on.
> 

After a talk on freenode, Alex found the problem.
A patch is on its way.

Thanks !

Patch

--- a/drivers/tty/serial/atmel_serial.c
+++ b/drivers/tty/serial/atmel_serial.c
@@ -1056,8 +1056,6 @@  static int atmel_prepare_tx_dma(struct uart_port *port)
 	atmel_port->chan_tx = dma_request_slave_channel(mfd_dev, "tx");
 	if (atmel_port->chan_tx == NULL)
 		goto chan_err;
-	dev_info(port->dev, "using %s for tx DMA transfers\n",
-		dma_chan_name(atmel_port->chan_tx));
 
 	spin_lock_init(&atmel_port->lock_tx);
 	sg_init_table(&atmel_port->sg_tx, 1);
@@ -1239,8 +1237,6 @@  static int atmel_prepare_rx_dma(struct uart_port *port)
 	atmel_port->chan_rx = dma_request_slave_channel(mfd_dev, "rx");
 	if (atmel_port->chan_rx == NULL)
 		goto chan_err;
-	dev_info(port->dev, "using %s for rx DMA transfers\n",
-		dma_chan_name(atmel_port->chan_rx));
 
 	spin_lock_init(&atmel_port->lock_rx);
 	sg_init_table(&atmel_port->sg_rx, 1);