Parca agent is leaking memory and occasionally making the machine unresponsive until the kernel kills it

[r00ta]

Hey there,

Context

In our labs we are using the parca agent (snap) to monitor our MAAS instances.
We are running

parca-agent         v0.35.3                  2587   latest/edge    parca-team✓  classic

The problem

We had an outage of ~3h on a VM that was hosting a MAAS controller. After inspecting the logs we found out that the parca agent has been leaking memory for days until a point that caused an entire crash of the VM until the kernel decided to kill it after 3 hours.

Evidences

See the following graph for the memory usage over the last 3 months

If I take a more close look at the last jump in the memory usage, we can clearly see a log statement that the parca-agent was killed and restarted

(logs timestamps are in UTC, timestamps in the graphs are UTC+1)

2025-03-07 18:56:04	
Mar  7 18:56:04 maas-tor3-1 systemd[1]: snap.parca-agent.parca-agent-svc.service: Killing process 3164211 (parca-agent) with signal SIGKILL.
2025-03-07 18:56:04	
Mar  7 18:56:04 maas-tor3-1 systemd[1]: snap.parca-agent.parca-agent-svc.service: Killing process 3133650 (parca-agent) with signal SIGKILL.
2025-03-07 18:56:04	
Mar  7 18:56:04 maas-tor3-1 systemd[1]: snap.parca-agent.parca-agent-svc.service: State 'final-sigterm' timed out. Killing.

As additional proof, over the last days I've extracted the memory usage of the parca-agent

11-03-2025: root 3418267 2.0 5.2 4499096 0.81 GB ? Sl Mar07 109:11 /snap/parca-agent/2587/parca-agent
13-03-2025 root 3418267 2.1 6.9 4899868 1.08 GB ? Sl Mar07 174:44 /snap/parca-agent/2587/parca-agent
14-03-2025: root 3418267 2.2 7.6 5165996 1.19 GB ? Sl Mar07 221:35 /snap/parca-agent/2587/parca-agent
17-03-2025: root 3418267 2.2 9.3 5167148 1.46 GB ? Sl Mar07 310:59 /snap/parca-agent/2587/parca-agent

The outage

As I mentioned in the initial section, it seems that the parca agent took down our VM for about 3 hours (in the first grafana graph you can see a hole on 17th February, because the VM was completely unresponsive and prometheus could not scrape any data). Still, with a combination of the grafana graphs and journal we managed to understand what happened. Here the last data scraped by prometheus

and in the journal we have track of the kernel killing the parca agent and all the other processes get back into the business.

Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004358] systemd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004367] CPU: 0 PID: 1 Comm: systemd Tainted: G        W         5.15.0-130-generic #140-Ubuntu
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004370] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009)/LXD, BIOS unknown 2/2/2022
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004372] Call Trace:
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004373]  <TASK>
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004375]  show_stack+0x52/0x5c
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004382]  dump_stack_lvl+0x4a/0x63
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004386]  dump_stack+0x10/0x16
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004388]  dump_header+0x53/0x228
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004391]  oom_kill_process.cold+0xb/0x10
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004394]  out_of_memory+0x106/0x2e0
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004398]  __alloc_pages_slowpath.constprop.0+0x9a0/0xac0
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004403]  __alloc_pages+0x311/0x330
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004406]  alloc_pages+0x9e/0x1e0
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004409]  __page_cache_alloc+0x7e/0x90
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004412]  pagecache_get_page+0x152/0x590
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004414]  ? srso_alias_return_thunk+0x5/0x7f
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004418]  ? page_cache_ra_unbounded+0x163/0x210
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004422]  filemap_fault+0x488/0xab0
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004423]  ? srso_alias_return_thunk+0x5/0x7f
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004425]  ? filemap_map_pages+0x309/0x400
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004428]  __do_fault+0x39/0x120
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004431]  do_read_fault+0xeb/0x160
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004433]  do_fault+0xa0/0x2e0
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004435]  handle_pte_fault+0x1cd/0x240
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004437]  __handle_mm_fault+0x405/0x6f0
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004441]  handle_mm_fault+0xd8/0x2c0
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004443]  do_user_addr_fault+0x1c9/0x640
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004448]  exc_page_fault+0x77/0x170
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004451]  asm_exc_page_fault+0x27/0x30
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004455] RIP: 0033:0x7fb4c74a3cc9
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004464] Code: Unable to access opcode bytes at RIP 0x7fb4c74a3c9f.
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004465] RSP: 002b:00007ffe54c91a10 EFLAGS: 00010213
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004468] RAX: 678898a0ac9b1aa3 RBX: 000055586980af2d RCX: 000055586980af2d
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004469] RDX: 00007fb4c758e8c0 RSI: 0000000000000000 RDI: 007465677261742e
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004471] RBP: 00007ffe54c91a40 R08: b1d73852e218d169 R09: 3fff32cc8ecc6561
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004472] R10: c4e291cc8f447043 R11: 00007fb4c7608820 R12: 000055586980af2d
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004473] R13: 0000000000000010 R14: 000055588a80c5f0 R15: 000055588a73ff20
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004477]  </TASK>
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004478] Mem-Info:
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004483] active_anon:29322 inactive_anon:3797599 isolated_anon:0
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004483]  active_file:196 inactive_file:4022 isolated_file:169
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004483]  unevictable:7929 dirty:0 writeback:0
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004483]  slab_reclaimable:31714 slab_unreclaimable:40818
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004483]  mapped:530612 shmem:527997 pagetables:12373 bounce:0
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004483]  kernel_misc_reclaimable:0
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004483]  free:36477 free_pcp:4 free_cma:0
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004487] Node 0 active_anon:117288kB inactive_anon:15190396kB active_file:784kB inactive_file:16088kB unevictable:31716kB isolated(anon):0kB isolated(file):676kB mapped:2122448kB dirty:0kB writeback:0kB shmem:2111988kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:12752kB pagetables:49492kB all_unreclaimable? yes
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004492] Node 0 DMA free:13312kB min:60kB low:72kB high:84kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004496] lowmem_reserve[]: 0 1862 15842 15842 15842
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004500] Node 0 DMA32 free:58904kB min:7936kB low:9920kB high:11904kB reserved_highatomic:0KB active_anon:2780kB inactive_anon:1899428kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:2076064kB managed:2009636kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004504] lowmem_reserve[]: 0 0 13979 13979 13979
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004509] Node 0 Normal free:73692kB min:59580kB low:74472kB high:89364kB reserved_highatomic:14336KB active_anon:114508kB inactive_anon:13290968kB active_file:296kB inactive_file:16052kB unevictable:31716kB writepending:0kB present:14680064kB managed:14324052kB mlocked:27716kB bounce:0kB free_pcp:16kB local_pcp:8kB free_cma:0kB
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004513] lowmem_reserve[]: 0 0 0 0 0
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004517] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 2*2048kB (UM) 2*4096kB (M) = 13312kB
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004529] Node 0 DMA32: 1808*4kB (ME) 283*8kB (ME) 514*16kB (ME) 399*32kB (ME) 194*64kB (ME) 61*128kB (ME) 12*256kB (ME) 4*512kB (M) 3*1024kB (M) 0*2048kB 0*4096kB = 58904kB
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004544] Node 0 Normal: 9440*4kB (UME) 2312*8kB (UME) 1255*16kB (UME) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 76336kB
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004563] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004565] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004566] 534752 total pagecache pages
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004567] 0 pages in swap cache
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004568] Swap cache stats: add 0, delete 0, find 0/0
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004569] Free swap  = 0kB
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004570] Total swap = 0kB
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004571] 4193031 pages RAM
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004572] 0 pages HighMem/MovableOnly
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004573] 105769 pages reserved
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004573] 0 pages hwpoisoned
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004574] Tasks state (memory values in pages):
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004575] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004578] [    474]     0   474    47146     1299   397312        0          -250 systemd-journal
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004582] [    519]     0   519    72337     6774   114688        0         -1000 multipathd
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004584] [    520]     0   520   309754     4239   143360        0             0 lxd-agent
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004587] [    531]     0   531     2904      950    61440        0         -1000 systemd-udevd
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004589] [    711]   103   711    22341      544    81920        0             0 systemd-timesyn
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004592] [    830]   100   830     4063      630    73728        0             0 systemd-network
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004594] [    832]   101   832     6482     1640    94208        0             0 systemd-resolve
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004596] [    869]     0   869     1822      472    49152        0             0 cron
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004598] [    871]   102   871     2242     1018    65536        0          -900 dbus-daemon
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004600] [    877]     0   877    20708      712    69632        0             0 irqbalance
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004602] [    878]     0   878     8280     2914   102400        0             0 networkd-dispat
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004605] [    879]   104   879    55618      886    81920        0             0 rsyslogd
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004607] [    882] 584788   882   270549      385   143360        0             0 prometheus-post
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004609] [    883]     0   883   893025    49259  2277376        0             0 agent
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004611] [    884]     0   884   540872    63823  1372160        0             0 mongod
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004613] [    888]     0   888   678341    33997   602112        0             0 sunbeamd
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004615] [    938]     0   938     3874      279    69632        0             0 systemd-logind
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004617] [    976]     0   976     1555      199    49152        0             0 agetty
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004619] [   1002]     0  1002     1544      207    53248        0             0 agetty
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004621] [   1068]     0  1068    27534     2854   118784        0             0 unattended-upgr
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004624] [   2400]   107  2400     2408      311    57344        0             0 uuidd
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004626] [   2817]     0  2817    23942     1725    94208        0             0 haproxy
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004628] [   2819]   114  2819   134606     2592   143360        0             0 haproxy
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004630] [  22794]     0 22794     1195       59    49152        0             0 start-patroni.s
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004632] [  22832] 584788 22832   123294     6753   188416        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004634] [1448172]     0 1448172    58626     1053    94208        0             0 polkitd
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004636] [2966210]     0 2966210   506804     2385   331776        0             0 parca-agent
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004638] [2966418]     0 2966418  1785746   932916  8708096        0             0 parca-agent
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004641] [1653021] 584788 1653021  1122448    29238   385024        0             0 postgres
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004643] [1653038] 584788 1653038    17543      655   131072        0             0 postgres
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004645] [1653040] 584788 1653040  1122561   505893  4235264        0             0 postgres
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004647] [1653041] 584788 1653041  1122503   504713  4202496        0             0 postgres
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004649] [1653042] 584788 1653042  1122448     9120   221184        0             0 postgres
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004651] [1653043] 584788 1653043    17576      677   131072        0             0 postgres
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004653] [2529772]     0 2529772     1941      550    49152        0             0 bash
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004655] [2529776]     0 2529776   276591    43360   946176        0             0 jujud
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004658] [3236004]     0 3236004    74001      758   167936        0             0 packagekitd
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004660] [3236009]     0 3236009     3859     1158    65536        0         -1000 sshd
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004662] [2395552]     0 2395552   549074     2837   258048        0             0 pebble
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004665] [2395605]     0 2395605   579444   106125  1232896        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004667] [2395607]     0 2395607    64603    21914   278528        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004669] [2395624]     0 2395624   221067    29118   430080        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004671] [2395650]     0 2395650   656000   313689  2912256        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004673] [2395653]     0 2395653   712554   392560  3596288        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004675] [2395655]     0 2395655   685409   370512  3383296        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004677] [2395656]     0 2395656   707076   395009  3530752        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004679] [2395666]     0 2395666    14758     9713   159744        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004681] [2395667]     0 2395667    14750     9713   159744        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004684] [2395669]     0 2395669   200400    21132   380928        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004686] [2395681]     0 2395681     2690      394    65536        0             0 nginx
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004688] [2395682]     0 2395682   229946    43593   602112        0             0 temporal-server
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004690] [2395715]     0 2395715     4493     1127    69632        0             0 tcpdump
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004692] [2395721]     0 2395721     4493     1182    77824        0             0 tcpdump
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004694] [2395755]     0 2395755    93548     1667   147456        0             0 rsyslogd
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004696] [2395758]     0 2395758     1200       71    49152        0             0 run-squid
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004698] [2395778] 584788 2395778   175542   159870  1396736        0             0 squid
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004700] [2396060]     0 2396060   311957     2884   184320        0             0 maas-agent
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004702] [2398086]     0 2398086     2813      520    61440        0             0 nginx
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004704] [2398087]     0 2398087     2716      408    61440        0             0 nginx
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004706] [2398088]     0 2398088     2716      403    61440        0             0 nginx
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004708] [2398089]     0 2398089     2690      396    61440        0             0 nginx
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004710] [2398090]     0 2398090     2690      394    61440        0             0 nginx
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004712] [2398091]     0 2398091     2690      394    61440        0             0 nginx
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004714] [2398092]     0 2398092     2690      394    61440        0             0 nginx
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004716] [2398093]     0 2398093     2690      394    61440        0             0 nginx
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004718] [2398713]     0 2398713    14657     9543   167936        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004720] [2398765]     0 2398765   309442     2235   126976        0             0 maas-netmon
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004722] [2398779]     0 2398779   309506     2209   135168        0             0 maas-netmon
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004724] [2398790]     0 2398790     1792       77    57344        0             0 avahi-browse
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004726] [ 803300]     0 803300    21116      196    69632        0             0 chronyd
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004729] [1894232]     0 1894232    27098     2026    98304        0             0 dhcpd
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004731] [2610311]     0 2610311   510556      985   208896        0             0 recv_buffer_wat
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004733] [3105345]     0 3105345    47131    13627   225280        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004735] [3105712]     0 3105712    46876    13632   221184        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004738] [3107467]     0 3107467    25851    11155   196608        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004739] [3107477]     0 3107477    25852    11155   200704        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004741] [3107491]     0 3107491     1200       73    49152        0             0 run-named
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004743] [3107506]     0 3107506    11737     6904   139264        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004745] [3107596]     0 3107596    16092    10910   172032        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004747] [3107597]     0 3107597    25852    11152   196608        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004749] [3107598]     0 3107598    16132    11135   180224        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004751] [3107599]     0 3107599    25852    11157   204800        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004753] [3107600]     0 3107600    15844    10892   163840        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004755] [3107601]     0 3107601    25851    11152   196608        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004757] [3107602]     0 3107602    25851    11154   192512        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004759] [3107603]     0 3107603    25852    11156   192512        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004761] [3107607]     0 3107607    25851    11155   188416        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004763] [3107608]     0 3107608    25847    11155   196608        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004765] [3107610]     0 3107610    16132    11107   172032        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004767] [3107613]     0 3107613    25852    11155   192512        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004768] [3107614]     0 3107614    25851    11154   196608        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004770] [3107615]     0 3107615    25847    11154   196608        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004772] [3107618]     0 3107618    25851    11155   188416        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004774] [3107620]     0 3107620    25851    11153   192512        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004776] [3107621]     0 3107621    25852    11156   200704        0             0 python3
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004778] [3107739]     0 3107739    12406       93    73728        0             0 rndc
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004780] [3107854]     0 3107854   308027      105    98304        0             0 amd64
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004782] [3107855]     0 3107855   308027      106    98304        0             0 amd64
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004784] [3107966]     0 3107966   308027      105    90112        0             0 amd64
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004786] [3108039]     0 3108039     2643      154    57344        0             0 ipmipower
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004788] [3108042]     0 3108042     2643      156    57344        0             0 ipmipower
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004791] [3109790] 584788 3109790     4115      113    65536        0             0 pg_isready
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004793] [3111483] 584788 3111483  1122539      732   147456        0             0 postgres
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004796] [3133389]     0 3133389   455395      251   167936        0             0 tor3-dns-debug
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004798] [3133396]     0 3133396   274409     1155   176128        0          -900 snapd
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004800] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=init.scope,mems_allowed=0,global_oom,task_memcg=/system.slice/snap.parca-agent.parca-agent-svc.service,task=parca-agent,pid=2966418,uid=0
Feb 17 20:13:36 maas-tor3-1 kernel: [2448869.004876] Out of memory: Killed process 2966418 (parca-agent) total-vm:7142984kB, anon-rss:3731664kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:8504kB oom_score_adj:0
Feb 17 20:13:36 maas-tor3-1 tor3-dns-debug[3133389]: time=2025-02-17T20:13:36.532Z level=ERROR msg="query encountered an error" server=10.239.8.11:53 error="read udp 10.239.8.11:60731->10.239.8.11:53: read: connection refused"
Feb 17 20:13:36 maas-tor3-1 tor3-dns-debug[3133389]: time=2025-02-17T20:13:36.534Z level=ERROR msg="fatal error" err="read udp 10.239.8.11:60731->10.239.8.11:53: read: connection refused"
Feb 17 20:13:36 maas-tor3-1 systemd[1]: snap.parca-agent.parca-agent-svc.service: A process of this unit has been killed by the OOM killer.

Here the journal (I filtered out all our logs and kept only the kernel/parca/systemd logs)
parca-agent-crash-17-02-2025-filtered.log

[brancz]

That's certainly many times the memory we would expect to see. Could you grab a memory profile from the agent anywhere over 1gb of memory usage? You can just scrape it from :7071/debug/pprof/heap and upload it to https://pprof.me. That should tell us where the problem is pretty clearly.

[r00ta]

Sure thing, here it is https://pprof.me/6bbf8ad4fbd767c8a930db87f7ba8a08

root 1093761 2.4 13.5 5755636 2.11 GB ? Sl Mar07 495:38 /snap/parca-agent/2587/parca-agent

[gnurizen]

Is this still happening with the latest version ( 0.39.1), there's been a lot of memory related fixes.

[jurekh]

We still see memory increase over time with v0.39.1, though much more slowly than before. We haven't had outages caused by it yet, and we're monitoring memory increase.

In the graph you can see the effect of parca-agent restart on 10th of June - significant drop in memory, followed by a slow but steady increase.

[brancz]

Can you try 0.39.2? I believe that fixes it.

[jurekh]

Sure, just updated all our nodes to 0.39.2, will keep an eye on memory consumption for a while and will report back.

[brancz]

Awesome, thank you!

[jurekh]

Quick update after running v0.39.2 for 10 days:

• To the left of the first cyan line, you can see parca-agent v.0.39.1. To the right of that line - parca-agent v0.39.2. The slopes of memory utilization chart before the upgrade are much steeper than after the upgrade. With v0.39.2, the slope is nearly flat.

• The second cyan line is an upgrade and restart of MAAS, this freed up some memory which explains the drop. This is irrelevant to this issue, but we disturbed the environment which made it hard to see potential memory leaks.

In short, v0.39.2 is not leaking memory as fast as the previous version. We'll continue monitoring it for a longer period and report for the final time.

[jurekh]

After monitoring the memory usage in our system for a significant while, we haven't seen any parca-related memory leaks, and memory usage remains flat over time. The issue can be closed.

[brancz]

Awesome! Thank you so much for confirming!