Page 1 of 2

v21 on RPI4 suddenly crashing

Posted: 25 Nov 2020, 17:37
by ProXy
Hello,

I am running v21 on a RPI4 for several months now and never had issues with nzgbet crashing. Now I added some more downloads to the queue (1 downloading, 5 queued, 30 paused) and suddenly nzbget crashes every few minutes. I checked the logs but nothing gets written. After restarting the service it resumes working without any issues but crashes only a few minutes later. I disabled unpacking because I thought there could an issue but that didn't change anything.

Does anyone have a clue what could be the reason for the crashes?

Thank you!

Re: v21 on RPI4 suddenly crashing

Posted: 25 Nov 2020, 19:17
by hugbug
Is this reproducible with a certain nzb or random?
You don't use an sd-card with nzbget, right? Sd-cards wear off pretty quick.

Re: v21 on RPI4 suddenly crashing

Posted: 25 Nov 2020, 21:14
by ProXy
Hello Hugbug,

this happens with multiple NZBs in the last few days. The downloads are quite big (~100GB each), but memory usage looked fine. I am downloading on a 2TB HDD attached to the RPI, the setup hasn‘t change since a few months and it was rock solid until now.

I had to restart the service 4 or 5 times in the last 6 hours but there was no hint in the log what could have caused the crash.

Re: v21 on RPI4 suddenly crashing

Posted: 26 Nov 2020, 09:53
by sanderj
Do you mean a real crash, so a crash-crash? Or "not responding", or "exited", or "slow"?

If a crash-crash, run nzbget from console (not as a daemon / service). A real crash looks like this:

Code: Select all

$ ./crash_me
Hello, World!
Floating point exception (core dumped)
So: it tells what went wrong. To really have the core file generated, do this (assuming running from bash):

Code: Select all

$ ulimit -c unlimited
$ ./crash_me
Hello, World!
Floating point exception (core dumped)
and you'll have the core file for analysis:

Code: Select all

$ ll core
-rw------- 1 sander sander 380928 Nov 26 10:47 core
And then

Code: Select all

$ gdb crash_me core
GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
...
Reading symbols from crash_me...(no debugging symbols found)...done.
[New LWP 27446]
Core was generated by `./crash_me'.
Program terminated with signal SIGFPE, Arithmetic exception.
#0  0x000055b5af35c660 in main ()
(gdb)
So the Arithmetic exception happened in main().

Re: v21 on RPI4 suddenly crashing

Posted: 26 Nov 2020, 10:06
by sanderj
... and the OS can also decide to kill a process if the OS thinks it's causing problems (for example eating all memory):

Code: Select all

$ ./mem_eater
Hello, World!
Killed
... and /var/log/ will tell what happened, so what the Linux kernel decided to do:

Code: Select all

$ sudo grep mem_eater /var/log/syslog
Nov 26 11:02:58 brixit kernel: [252348.826442] [ 922115]  1000 922115 4962939054   196375 909447168     6696             0 mem_eater
Nov 26 11:02:58 brixit kernel: [252348.826461] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-12388.scope,task=mem_eater,pid=922115,uid=1000
Nov 26 11:02:58 brixit kernel: [252348.826475] Out of memory: Killed process 922115 (mem_eater) total-vm:19851756216kB, anon-rss:785496kB, file-rss:4kB, shmem-rss:0kB, UID:1000 pgtables:888132kB oom_score_adj:0
Nov 26 11:02:58 brixit kernel: [252349.247163] oom_reaper: reaped process 922115 (mem_eater), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
So:

Code: Select all

Out of memory: Killed process 922115 (mem_eater) total-vm:19851756216kB, anon-rss:785496kB
Haha, mem_eater ate 19000 GB of virtual memory (so: assigned, but not really used), and 785MB of real memory. "RSS" stands for "resident set size," one way to measure the amount of memory used by a process in Linux.

Re: v21 on RPI4 suddenly crashing

Posted: 26 Nov 2020, 15:47
by ProXy
It seems that the data on the hdd was corrupted. journalctl showed up some inode errors on the drive, so I used fsck to repair the drive. After a reboot no more errors are shown, so I guess that was the issue.

Something starts to corrupt my data as soon as the system gets high load. The drive itselfs works fine on other devices and I changed it to be sure. Sadly the issue remains.

Re: v21 on RPI4 suddenly crashing

Posted: 27 Nov 2020, 10:50
by sanderj
ProXy wrote:
26 Nov 2020, 15:47
Sadly the issue remains.
And ... did you do anything with what I wrote? Did you answer the first question?

Re: v21 on RPI4 suddenly crashing

Posted: 27 Nov 2020, 13:43
by hugbug
RPi needs a good power supply unit.
Is this HDD powered via USB from RPi? In that case you need an even better power supply unit.

Re: v21 on RPI4 suddenly crashing

Posted: 23 Dec 2020, 12:35
by ProXy
Hey again.

I changed my drive to a 3.5 hdd and don't have any file system issues so far. Today NZBGet crashed again with an out of memory exception. Here is the log:

Code: Select all

-- Logs begin at Sun 2020-12-13 13:52:29 CET, end at Wed 2020-12-23 13:25:20 CET
. --
Dez 23 08:09:23 DietPi kernel: unrar invoked oom-killer:
 gfp_mask=0x40cd0(GFP_KERNEL|__GFP_COMP|__GFP_RECLAIMABLE), order=0, oom_score_a
dj=0
Dez 23 08:09:23 DietPi kernel: CPU: 2 PID: 18461 Comm: u
nrar Tainted: G         C        5.4.72-v7l+ #1356
Dez 23 08:09:23 DietPi kernel: Hardware name: BCM2711
Dez 23 08:09:23 DietPi kernel: Backtrace:
Dez 23 08:09:23 DietPi kernel: [<c020e1c8>] (dump_backtr
ace) from [<c020e544>] (show_stack+0x20/0x24)
Dez 23 08:09:23 DietPi kernel:  r7:ffffffff r6:00000000
r5:60000013 r4:c12a1130
Dez 23 08:09:23 DietPi kernel: [<c020e524>] (show_stack)
 from [<c0a84cb0>] (dump_stack+0xd4/0x118)
Dez 23 08:09:23 DietPi kernel: [<c0a84bdc>] (dump_stack)
 from [<c039250c>] (dump_header+0x64/0x1f8)
Dez 23 08:09:23 DietPi kernel:  r10:00040cd0 r9:c259c000
 r8:00000000 r7:00000558 r6:c0ddc7ac r5:ddc6db80
Dez 23 08:09:23 DietPi kernel:  r4:c259da68 r3:00000000

Dez 23 08:09:23 DietPi kernel: [<c03924a8>] (dump_header
) from [<c03918a4>] (oom_kill_process+0x17c/0x188)
Dez 23 08:09:23 DietPi kernel:  r7:00000558 r6:c0ddc7ac r5:c259da68 r4:ddc6db80
Dez 23 08:09:23 DietPi kernel: [<c0391728>] (oom_kill_process) from [<c039230c>] (out_of_memory+0x244/0x344)
Dez 23 08:09:23 DietPi kernel:  r7:c1204ff0 r6:c12083c0 r5:c259da68 r4:ddc6db80
Dez 23 08:09:23 DietPi kernel: [<c03920c8>] (out_of_memory) from [<c03e1154>] (__alloc_pages_nodemask+0xbfc/0x1230)
Dez 23 08:09:23 DietPi kernel:  r6:c12a3550 r5:00002eed r4:00000000
Dez 23 08:09:23 DietPi kernel: [<c03e0558>] (__alloc_pages_nodemask) from [<c03f28a4>] (alloc_slab_page+0x34/0x430)
Dez 23 08:09:23 DietPi kernel:  r10:00000000 r9:0000000d r8:00040cd0 r7:ffffffff r6:00000000 r5:ef803f00
Dez 23 08:09:23 DietPi kernel:  r4:00000000
Dez 23 08:09:23 DietPi kernel: [<c03f2870>] (alloc_slab_page) from [<c03f2f58>] (new_slab+0x2b8/0x32c)
Dez 23 08:09:23 DietPi kernel:  r10:00000000 r9:0000000d r8:00000400 r7:ffffffff r6:00040cd0 r5:ef803f00
Dez 23 08:09:23 DietPi kernel:  r4:00000000
Dez 23 08:09:23 DietPi kernel: [<c03f2ca0>] (new_slab) from [<c03f5440>] (___slab_alloc.constprop.0+0x374/0x55c)
Dez 23 08:09:23 DietPi kernel:  r9:c259dca0 r8:eff9e530 r7:ef803f00 r6:ef800880 r5:ef803f00 r4:00000000
Dez 23 08:09:23 DietPi kernel: [<c03f50cc>] (___slab_alloc.constprop.0) from [<c03f566c>] (__slab_alloc.constprop.0+0x44/0x90)
Dez 23 08:09:23 DietPi kernel:  r10:00000000 r9:c259dca0 r8:eff9e530 r7:c0a9bcb8 r6:00400cc0 r5:ef803f00
Dez 23 08:09:23 DietPi kernel:  r4:60010013
Dez 23 08:09:23 DietPi kernel: [<c03f5628>] (__slab_alloc.constprop.0) from [<c03f6044>] (kmem_cache_alloc+0x2d4/0x2dc)
Dez 23 08:09:23 DietPi kernel:  r9:c259dca0 r8:c0a9bcb8 r7:ef803f00 r6:00400cc0 r5:00000000 r4:ef803f00
Dez 23 08:09:23 DietPi kernel: [<c03f5d70>] (kmem_cache_alloc) from [<c0a9bcb8>] (xas_nomem+0x68/0x8c)
Dez 23 08:09:23 DietPi kernel:  r10:00000000 r9:c259dca0 r8:00000000 r7:ffffc005 r6:00000cc0 r5:e8eb2d10
Dez 23 08:09:23 DietPi kernel:  r4:c259dc4c
Dez 23 08:09:23 DietPi kernel: [<c0a9bc50>] (xas_nomem) from [<c038a990>] (__add_to_page_cache_locked+0x1b0/0x378)
Dez 23 08:09:23 DietPi kernel:  r5:e8eb2d10 r4:e1b25070
Dez 23 08:09:23 DietPi kernel: [<c038a7e0>] (__add_to_page_cache_locked) from [<c038abf0>] (add_to_page_cache_lru+0x6c/0x130)
Dez 23 08:09:23 DietPi kernel:  r10:c12a7d23 r9:c120594c r8:00674900 r7:e1b25070 r6:00101cca r5:00101cca
Dez 23 08:09:23 DietPi kernel:  r4:e8eb2d10
Dez 23 08:09:23 DietPi kernel: [<c038ab84>] (add_to_page_cache_lru) from [<c038ae4c>] (pagecache_get_page+0x198/0x384)
Dez 23 08:09:23 DietPi kernel:  r6:00101cca r5:0000000e r4:e8eb2d10
Dez 23 08:09:23 DietPi kernel: [<c038acb4>] (pagecache_get_page) from [<c038b814>] (grab_cache_page_write_begin+0x28/0x3c)
Dez 23 08:09:23 DietPi kernel:  r10:c0c13b4c r9:00000006 r8:74900000 r7:c259ddc0 r6:e1b24f60 r5:c259dec0
Dez 23 08:09:23 DietPi kernel:  r4:d0d6cd80
Dez 23 08:09:23 DietPi kernel: [<c038b7ec>] (grab_cache_page_write_begin) from [<c04e7c2c>] (ext4_da_write_begin+0xc4/0x4a8)
Dez 23 08:09:23 DietPi kernel:  r5:c259dec0 r4:d0d6cd80
Dez 23 08:09:23 DietPi kernel: [<c04e7b68>] (ext4_da_write_begin) from [<c0388b68>] (generic_perform_write+0xd0/0x1f0)
Dez 23 08:09:23 DietPi kernel:  r10:c0c13b4c r9:00000000 r8:d0d6cd80 r7:c04e7b68 r6:e1b25070 r5:c259dec0
Dez 23 08:09:23 DietPi kernel:  r4:00001000
Dez 23 08:09:23 DietPi kernel: [<c0388a98>] (generic_perform_write) from [<c038dc34>] (__generic_file_write_iter+0x16c/0x1fc)
Dez 23 08:09:23 DietPi kernel:  r10:c259dec0 r9:e1b25070 r8:00000000 r7:e1b24f60 r6:d0d6cd80 r5:c259ded8
Dez 23 08:09:23 DietPi kernel:  r4:00000000
Dez 23 08:09:23 DietPi kernel: [<c038dac8>] (__generic_file_write_iter) from [<c04d16e0>] (ext4_file_write_iter+0x280/0x474)
Dez 23 08:09:23 DietPi kernel:  r10:00000000 r9:e1b24fe8 r8:c259dec0 r7:e1b24f60 r6:00100000 r5:e1b24f60
Dez 23 08:09:23 DietPi kernel:  r4:c259ded8
Dez 23 08:09:23 DietPi kernel: [<c04d1460>] (ext4_file_write_iter) from [<c040c218>] (__vfs_write+0x15c/0x1ec)
Dez 23 08:09:23 DietPi kernel:  r10:00000004 r9:00000006 r8:748cd13c r7:00000000 r6:00100000 r5:c259df60
Dez 23 08:09:23 DietPi kernel:  r4:d0d6cd80
Dez 23 08:09:23 DietPi kernel: [<c040c0bc>] (__vfs_write) from [<c040f090>] (vfs_write+0xc0/0x1c8)
Dez 23 08:09:23 DietPi kernel:  r9:00100000 r8:00000000 r7:c259df60 r6:b6eba008 r5:d0d6cd80 r4:00100000
Dez 23 08:09:23 DietPi kernel: [<c040efd0>] (vfs_write) from [<c040f37c>] (ksys_write+0x74/0x100)
Dez 23 08:09:23 DietPi kernel:  r8:b6eba008 r7:00000006 r6:748cd13c r5:d0d6cd80 r4:d0d6cd80
Dez 23 08:09:23 DietPi kernel: [<c040f308>] (ksys_write) from [<c040f420>] (sys_write+0x18/0x1c)
Dez 23 08:09:23 DietPi kernel:  r9:c259c000 r8:c02011c4 r7:00000004 r6:b6eba008 r5:00100000 r4:bee99d38
Dez 23 08:09:23 DietPi kernel: [<c040f408>] (sys_write) from [<c0201000>] (ret_fast_syscall+0x0/0x28)
Dez 23 08:09:23 DietPi kernel: Exception stack(0xc259dfa8 to 0xc259dff0)
Dez 23 08:09:23 DietPi kernel: dfa0:                   bee99d38 00100000 0000001b b6eba008 00100000 00000000
Dez 23 08:09:23 DietPi kernel: dfc0: bee99d38 00100000 b6eba008 00000004 bee99d58 b6eba008 bee99d38 beea7228
Dez 23 08:09:23 DietPi kernel: dfe0: 00000000 bee93c8c 0000a4ff 0002deb6
Dez 23 08:09:23 DietPi kernel: Mem-Info:
Dez 23 08:09:23 DietPi kernel: active_anon:446892 inactive_anon:21222 isolated_anon:0
                                active_file:6108 inactive_file:460941 isolated_file:28
                                unevictable:352 dirty:308 writeback:0 unstable:0
                                slab_reclaimable:19581 slab_unreclaimable:4647
                                mapped:6271 shmem:26442 pagetables:1531 bounce:0
                                free:23888 free_pcp:1233 free_cma:21
Dez 23 08:09:23 DietPi kernel: Node 0 active_anon:1787568kB inactive_anon:84888kB active_file:24432kB inactive_file:1843764kB unevictable:1408kB isolated(anon):0kB isolated(file):112kB mapped:25084kB dirty:1232kB
writeback:0kB shmem:105768kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Dez 23 08:09:23 DietPi kernel: DMA free:48168kB min:53248kB low:57344kB high:61440kB active_anon:442760kB inactive_anon:36812kB active_file:2264kB inactive_file:1304kB unevictable:0kB writepending:0kB present:7864
32kB managed:655036kB mlocked:0kB kernel_stack:1872kB pagetables:1112kB bounce:0kB free_pcp:2396kB local_pcp:684kB free_cma:84kB
Dez 23 08:09:23 DietPi kernel: lowmem_reserve[]: 0 0 3238 3238
Dez 23 08:09:23 DietPi kernel: HighMem free:47384kB min:512kB low:21244kB high:41976kB active_anon:1344512kB inactive_anon:48076kB active_file:22556kB inactive_file:1842816kB unevictable:1408kB writepending:1232kB
 present:3315712kB managed:3315712kB mlocked:1408kB kernel_stack:0kB pagetables:5012kB bounce:0kB free_pcp:2536kB local_pcp:348kB free_cma:0kB
Dez 23 08:09:23 DietPi kernel: lowmem_reserve[]: 0 0 0 0
Dez 23 08:09:23 DietPi kernel: DMA: 171*4kB (UEHC) 989*8kB (UMHC) 2346*16kB (UEH) 30*32kB (H) 9*64kB (H) 1*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48052kB
Dez 23 08:09:23 DietPi kernel: HighMem: 661*4kB (M) 1182*8kB (UM) 316*16kB (M) 107*32kB (UM) 217*64kB (UM) 95*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46628kB
Dez 23 08:09:23 DietPi kernel: 493812 total pagecache pages
Dez 23 08:09:23 DietPi kernel: 0 pages in swap cache
Dez 23 08:09:23 DietPi kernel: Swap cache stats: add 0, delete 0, find 0/0
Dez 23 08:09:23 DietPi kernel: Free swap  = 0kB
Dez 23 08:09:23 DietPi kernel: Total swap = 0kB
Dez 23 08:09:23 DietPi kernel: 1025536 pages RAM
Dez 23 08:09:23 DietPi kernel: 828928 pages HighMem/MovableOnly
Dez 23 08:09:23 DietPi kernel: 32849 pages reserved
Dez 23 08:09:23 DietPi kernel: 16384 pages cma reserved
Dez 23 08:09:23 DietPi kernel: Tasks state (memory values in pages):
Dez 23 08:09:23 DietPi kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Dez 23 08:09:23 DietPi kernel: [    120]     0   120     9401      862   118784        0             0 systemd-journal
Dez 23 08:09:23 DietPi kernel: [    166]     0   166     4439      278    53248        0         -1000 systemd-udevd
Dez 23 08:09:23 DietPi kernel: [    252]     0   252      820      392    32768        0             0 vmtouch
Dez 23 08:09:23 DietPi kernel: [    278]     0   278     6914      103    36864        0             0 rngd
Dez 23 08:09:23 DietPi kernel: [    298]    33   298    16903     3524   143360        0             0 mono
Dez 23 08:09:23 DietPi kernel: [    413]     0   413     1978      294    40960        0             0 dhclient
Dez 23 08:09:23 DietPi kernel: [    664]     0   664     1441       22    36864        0             0 agetty
Dez 23 08:09:23 DietPi kernel: [    670]     0   670     1002       40    28672        0             0 dropbear
Dez 23 08:09:23 DietPi kernel: [  20161]     0 20161     7782      711    86016        0             0 nmbd
Dez 23 08:09:23 DietPi kernel: [  20173]     0 20173    11876      874   122880        0             0 smbd
Dez 23 08:09:23 DietPi kernel: [  20176]     0 20176    11144      582   110592        0             0 smbd-notifyd
Dez 23 08:09:23 DietPi kernel: [  20177]     0 20177    11145      533   106496        0             0 cleanupd
Dez 23 08:09:23 DietPi kernel: [  20187]   107 20187    15930     1351    90112        0             0 transmission-da
Dez 23 08:09:23 DietPi kernel: [  20203]   999 20203   333988   249840  2367488        0             0 nzbget
Dez 23 08:09:23 DietPi kernel: [  20231]   998 20231    48506    24147   499712        0             0 mono
Dez 23 08:09:23 DietPi kernel: [  20241]   997 20241    61668    34334   679936        0             0 mono
Dez 23 08:09:23 DietPi kernel: [  20252]     0 20252     1774      132    36864        0             0 cron
Dez 23 08:09:23 DietPi kernel: [  21181]  1000 21181     4005     1061    57344        0             0 python
Dez 23 08:09:23 DietPi kernel: [  21200]  1000 21200   122389   104238   978944        0             0 java
Dez 23 08:09:23 DietPi kernel: [    347]   995   347      485       13    24576        0             0 dash
Dez 23 08:09:23 DietPi kernel: [    348]   995   348    89969    21164   516096        0             0 jackett
Dez 23 08:09:23 DietPi kernel: [  19323]     0 19323    25645     4650   208896        0             0 smbd
Dez 23 08:09:23 DietPi kernel: [  18461]   999 18461      610      453    28672        0             0 unrar
Dez 23 08:09:23 DietPi kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=nzbget,pid=20203,uid=999
Dez 23 08:09:23 DietPi kernel: Out of memory: Killed process 20203 (nzbget) total-vm:1335952kB, anon-rss:997624kB, file-rss:1736kB, shmem-rss:0kB, UID:999 pgtables:2312kB oom_score_adj:0
Dez 23 08:09:24 DietPi kernel: oom_reaper: reaped process 20203 (nzbget), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Dez 23 08:09:24 DietPi systemd[1]: nzbget.service: Main process exited, code=killed, status=9/KILL
Dez 23 08:09:28 DietPi systemd[1]: nzbget.service: Failed with result 'signal'.
NZBGet is running on a rpi4 with 4GB of memory. Not sure if NZBGet uses more memory over time or some of the other tools have some memory leaks. Is there a way to narrow down the issue?

Re: v21 on RPI4 suddenly crashing

Posted: 23 Dec 2020, 16:06
by sanderj
My guess looking at your log

Code: Select all

Dez 23 08:09:23 DietPi kernel: unrar invoked oom-killer:
Ah, funny: unrar is the trigger of the oom-killer

Code: Select all

Dez 23 08:09:23 DietPi kernel: Free swap  = 0kB
Dez 23 08:09:23 DietPi kernel: Total swap = 0kB
Don't you have swap space? That indeed causes hard OoM's. My advice: have swap space/file of 2 or 4GB.

Code: Select all

Dez 23 08:09:23 DietPi kernel: Out of memory: Killed process 20203 (nzbget) total-vm:1335952kB, anon-rss:997624kB, file-rss:1736kB, shmem-rss:0kB, UID:999 pgtables:2312kB oom_score_adj:0
That is not absurd much memory: anon-rss:997624kB = 1 GB real RAM.
If I add up all rss page values, it's below 4GB RAM. Weird. Is unrar asking for too much memory?

What kind of download were you doing? What if you just download, and do not let unrar run?

EDIT

What is the output of

Code: Select all

free -m


From my small SBC:

Code: Select all

$ free -m
              total        used        free      shared  buff/cache   available
Mem:            475         170          82           3         222         290
Swap:           237           3         234
Swap is important. And the column "free" is important.