Page 1 of 2

Slow unpacking - can I improve?

Posted: 12 Jul 2018, 20:00
by Binson_Buzz
I'm struggling to unpack/extract files quickly - fairly certain it's my settings as I have a decent system E5-2683V3, 64GB ram with downloads going to a HDD and extraction to a SSD. I've done it this way so that DestDir and InterDir are on different drives - I've tried the other way around (SSD->HDD) which has had no impact. By slow unpacking, we're talking over a day for a 50-70GB series pack.

I'm running the linuxserver docker on unraid and nzbget has access to all cores (14).

I've run through the performance tips and I have a few questions about some of the variables which might explain why I'm so slow. Can someone help please.

Thanks in advance
  • ParScan- ??? Performance tips say Auto, but I only have Limited, Extended, Full or Dupe.I'm trying Limited
  • PostStrategy - Balanced. This one I'm really not sure about. Should I go Rocket, as I'm guessing that unpacks take say 50% longer, but because it can do 6 at a time, overall it's faster?
  • ArticleCache - 1800. Are there any advantages in going beyond the recommended 100MB, even a couple of GB???
  • DirectWrite - Yes. Think this makes sense
  • WriteBuffer - 1024. Are there any advantages in going beyond the recommended 1024, even a couple of MB/GB???
  • ParBuffer - 4096MB. It says a few hundreds in the guide - is bigger worse?
  • CrcCheck- Yes
  • ParCheck- Auto
  • ParRepair, ParRename, RarRename, DirectRename, DirectUnpack- Yes
  • ParQuick- Yes
  • ParThreads- 14
  • ParScan- Full
  • ParPauseQueue, UnpackPauseQueue- No

Re: Slow unpacking - can I improve?

Posted: 12 Jul 2018, 20:18
by hugbug
All settings are OK.

What do you mean with "slow"? How do they compare to unpacking same archives on a PC?

Re: Slow unpacking - can I improve?

Posted: 12 Jul 2018, 21:47
by Binson_Buzz
Thanks for replying - here's an example
Statistic Data
Downloaded size 4.31 GB
Download speed 3.64 MB/s
Total time 2:36:58
Download time 0:20:13
Verification time 0:00:00
Repair time 0:00:00
Unpack time 2:09:28
It's been a while since I used to use winrar and do manually, but for a 4GB file that would take me a few mins not 2 hours.

Another one:
Statistic Data
Downloaded size 2.65 GB
Download speed 7.23 MB/s
Total time 0:41:21
Download time 0:06:15
Verification time 0:00:00
Repair time 0:00:00
Unpack time 0:34:46
TV Series Pack:
Statistic Data
Downloaded size 76.5 GB
Download speed 4.35 MB/s
Total time 23:32:15
Download time 5:00:04
Verification time 0:00:00
Repair time 0:00:00
Unpack time 18:02:32

Re: Slow unpacking - can I improve?

Posted: 12 Jul 2018, 23:07
by hugbug
That's ridiculously slow indeed.
Unpacking is performed by unrar which nzbget executes as a separate process. You have to find why unrar is slow in your case. To do a manual unpack disable postprocessing in nzbget via menu near main round button. For this test you need a download item containing archive files which doesn't require repairing nor renaming (because postprocessing is off). When the download completes put nzbget to pause and try unpacking using unrar from terminal inside docker container running nzbget.

Is it as slow as unpack from nzbget? Then try unpack from terminal running on host (not in docker). Is there any difference? Try copying files from InterDir to DestDir: do copying inside docker and then on host. How fast is it?

Are your downloads password protected?

Re: Slow unpacking - can I improve?

Posted: 12 Jul 2018, 23:20
by Binson_Buzz
will give it a go. Can I get a quick pointer on how to unpack with the docker please? I know how to run commands within the docker, just not the command

Re: Slow unpacking - can I improve?

Posted: 13 Jul 2018, 08:47
by hugbug
Within docker terminal navigate to directory with rar-files, then:
/path/to/unrar x "*.rar" /path/to/extract
In linuxserver docker image "unrar" should be near "nzbget" executable.

Re: Slow unpacking - can I improve?

Posted: 13 Jul 2018, 09:34
by sanderj
Binson_Buzz wrote:
12 Jul 2018, 20:00
I'm running the linuxserver docker on unraid and <snip>
Ouch ... again "docker on unraid" with (disk) problems. Related to https://github.com/nzbget/nzbget/issues/558 ?

On Linux, can we see what's going on at disk / I/O level? Maybe tools can show if the application or library or OS is waiting for the disk? Or even (alleged) disk errors?


Code: Select all

iostat -y 5

Code: Select all

sar
@Binson_Buzz : unraid is the host os and it's file system oriented OS, correct?

Re: Slow unpacking - can I improve?

Posted: 13 Jul 2018, 10:32
by Binson_Buzz
hugbug wrote:
13 Jul 2018, 08:47
Within docker terminal navigate to directory with rar-files, then:
/path/to/unrar x "*.rar" /path/to/extract
In linuxserver docker image "unrar" should be near "nzbget" executable.
Thanks will try tonight

Re: Slow unpacking - can I improve?

Posted: 13 Jul 2018, 10:59
by Binson_Buzz
sanderj wrote:
13 Jul 2018, 09:34
Binson_Buzz wrote:
12 Jul 2018, 20:00
I'm running the linuxserver docker on unraid and <snip>
Ouch ... again "docker on unraid" with (disk) problems. Related to https://github.com/nzbget/nzbget/issues/558 ?

On Linux, can we see what's going on at disk / I/O level? Maybe tools can show if the application or library or OS is waiting for the disk? Or even (alleged) disk errors?


Code: Select all

iostat -y 5

Code: Select all

sar
@Binson_Buzz : unraid is the host os and it's file system oriented OS, correct?
Thanks - just read through that thread. My problems do sound similar, although I'm using the linuxserver docker not binhex. Yes, I'm running this as a docker within unraid.

One thing that jumped out at me in the thread was the mention of InterDir. Previously I had my DestDir and InterDir on the same drive outside of my unRAID array, whereas I moved to a seperate 'disk' around the same time I started having problems with the DestDir on the array and the InterDir outside of the array (well, kind of - on my cache drive). It was the only way I could simulate using two different drives. I'll try just using my cache drive and also the same array drive for both DestDir and InterDir to see if that's a quick fix.

Re: Slow unpacking - can I improve?

Posted: 13 Jul 2018, 16:12
by Binson_Buzz
Major improvement - down to seconds from hours!!

I've switched to nzbget downloading to my SSD cache drive and then extracting to the array. My post-processing queue has fully cleared itself. One repair was a bit long, but maybe that was necessary. Will monitor and see what happens after a couple of days running if I extract to the SSD cache drive as well.

Thanks for the help.

Oldest history examples at the top. You can see it getting faster as the backlog clears

Statistic Data
Downloaded size 8.40 GB
Download speed 4.71 MB/s
Total time 0:50:27
Download time 0:30:28
Verification time 0:00:00
Repair time 0:00:00
Unpack time 0:19:41


Statistic Data
Downloaded size 8.29 GB
Download speed 11.3 MB/s
Total time 0:32:57
Download time 0:12:29
Verification time 0:00:00
Repair time 0:00:00
Unpack time 0:20:05


Statistic Data
Downloaded size 9.03 GB
Download speed 12.5 MB/s
Total time 0:13:55
Download time 0:12:18
Verification time 0:00:00
Repair time 0:00:00
Unpack time 0:01:33

Love this one!

Statistic Data
Downloaded size 9.16 GB
Download speed 7.00 MB/s
Total time 0:22:29
Download time 0:22:21
Verification time 0:00:00
Repair time 0:00:00
Unpack time 0:00:03


Statistic Data
Downloaded size 8.45 GB
Download speed 13.5 MB/s
Total time 2:07:28
Download time 0:10:40
Verification time 0:28:25
Repair time 1:17:56
Unpack time 0:09:58


Statistic Data
Downloaded size 9.24 GB
Download speed 12.8 MB/s
Total time 0:12:24
Download time 0:12:17
Verification time 0:00:00
Repair time 0:00:00
Unpack time 0:00:02

Downloaded size 10.2 GB
Download speed 13.0 MB/s
Total time 0:13:33
Download time 0:13:27
Verification time 0:00:00
Repair time 0:00:00
Unpack time 0:00:03