Slow unpacking - can I improve?

Get help, report and discuss bugs.
Binson_Buzz
Posts: 13
Joined: 24 Jun 2018, 10:29

Slow unpacking - can I improve?

Post by Binson_Buzz » 12 Jul 2018, 20:00

I'm struggling to unpack/extract files quickly - fairly certain it's my settings as I have a decent system E5-2683V3, 64GB ram with downloads going to a HDD and extraction to a SSD. I've done it this way so that DestDir and InterDir are on different drives - I've tried the other way around (SSD->HDD) which has had no impact. By slow unpacking, we're talking over a day for a 50-70GB series pack.

I'm running the linuxserver docker on unraid and nzbget has access to all cores (14).

I've run through the performance tips and I have a few questions about some of the variables which might explain why I'm so slow. Can someone help please.

Thanks in advance
  • ParScan- ??? Performance tips say Auto, but I only have Limited, Extended, Full or Dupe.I'm trying Limited
  • PostStrategy - Balanced. This one I'm really not sure about. Should I go Rocket, as I'm guessing that unpacks take say 50% longer, but because it can do 6 at a time, overall it's faster?
  • ArticleCache - 1800. Are there any advantages in going beyond the recommended 100MB, even a couple of GB???
  • DirectWrite - Yes. Think this makes sense
  • WriteBuffer - 1024. Are there any advantages in going beyond the recommended 1024, even a couple of MB/GB???
  • ParBuffer - 4096MB. It says a few hundreds in the guide - is bigger worse?
  • CrcCheck- Yes
  • ParCheck- Auto
  • ParRepair, ParRename, RarRename, DirectRename, DirectUnpack- Yes
  • ParQuick- Yes
  • ParThreads- 14
  • ParScan- Full
  • ParPauseQueue, UnpackPauseQueue- No

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Slow unpacking - can I improve?

Post by hugbug » 12 Jul 2018, 20:18

All settings are OK.

What do you mean with "slow"? How do they compare to unpacking same archives on a PC?

Binson_Buzz
Posts: 13
Joined: 24 Jun 2018, 10:29

Re: Slow unpacking - can I improve?

Post by Binson_Buzz » 12 Jul 2018, 21:47

Thanks for replying - here's an example
Statistic Data
Downloaded size 4.31 GB
Download speed 3.64 MB/s
Total time 2:36:58
Download time 0:20:13
Verification time 0:00:00
Repair time 0:00:00
Unpack time 2:09:28
It's been a while since I used to use winrar and do manually, but for a 4GB file that would take me a few mins not 2 hours.

Another one:
Statistic Data
Downloaded size 2.65 GB
Download speed 7.23 MB/s
Total time 0:41:21
Download time 0:06:15
Verification time 0:00:00
Repair time 0:00:00
Unpack time 0:34:46
TV Series Pack:
Statistic Data
Downloaded size 76.5 GB
Download speed 4.35 MB/s
Total time 23:32:15
Download time 5:00:04
Verification time 0:00:00
Repair time 0:00:00
Unpack time 18:02:32

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Slow unpacking - can I improve?

Post by hugbug » 12 Jul 2018, 23:07

That's ridiculously slow indeed.
Unpacking is performed by unrar which nzbget executes as a separate process. You have to find why unrar is slow in your case. To do a manual unpack disable postprocessing in nzbget via menu near main round button. For this test you need a download item containing archive files which doesn't require repairing nor renaming (because postprocessing is off). When the download completes put nzbget to pause and try unpacking using unrar from terminal inside docker container running nzbget.

Is it as slow as unpack from nzbget? Then try unpack from terminal running on host (not in docker). Is there any difference? Try copying files from InterDir to DestDir: do copying inside docker and then on host. How fast is it?

Are your downloads password protected?

Binson_Buzz
Posts: 13
Joined: 24 Jun 2018, 10:29

Re: Slow unpacking - can I improve?

Post by Binson_Buzz » 12 Jul 2018, 23:20

will give it a go. Can I get a quick pointer on how to unpack with the docker please? I know how to run commands within the docker, just not the command

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Slow unpacking - can I improve?

Post by hugbug » 13 Jul 2018, 08:47

Within docker terminal navigate to directory with rar-files, then:
/path/to/unrar x "*.rar" /path/to/extract
In linuxserver docker image "unrar" should be near "nzbget" executable.

sanderj
Posts: 184
Joined: 10 Feb 2014, 21:46

Re: Slow unpacking - can I improve?

Post by sanderj » 13 Jul 2018, 09:34

Binson_Buzz wrote:
12 Jul 2018, 20:00
I'm running the linuxserver docker on unraid and <snip>
Ouch ... again "docker on unraid" with (disk) problems. Related to https://github.com/nzbget/nzbget/issues/558 ?

On Linux, can we see what's going on at disk / I/O level? Maybe tools can show if the application or library or OS is waiting for the disk? Or even (alleged) disk errors?


Code: Select all

iostat -y 5

Code: Select all

sar
@Binson_Buzz : unraid is the host os and it's file system oriented OS, correct?

Binson_Buzz
Posts: 13
Joined: 24 Jun 2018, 10:29

Re: Slow unpacking - can I improve?

Post by Binson_Buzz » 13 Jul 2018, 10:32

hugbug wrote:
13 Jul 2018, 08:47
Within docker terminal navigate to directory with rar-files, then:
/path/to/unrar x "*.rar" /path/to/extract
In linuxserver docker image "unrar" should be near "nzbget" executable.
Thanks will try tonight

Binson_Buzz
Posts: 13
Joined: 24 Jun 2018, 10:29

Re: Slow unpacking - can I improve?

Post by Binson_Buzz » 13 Jul 2018, 10:59

sanderj wrote:
13 Jul 2018, 09:34
Binson_Buzz wrote:
12 Jul 2018, 20:00
I'm running the linuxserver docker on unraid and <snip>
Ouch ... again "docker on unraid" with (disk) problems. Related to https://github.com/nzbget/nzbget/issues/558 ?

On Linux, can we see what's going on at disk / I/O level? Maybe tools can show if the application or library or OS is waiting for the disk? Or even (alleged) disk errors?


Code: Select all

iostat -y 5

Code: Select all

sar
@Binson_Buzz : unraid is the host os and it's file system oriented OS, correct?
Thanks - just read through that thread. My problems do sound similar, although I'm using the linuxserver docker not binhex. Yes, I'm running this as a docker within unraid.

One thing that jumped out at me in the thread was the mention of InterDir. Previously I had my DestDir and InterDir on the same drive outside of my unRAID array, whereas I moved to a seperate 'disk' around the same time I started having problems with the DestDir on the array and the InterDir outside of the array (well, kind of - on my cache drive). It was the only way I could simulate using two different drives. I'll try just using my cache drive and also the same array drive for both DestDir and InterDir to see if that's a quick fix.

Binson_Buzz
Posts: 13
Joined: 24 Jun 2018, 10:29

Re: Slow unpacking - can I improve?

Post by Binson_Buzz » 13 Jul 2018, 16:12

Major improvement - down to seconds from hours!!

I've switched to nzbget downloading to my SSD cache drive and then extracting to the array. My post-processing queue has fully cleared itself. One repair was a bit long, but maybe that was necessary. Will monitor and see what happens after a couple of days running if I extract to the SSD cache drive as well.

Thanks for the help.

Oldest history examples at the top. You can see it getting faster as the backlog clears

Statistic Data
Downloaded size 8.40 GB
Download speed 4.71 MB/s
Total time 0:50:27
Download time 0:30:28
Verification time 0:00:00
Repair time 0:00:00
Unpack time 0:19:41


Statistic Data
Downloaded size 8.29 GB
Download speed 11.3 MB/s
Total time 0:32:57
Download time 0:12:29
Verification time 0:00:00
Repair time 0:00:00
Unpack time 0:20:05


Statistic Data
Downloaded size 9.03 GB
Download speed 12.5 MB/s
Total time 0:13:55
Download time 0:12:18
Verification time 0:00:00
Repair time 0:00:00
Unpack time 0:01:33

Love this one!

Statistic Data
Downloaded size 9.16 GB
Download speed 7.00 MB/s
Total time 0:22:29
Download time 0:22:21
Verification time 0:00:00
Repair time 0:00:00
Unpack time 0:00:03


Statistic Data
Downloaded size 8.45 GB
Download speed 13.5 MB/s
Total time 2:07:28
Download time 0:10:40
Verification time 0:28:25
Repair time 1:17:56
Unpack time 0:09:58


Statistic Data
Downloaded size 9.24 GB
Download speed 12.8 MB/s
Total time 0:12:24
Download time 0:12:17
Verification time 0:00:00
Repair time 0:00:00
Unpack time 0:00:02

Downloaded size 10.2 GB
Download speed 13.0 MB/s
Total time 0:13:33
Download time 0:13:27
Verification time 0:00:00
Repair time 0:00:00
Unpack time 0:00:03

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 52 guests