Page 1 of 1

Fast downloads but slow unpack

Posted: 05 Aug 2020, 00:06
by seannymurrs
I’m running NZBGet on my seedbox through Whatbox.ca. I’m very happy with my download speeds; they are typically between 300-500MB/s. Even though I’m on one of their SSD slots, however, I’m finding that unpacking seems to take quite a long time. Most of what I’m downloading are quite large files (4K Remux files >50GB), so I know it’s not going to be instantaneous. That being said, I still feel like it’s slower than I should be. Up until now, I’ve had all the “direct” options enabled (directwrite, directrename, directunpack, etc...). I’ve noticed that, unless the download is going slower than normal, I don’t actually see any evidence of directunpack actually happening. I’m assuming this is because the file is being downloaded fast enough that it doesn’t have an opportunity to start unpacking (though I could be completely wrong on this). What I’m wondering, is whether there are any specific settings to help in situations where my download speed is plenty fast, but unpacking seems to be the definite bottleneck. While I will occasionally have a large queue of files (if I’m trying to get an entire season of a specific tv show), most of the time I’m downloading only one or two things at once. Any suggestions would be greatly appreciated.

Re: Fast downloads but slow unpack

Posted: 06 Aug 2020, 09:44
by hugbug
Unpack is performed by external tool "unrar". For a test you could disable unpack in nzbget completely, then unpack manually using unrar and compare the results (unpack time).

Re: Fast downloads but slow unpack

Posted: 06 Aug 2020, 16:37
by seannymurrs
Is it possible to see the unpack time for an individual NZB? I’m sure I could check the time stamps for when it started/finished unpacking, but I wasn’t sure if that info was easily viewable somewhere.

Re: Fast downloads but slow unpack

Posted: 06 Aug 2020, 16:52
by hugbug
Click on an item in history. Then on label "Total ->" in statistics box.

Re: Fast downloads but slow unpack

Posted: 06 Aug 2020, 21:16
by seannymurrs
hugbug wrote:
06 Aug 2020, 16:52
Click on an item in history. Then on label "Total ->" in statistics box.
Got it, thanks. I downloaded a few large files, and below are their sizes and unpack times. To be honest, I don't know what is considered "good" or "bad" speeds for an SSD when it comes to unpacking. My results are confusing, to say the least. There seems to be a really wide range of times, and it doesn't seem to be directly related to the size of the file. Is there anything else I can look at to try and figure out what's going on?

45.0 GB, 0:03:35 Unpack time
47.2 GB, 0:08:50 Unpack time
50.8 GB, 0:31:32 Unpack time
15.5 GB, 0:00:15 Unpack time
49.0 GB, 0:54:35 Unpack time

Re: Fast downloads but slow unpack

Posted: 07 Aug 2020, 08:29
by sanderj
seannymurrs wrote:
06 Aug 2020, 21:16
45.0 GB, 0:03:35 Unpack time
47.2 GB, 0:08:50 Unpack time
50.8 GB, 0:31:32 Unpack time
15.5 GB, 0:00:15 Unpack time
49.0 GB, 0:54:35 Unpack time
15.5 GB, 0:00:15 Unpack time ... so 15 GB in 15 seconds, or 1GB/s = 1000 MB/s ... which is very fast.
- A M.2 SATA SSD can achieve 500 MB/s or so.
- A M.2 NVME SSD can indeed achieve Gigabytes per second: Samsung’s 970 Evo Plus seems to do 2500 MB/s
So ... what kind of SSD have you got? Is 1000 MB/s possible at all? If not, it must be a bad measurement or presentation in NZBget (unlikely, but still).

Then:
45.0 GB, 0:03:35 Unpack time
49.0 GB, 0:54:35 Unpack time

Same size, very different unpack times (almost factor 20 !). Nzbget's "Unpack time" is the pure unpacking, not repairing (which can take quite some time), so I can only guess ... disk problem?

So: If you retry the same 45 resp 49 GB downloads, do you get the same Unpack Time? In other words: is it reproducible?

Re: Fast downloads but slow unpack

Posted: 07 Aug 2020, 09:56
by hugbug
"Unpack time" is the pure length of unpack post-processing stage. When direct unpack is enabled the file can be almost completely unpacked when the unpack post-processing stage starts. In that case the reported "Unpack time" can be very short.

When unpack is performed simultaneously with other activities (options UnpackPauseQueue, DirectUnpack, PostStrategy) then we can't really compare the numbers. Other activities (download, repair) may take a lot of CPU and disk resources.

Some releases use rar only as container, the compression is disabled. Such archives can be unpacked very fast, as fast as copying a file. Other releases may have compression enabled, then the unpack requires a lot of resources to decode compressed data.

To determine if unpack times in nzbget are ok the same archives must be unpacked using unrar tool manually for comparison (on the same computer). That's what I suggested in my first reply in this topic.