[New Feature] Unpack during downloading

Discuss newly added features or request new features.
hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

[New Feature] Unpack during downloading

Post by hugbug » 15 May 2017, 08:40

NZBGet now can start unpacking as soon as the first archive part is downloaded. Then it waits for the next archive part and continues unpacking part by part as they arrive. In a typical case when the download speed is limited by Internet connection the unpack completes a few seconds after the last archive part is downloaded.

This almost completely eliminates the extra time needed for unpack.

The new feature with the code name "direct unpack" works together with two other features:
  • Direct Rename deobfuscates files during downloading; it works similar to par-rename but works during downloading stage;
  • Reorder Files ensures the archive parts are downloaded in correct order for unpacking.
To activate direct unpacking you need to active three new options:
  • DirectUnpack=yes (disabled by default) in settings section "UNPACK";
  • DirectRename=yes (disabled by default) in settings section "CHECK AND REPAIR";
  • ReorderFiles=yes (active by default) in settings section "DOWNLOAD QUEUE";
Direct unpack requires NZBGet 19.0-testing-r1991 or newer.

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Unpack during downloading

Post by hugbug » 15 May 2017, 08:55

Examples

Typical statistics
  1. Direct unpack, nzb-1:
    Downloaded size 3.42 GB
    Download speed 9.93 MB/s
    Total time 0:06:03
    Download time 0:05:53
    Verification time 0:00:00
    Repair time 0:00:00
    Unpack time 0:00:02
  2. Direct unpack, nzb-2:
    Downloaded size 1.75 GB
    Download speed 10.5 MB/s
    Total time 0:03:06
    Download time 0:02:51
    Verification time 0:00:00
    Repair time 0:00:00
    Unpack time 0:00:07
Not a typical case
Another case which isn't so typical but interesting anyway. Here we have a download with compressed rar-files. Usually video files are not compressed (rar can't reduce size of video files) and rar is used only as convenient split tool. But sometimes posters mistakenly activate compression and the downloader needs more time to unpack such downloads.
  1. Without direct unpack (DirectUnpack=no, DirectRename=no)
    Downloaded size: 8.29 GB
    Download speed: 10.85 MB/s
    Total time: 0:21:11
    Download time: 0:13:02
    Verification time: 0:00:00
    Repair time: 0:00:00
    Unpack time: 0:07:52
  2. With direct unpack (DirectUnpack=yes, DirectRename=yes)
    Downloaded size: 7.52 GB
    Download speed: 10.75 MB/s
    Total time: 0:14:38
    Download time: 0:11:56
    Verification time: 0:00:00
    Repair time: 0:00:00
    Unpack time: 0:02:35
Three interesting things happen here:
  • It's much faster with direct unpack. Total time 14m38s vs 21m11s
  • Without direct unpack more data was downloaded: 7.52GB vs 8.29GB. Why so? This download has obfuscated file names where each par2-file has a different name. Because NZBGet can't determine par-sets it has to download all par2-files before post-processing. With direct rename NZBGet was able to determine that all par-files belong to the same par-set and it downloaded only one par2-file.
  • Reported unpack time in direct unpack mode took 2m35 which is much longer than in typical case. According to the log many archive parts were unpacked after the whole download has completed. That's because of compressed rar archive, which needs much more CPU power to process. The test machine is a not so powerful Linux box with ARM CPU 2x1700MHz.

rxp
Posts: 55
Joined: 07 Mar 2014, 18:51

Re: [New Feature] Unpack during downloading

Post by rxp » 15 May 2017, 15:04

Just updated yesterday and enabled the features today. Looking forward to trying it out! I actually downgraded my internet connection a couple years ago because unpacking took so much longer than downloading, it wasn't really worth it. This could be a big changer!

Thanks for always adding these great features.

Edit - just tried it. Wow! Really does work

Statistic Data
Downloaded size 3.16 GB
Download speed 7.96 MB/s
Total time 0:06:52
Download time 0:06:47
Verification time 0:00:00
Repair time 0:00:00
Unpack time 0:00:01

Edit 2- Works on encrypted archives too.

neilt0
Posts: 261
Joined: 10 Sep 2008, 19:11

Re: [New Feature] Unpack during downloading

Post by neilt0 » 17 May 2017, 09:25

Loving this feature, hugbug! Thanks! Time to make a donation, I think. What's a reasonable amount? :mrgreen:

Even with a fast server (in my case, Xeon® CPU E3-1225 v3 @ 3.20GHz) and SSD, this dramatically reduces unrar times:
Neil T wrote:A couple of real-world examples -- not scientific, but indicative. The first is undamaged, so unraring during download, the second required a repair first.
I think I could get the 15 second unrar quicker as I had too many NNTP connections open (hundreds) from when I was getting stuff that was 8+ years old.

Image

Image

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Unpack during downloading

Post by hugbug » 17 May 2017, 10:11

neilt0 wrote:Even with a fast server (in my case, Xeon® CPU E3-1225 v3 @ 3.20GHz) and SSD, this dramatically reduces unrar times:
The feature works actually best on fast computers where "fast" is relative to download speed meaning if the CPU and disk are not on their limits during downloading then we can unpack at the same time and this doesn't negatively affect download speed. On systems already limited by CPU performance it can be better to separate downloading and unpacking (no direct unpack, pause queue during post-processing with ParPauseQueue, UnpackPauseQueue and PostPauseQueue).

dagoob
Posts: 21
Joined: 15 Jan 2016, 23:27

Re: [New Feature] Unpack during downloading

Post by dagoob » 17 May 2017, 14:16

hugbug wrote:ReorderFiles=yes (active by default) in settings section "DOWNLOAD QUEUE";
Do scripts like FakeDetector and PasswordDetector play nice with this, since they reorder RARs for faster detection?

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Unpack during downloading

Post by hugbug » 17 May 2017, 15:02

dagoob wrote:Do scripts like FakeDetector and PasswordDetector play nice with this, since they reorder RARs for faster detection?
Probably not. There is a new queue-event NZB_NAMED, sent after the inner files are renamed. The scripts must be extended to process the new event. Pull requests for FakeDetector (which I maintain) are welcome.

salami
Posts: 68
Joined: 14 Apr 2014, 11:09
Location: Switzerland

Re: [New Feature] Unpack during downloading

Post by salami » 22 May 2017, 07:20

Awesome, seems to work really well! I had one case where this happened:
Found archive files not processed by direct unpack, unpacking all files again
Would it be possible to add an option to skip this second unpack and just move (or delete) the remaining RARs?

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Unpack during downloading

Post by hugbug » 22 May 2017, 07:35

That's a safety check which detects if something goes wrong during direct unpack. It prevents loosing of data.

The check should detect files renamed during rar-rename stage if par-rename couldn't rename them. This is a rare case but I know some uploaders obfuscate files before creating par-sets. In such case par-rename and direct-rename do not work; direct unpack doesn't work either.

You may also have this message if direct rename isn't active but direct unpack is.

In both cases the default unpack must process files. Skipping it isn't a good idea.

Nonetheless maybe there is a bug to fix. Can you please send me the nzb-file for inspection (and also the log-file of this nzb download)?

salami
Posts: 68
Joined: 14 Apr 2014, 11:09
Location: Switzerland

Re: [New Feature] Unpack during downloading

Post by salami » 22 May 2017, 08:11

Sure, I have just sent you an e-mail with all the details.

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests