[New Feature] Unpack during downloading

Discuss newly added features or request new features.
neilt0
Posts: 261
Joined: 10 Sep 2008, 19:11

Re: [New Feature] Unpack during downloading

Post by neilt0 » 31 May 2017, 07:28

Still loving this feature. However, if you have fast download speeds or the individual NZBs are small enough, nzbget now no longer respects PostStrategy: Sequential. It will keep unraring multiple NZBs simultaneously, in fact, it'll go even past the Aggressive strategy:

Image

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Unpack during downloading

Post by hugbug » 31 May 2017, 08:00

Yes, because direct unpack is not part of post-processing and isn't controlled by post-processor.

I could still probably do something to make them play together better:
  1. Direct unpack should start as soon as the first archive part is downloaded - regardless of what is currently post-processed. Just like it works now. Why so?
    • If no new archive parts come in the unpack is waiting and no resources are spent (only a little bit by waiting unrar process).
    • If you reorder downloads or change priorities or new download with higher priority is added to queue - the new nzb will be downloaded and directly unpacked. Waiting direct unpacking jobs from non-active downloads should not prevent unpacking of active job.
  2. When all files are downloaded the post-processor waits for direct unpack to complete before it can start post-processing that item.
    • What happens in your case is that multiple items are completed and all are being unpacked and post-processor waits for all of them. For post-processor there are no active post-processing jobs at all and the post-processing strategy is irrelevant at this moment.
    • Possible improvement: when direct unpack enters state "all parts are downloaded" it should start respecting post-processing strategy.
    • Although we cannot pause unrar at arbitrary moments we still can pause unpacking at archive part borders where NZBGet communicates with unrar to send it a command to continue with the next part.
    • Possible improvement: when direct unpack processes the next archive part it should check how many unrar's are actively unpacking at this moment (respecting PostStrategy) and wait if necessary. However if there are many direct unpacks they should synchronize with each other to ensure that the job with higher priority or higher queue order is processed first.
Thanks for your feedback.

neilt0
Posts: 261
Joined: 10 Sep 2008, 19:11

Re: [New Feature] Unpack during downloading

Post by neilt0 » 31 May 2017, 08:07

Thanks, hugbug. Those look like sensible ideas.

Cheers,

Neil.

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Unpack during downloading

Post by hugbug » 06 Jun 2017, 16:27

Update r2013:
  • PostStrategy is now respected by direct unpack. This should prevent creating of too many simultaneous unpack jobs.
@neilt0: I've sent you the latest develop version to try.

neilt0
Posts: 261
Joined: 10 Sep 2008, 19:11

Re: [New Feature] Unpack during downloading

Post by neilt0 » 07 Jun 2017, 12:28

Tested and working. Thanks!

rxp
Posts: 55
Joined: 07 Mar 2014, 18:51

Re: [New Feature] Unpack during downloading

Post by rxp » 23 Jun 2017, 05:25

I've been downloading hundreds of files recently and it's working perfectly!

Great job!

b0mb
Posts: 8
Joined: 26 Apr 2016, 07:57

Re: [New Feature] Unpack during downloading

Post by b0mb » 29 Jun 2017, 04:41

Hi!

Fist big thx for this nice new feature which really speeds up processes a lot ;)

But i have a problem with several BIG download. About 30 - 60 GiG´s of data stored into about 200 .rar files.

Everything works fine and i can also see that the files are unpacked during download in the /unpack subfolder but after the process is successfully finished the directory is empty.

I´ve also tried some single file downloads from 2 - 12 GiG Size where this did not happen.

with direct unpack

Code: Select all

Statistics:
Downloaded size: 59.84 GB
Average download speed: 49.50 MB/s
Total time: 0:37:52
Download time: 0:20:38
Verification time: 0:00:18
Repair time: 0:00:00
Unpack time: 0:16:37

Files:
<no files found in the destination directory (moved by a script?)>
without direct unpack

Code: Select all

Statistics:
Downloaded size: 59.84 GB
Average download speed: 47.46 MB/s
Total time: 0:54:44
Download time: 0:21:31
Verification time: 0:00:17
Repair time: 0:00:00
Unpack time: 0:32:36

Files:
So no files listed but everything is at it´s place without using direct unpack.

Does any1 have an idea?

I´m running nbzget from linuxserver.io latest docker container. As i am running the stabled build and V19 should be updated tomorrow or on saturday i´ve just updated nzbget within the docker container which is normally not causing any problems. ;)

Thx for help in advance!

btw the only post processing script i am running is the eMail script...

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Unpack during downloading

Post by hugbug » 29 Jun 2017, 06:31

Everything is logged. Please post the log of this nzb - click on history item, then button "Log", then button "Save to file" (via pastebin.com).

If no errors appear in log I would suppose that some other program has moved the files elsewhere? For example a media library scanner. For bigger downloads the chances are higher for that, since the unpack takes longer.

STRYKER
Posts: 12
Joined: 30 Jul 2013, 13:08

Re: [New Feature] Unpack during downloading

Post by STRYKER » 29 Jun 2017, 08:04

I have the same Problem on Synology NAS.
Directory is blank after finishing. Files are nowhere else on the server.
Smaller downloads no problems.
LOG gives no information what happened with the files.

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Unpack during downloading

Post by hugbug » 29 Jun 2017, 09:04

STRYKER wrote:
29 Jun 2017, 08:04
LOG gives no information what happened with the files.
May I have a look?

Also linuxserver.io docker container?

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests