New feature: delayed par-check/repair

Get help, report and discuss bugs.
hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

New feature: delayed par-check/repair

Post by hugbug » 18 Oct 2008, 19:34

This feature was discussed on forum.

The default processing scenario is: download -> parcheck -> postprocess.
With delayed parcheck a new scenario is possible: download -> postprocess -> parcheck -> postprocess.

This allows to try unrar first and only if it fails do par-check/repair, then unrar again.

This is achieved by returning a special exit code from post-process-script to request nzbget to do par-check/repair. The value is "28161" (hex 0x6E01 which means "n1" or "nzbget return code 1"). All other return codes are currently ignored, but for future compatibility use value "0", which is a common default result code).

To organize delayed parcheck:
  1. set options: LoadPars=one, ParCheck=no, ParRepair=yes;
  2. in post-process-script check the parameter "result of par-check". If it is "0", that means, the script is called the first time. Try to unpack files. If unpack fails, exit the script with magic return value;
  3. nzbget will start par-check/repair. After that it calls the script again;
  4. on second pass the parameter "result of par-check" will have value greater than "0". If it is "2" ("checked and successfully repaired") you can try unpack again.
I know, ydrol has implemented delayed parcheck in his comprehensive unpack-script. But to do that, he has programmed a very complex wrapper for external par2. With support in nzbget, the implementation of delayed parcheck in post-process-script is trivial.

Implemented in revision 242.

dalrun
Posts: 170
Joined: 09 Sep 2008, 17:12
Contact:

Re: New feature: delayed par-check/repair

Post by dalrun » 19 Oct 2008, 16:03

FWIW, PPWeb has always had delayed par check/repair (and categories and nzb zip extraction).

How are you handling multiple collection nzb's?
Do all collections get checked/repaired with your delayed par check?

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: New feature: delayed par-check/repair

Post by hugbug » 19 Oct 2008, 17:52

dalrun wrote:FWIW, PPWeb has always had delayed par check/repair (and categories and nzb zip extraction).
I know that PPWeb has builtin par-check/repair, but I didn't know if it can try unrar first (it is simple of cause as long as the parcheck is already implemented in script). Thanks for mention this.
dalrun wrote:How are you handling multiple collection nzb's?
Do all collections get checked/repaired with your delayed par check?
They are handled the same way as by standard parcheck. They should be checked properly, althouth I didn't test it yet :)

dalrun
Posts: 170
Joined: 09 Sep 2008, 17:12
Contact:

Re: New feature: delayed par-check/repair

Post by dalrun » 19 Oct 2008, 20:26

hugbug wrote:I didn't know if it can try unrar first
Basically, PPWeb almost always tries unrar first. The only exceptions (that I can think of) are when PPWeb is configured to do the check/repair and there's a _brokenlog.txt or its a split avi.
hugbug wrote:They are handled the same way as by standard parcheck.
So they all get checked, i.e. a single collection cannot be sent back for repair?

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: New feature: delayed par-check/repair

Post by hugbug » 19 Oct 2008, 21:08

dalrun wrote:So they all get checked
Yes. There is no way to say nzbget what collection to repair. Remember, if parcheck is disabled nzbget calls pp-script only once (not for every collection). The collection detection is based on par2-names and occurs during parcheck.

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: New feature: delayed par-check/repair

Post by hugbug » 20 Oct 2008, 07:58

hugbug wrote:if parcheck is disabled nzbget calls pp-script only once (not for every collection).
I think I can improve it. If pars were downloaded (option "LoadPars" is set to "one" or "all"), nzbget can detect collections even if parcheck is disabled and call pp-script for each par-collection. Then we can have two result codes:
1) code "repair current collection";
2) code "repair all collections".
The first code is supposed to be used by smart pp-scripts, which unrar only current collection. The second code is for simple scripts, which unrar all collections (they should unrar only if parameter "state of nzb-job" is set to "this is last collection").

To prevent multiple parchecks nzbget should accept code "repair all" only if it is returned by pp-script called for the last collection and only if there were no codes "repair current" returned for other collections for the same nzb-file. Or nzbget could check what collection were already repaired. Actually all that checks will probably just add unnecessary complexity to nzbget, because the script should either always process individual collections (smart script) or always process all of them (simple script) but not both.

Another issue. If pp-script performs par on its own (making usage of option AllowReProcess), calling the script for each collection is probably not a good idea. I think if option "AllowReProcess" is enabled, nzbget should call pp-script only once (not for each collection), exactly as it works now.

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: New feature: delayed par-check/repair

Post by hugbug » 21 Oct 2008, 22:28

hugbug wrote: I think I can improve it. If pars were downloaded (option "LoadPars" is set to "one" or "all"), nzbget can detect collections even if parcheck is disabled and call pp-script for each par-collection. Then we can have two result codes:
1) code "repair current collection";
2) code "repair all collections".
The first code is supposed to be used by smart pp-scripts, which unrar only current collection. The second code is for simple scripts, which unrar all collections (they should unrar only if parameter "state of nzb-job" is set to "this is last collection").
Implemented in revision 243.
hugbug wrote: To prevent multiple parchecks nzbget should accept code "repair all" only if it is returned by pp-script called for the last collection and only if there were no codes "repair current" returned for other collections for the same nzb-file. Or nzbget could check what collection were already repaired. Actually all that checks will probably just add unnecessary complexity to nzbget, because the script should either always process individual collections (smart script) or always process all of them (simple script) but not both.
The check marked in red was implemented.
hugbug wrote: Another issue. If pp-script performs par on its own (making usage of option AllowReProcess), calling the script for each collection is probably not a good idea. I think if option "AllowReProcess" is enabled, nzbget should call pp-script only once (not for each collection), exactly as it works now.
Not yet implemented. The pp-script is now called multiple times, once for each collection. That can cause problems if the script doesn't test parameter "state of nzb-job", assuming it is called once if parcheck is disabled (as in previous versions).
Dave, I can call the script only once as suggested in previous post to provide better compatibility with your existing script. Or you can add the check for parameter "state of nzb-job" if the script doesn't test it already. What you prefer?

Note: There were made many changes in code for managing of post-jobs. I tried to test all combinations. Please let me know if you find any issues, even (and especially) with existing scripts.

dalrun
Posts: 170
Joined: 09 Sep 2008, 17:12
Contact:

Re: New feature: delayed par-check/repair

Post by dalrun » 22 Oct 2008, 17:20

hugbug wrote:The pp-script is now called multiple times, once for each collection
Excellent, that has been on my wish list for some time :).
hugbug wrote:Another issue. If pp-script performs par on its own (making usage of option AllowReProcess), calling the script for each collection is probably not a good idea. I think if option "AllowReProcess" is enabled, nzbget should call pp-script only once (not for each collection), exactly as it works now.
I'm not sure what your concern is. I post process each collection individually. If the collection that I'm processing is broken, I use reprocess to fetch only the pars needed to repair that collection. For me, multiple postprocess and reprocess calls go hand in hand.

At the first 'unrar collection' error I 'kill 9 unrar' and start a parcheck on that collection. With the new feature I could send a "repair current collection" instead.

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: New feature: delayed par-check/repair

Post by hugbug » 22 Oct 2008, 20:44

dalrun wrote: I'm not sure what your concern is.
...
If the collection that I'm processing is broken, I use reprocess to fetch only the pars needed to repair that collection.
After the fetching of additional pars nzbget calls the script mutliple times again - once for each collection. It's because the event "nzb-file completed" triggers again for nzb-file. I wanted to highlight the changed behaviour.

It would be great if you adjust your script to use the new feature (because I like the progress-indication in internal parchecker ;)).
But in a case you'll keep using external par2 you might need to add a check for parameter "state of nzb-job". The script should be aware of that it can be called multiple times after reprocess.

So actually the solely puprose of option "AllowReProcess" is to be used with a script which makes par-check via external par2 and unpauses items in queue. Since that (very smart) script does not use nzbget's collection detection, it could make sense to disable multicalls. Just a thought.

dalrun
Posts: 170
Joined: 09 Sep 2008, 17:12
Contact:

Re: New feature: delayed par-check/repair

Post by dalrun » 25 Oct 2008, 00:16

hugbug wrote:This is achieved by returning a special exit code from post-process-script to request nzbget to do par-check/repair. The value is "28161" (hex 0x6E01 which means "n1" or "nzbget return code 1"). All other return codes are currently ignored, but for future compatibility use value "0", which is a common default result code).
I don't get it, exit codes are 1-255. Anything bigger and they get hacked up (e.g. 28161 = 1).

Post Reply

Who is online

Users browsing this forum: No registered users and 51 guests