please share your post processing scripts

Get help, report and discuss bugs.
nobody

RE: Proposal: call script only once for nzb-file

Post by nobody » 20 Mar 2008, 17:56

1 - It doesn't matter if the post processing script is only called once because current scripts are calling unrar with a wildcard, i.e. unrar will (try to) extract all collections... Nzbget calls PostProcess once no matter what (e.g. no par2's and no rar's).

2 - StrictParName=no seems to work. The par-name seldom matches the rar-name but par-check seems to know which rars belong to a particular par2.

$2 (nzb-name) - If its removal can be added to cleanup, I'd suggest sending just the the name. I need the path to remove it but have been concerned since reading that path cannot be counted on.

$3 (par2-name) - Currently I use this to detect nzbget joined split avi's. I've though about offloading par check/repair to remote post processing where I'd also want the par2-name. Since we already have path (destination dir), this could also be sent without a path.

$4 (par-check) - I currently use the current setup to provide informational messages about each collection and would need that information for optional remote par checking/repair.

$5 (nzb-state) - Needed for current messaging and when running a script that processes all collections at the same time. In collection by collection processing its needed to know when to handle anything not earlier accounted for, e.g. a collection that contained no pars.

I'm not sure I understand what's wrong with the current setup.

Dave

hugbug

RE: please share your post processing scripts

Post by hugbug » 20 Mar 2008, 19:21

>I'm not sure I understand what's wrong with the current setup.

Currently nzbget tries to call the script for each collection. But it does not work properly, because nzbget assumes that each collection has a par-set and it actually calls the script for each par-set (not for each collection).

Furthermore, nzbget passes the par-name to script, but this name does not always match the name of files in collection. The script cannot rely on par-name and this parameter cannot be used properly in the script.

That's why I wanted to make things simpler and call the script only once.

But if you have usage for "$3 (par2-name)" and "$4 (par-check) for each collection" (even if it is not reliable), I can leave all things as they are.

------------

>1 - It doesn't matter if the post processing script is only called
> once because current scripts are calling unrar with a wildcard, i.e.
>unrar will (try to) extract all collections...

It was exactly my concern. The simple scripts from this thread (the first post, for example) do not check the parameter $5 (nzb-state). But this is a vital mistake.


>2 - StrictParName=no seems to work. The par-name seldom matches
>the rar-name but par-check seems to know which rars belong to a particular par2.

nzbget does not know what rar-files belongs to par-set. The option "StrictParName" affects only the search for par-files with additional blocks (those names must match the name of main par-files, if the option is enabled).

------------

May be it make sense to add a new option "PostProcessEachParSet" with default value "no", which should be the best setting for most (simple) scripts. But for more comprehensive scripts, like yours, it could be enabled?

nobody

RE: please share your post processing scripts

Post by nobody » 21 Mar 2008, 06:19

I was on the wrong track with my earlier par2 comments (too much par2 r *.par2 * in the past). It's the contents of the par2-file that tells par2 which files to check. The passed par2 file-name doesn't need to match anything.

There is nothing wrong with single collection processing, every option works properly and is reliable.

I'm pretty sure the only potential issue is using par-check when processing multiple collections. While using unrar output offers more options than par-check, I do understand your desire to have par-check be a usable option when processing multiple collections.

hugbug

RE: please share your post processing scripts

Post by hugbug » 21 Mar 2008, 09:53

Yes, nzbget does not try to be too smart. Actually id does not try to detect collections. Instead it looks for main par-files, check them as a separate par-jobs, then call the script after each job is completed.

The only thing I wanted to change was - to call the script after the last par-job is completed and allow the script to be as smart as it want to find out what collections are available. Either by analysing of par-set-files or (better?) by parsing of rar-files.


nobody

RE: please share your post processing scripts

Post by nobody » 21 Mar 2008, 18:13

I understand the distinction between collections and par2-files. The only time there is a difference is when the number of collections and par2-files are unequal. To narrow that even further, a collection without a par2-file only makes a difference when it isn't the last collection.

It sounds like you want an option to call the script just once after after nzbget has done whatever it can with the collection(s). It cannot be after the last par-job because there may be no pars. Because there may be no pars or rars, you cannot count on parsing or analyzing either.

I'm not sure what you would gain. If you added a 'one or more collections contained no par-files' to par-check, how does that help?, you still have to send the collections through unrar. A 'no collections contained par-files' is the only thing that could be used to avoid running the collection(s) through unrar. I don't see any way to cover all the bases with a one time call to PostProcess.

...How about a different approach, what if nzbget sent the rar-name? With the rar-name, collections with par2-files could be processed as they come down the pipe.

if [ "$ParCheck" -eq 2 ]
nice -n19 unrar x -y "$RarName" >> $log

That would seem to solve all potential issues and remove the need for $ParFail... Maybe that would need to be the rar base-name and $RarName*... To prevent overlapping post processes, Nzbget would have to pause/wait for each PostProcess call to complete (not a bad idea in any scenario).

hugbug

RE: please share your post processing scripts

Post by hugbug » 22 Mar 2008, 10:38

>It sounds like you want an option to call the script just once
>after nzbget has done whatever it can with the collection(s).
>It cannot be after the last par-job because there may be no pars.

You are right, I mean - call the script after the last par-job is completed or (if there were no pars) after the last file is downloaded.


>I'm not sure what you would gain.

This would make the interface between nzbget and the script certain. No matter if all collections have par-sets or if there are no par-sets at all - the script is called once, at the end of download or check of nzb-file.

Pros:
- the script author does not make any assumptions what collections was checked with pars, based on parameters passed by script;
- no need to check the parameter "was last collection";

Cons(?):
- you can not distinguish between successful and failed par-sets. I believe this information is not very valuable anyway, because the name of par-set not always unambiguously match the name of collection-files. So it's better not to take this info into account at all.


>How about a different approach, what if nzbget sent the rar-name?

I don't want to add any collection-detection to nzbget. I want to entrust the script completely with this task.

For rar-archives the script can search for files like "*.part001.rar" and "*<no part>.rar" to detect the collections. It is better than rely on par-sets.


In a case if any of pars was failed for nzb-files, I would not extract any files at all. But the script can take a different approach, for example first try to extract the files, then if if fails - delete the partially extracted files.


>To prevent overlapping post processes, Nzbget would have to pause/wait
>for each PostProcess call to complete

It is already implemented. nzbget waits for script to complete before going to the next item in post-queue.

nobody

RE: please share your post processing scripts

Post by nobody » 22 Mar 2008, 22:23

The only thing you would gain under your proposal is the knowledge that one or more collections (in an nzb with multiple collections) had no par2-files. You could treat this just like one or more collections failing par-check, i.e. have a no-par2 argument and pass a 1 or 0 to the script.

Personally, I find the all or nothing approach too limiting and cannot imagine not processing because one or more collections had no par2-files. You seem to be underestimating the potential and power of the current options/setup.

If PostProcess is handling the par-check and/or repair, it doesn't care about rar-set names, i.e. ambiguity is not an issue and par2-name (as well as par-check and nzb-state) is valuable.

I have yet to need the rar-set base-name. I wrongly assumed that nzbget would know what it was. It would have been an elegant solution.

hugbug

RE: please share your post processing scripts

Post by hugbug » 23 Mar 2008, 09:46

Hi Dave,

I've checked out your script (1523_18_nzbgetpp.pl) and see, that even in this comprehensive script you not use the "call script for each par-set"-feature. You make all processing in the last call to the script.

The only place where you use mutlicalls is to add warnings/errors to log for failed par-check. But the same info is written to "_brokenlog.txt". You could just copy this info to your log, if you want.

This proves, that the idea of calling the script for each par-set is not working and is not used in any single script, even yours.

So why defend the non-working-feature? :)


hugbug

RE: please share your post processing scripts

Post by hugbug » 26 Mar 2008, 08:08

>http://dalrun.com/nzbgetpp/files/1523_1_test.pl

For the script to be called for each par-set the option "parcheck" must be enabled in nzbget.
This makes the second par-check in the script useless and proofs again that the current concept of multicalls just confuses.

Post Reply

Who is online

Users browsing this forum: No registered users and 54 guests