Page 1 of 8

[Queue-Script] FakeDetector - early detection of fake videos

Posted: 17 Aug 2014, 23:25
by hugbug
This is a queue-script which is executed during download, after every downloaded file containing in nzb-file (typically a rar-file). The script lists content of download rar-files and tries to detect fake nzbs.

What is considered fake
As a fake is considered an nzb containing both a media file (typically video) and an executable (typically codec.exe or vlc.exe).

Action
If a fake is detected the download is aborted and marked as bad and processed by duplicate handling (if active). All pp-scripts are executed as well. If option DeleteCleanupDisk is active in NZBGet the downloaded files are deleted.

How to use
You need NZBGet 14.0 or newer.
Select the script as a post-processing script in option PostScript (global or for a category).

Authors
This script is written by a joined force of forum members including:
  • Clinton Hall;
  • JVM;
  • hugbug (me).
License
GPLv2 as usual. Feel free to enhance and share.

Info
Home page and source code: https://github.com/nzbget/FakeDetector.

Download
See FakeDetector releases page.

More info on fake detection
See discussion in topic Fake detection.

Re: [Queue-Script] FakeDetector - early detection of fake vi

Posted: 19 Aug 2014, 20:41
by hugbug
Version 1.3:
A big speed improvement when processing NZB_ADDED-event. That's where the script reorders files for sooner detection.

Download
See the first post of the topic.

The whole story
Tests on a slow linux box showed a slow processing of the event "NZB_ADDED". This was strengthen by the fact that the queue-script is executed simultaneously with download, which alone takes all CPU time on the machine. As a result the script required 15 seconds to process the event - this is for a relatively small nzb-file (1GB, about 60 inner files). Then I did a test with a season pack nzb containing over 1000 files. The script were working 10 minutes. Even if download was paused during script execution the script still required one minute. Unacceptable.

The problem was in xmlrpc-module, which apparently is very slow in python.

Reworked the code to use JSON instead - 1m40s - much better.

But still not good enough. Rewritten the code using low level string parsing - 10 seconds, from which 4 seconds is the script boot time. That's on a big nzb. Small nzbs are processed almost immediately. If only python could boot faster.

Summary: execution time reduced from 10 minutes to 10 seconds.

Re: [Queue-Script] FakeDetector - early detection of fake vi

Posted: 19 Aug 2014, 21:39
by prinz2311
Did I overlook something or does the sorting algo only sort rar's with the new naming and not the old ones (.rar, .r00, r01, ...,.s00,....) ?

Re: [Queue-Script] FakeDetector - early detection of fake vi

Posted: 19 Aug 2014, 22:11
by hugbug
Yes, it sorts only the new scheme. All my example fakes (about 80) fall into this category. If you saw fakes with the old naming scheme the script can be extended for it as well.

Re: [Queue-Script] FakeDetector - early detection of fake vi

Posted: 20 Aug 2014, 11:36
by get39678
I had to upgrade to r1098 (from r1094) before the new 1.3 script would work. If you do not NZBGet displays communication errors and finally becomes unresponsive.

Re: [Queue-Script] FakeDetector - early detection of fake vi

Posted: 20 Aug 2014, 11:42
by hugbug
get39678 wrote:If you do not NZBGet displays communication errors and finally becomes unresponsive.
It's not because of the script but because of an unrelated bug fixed in r1096.

Re: [Queue-Script] FakeDetector - early detection of fake vi

Posted: 22 Aug 2014, 21:27
by prinz2311
BTW if a download gets deleted from the queue (including as dupe) the tmp file stays in the tmp folder forever if I'm not mistaken.

Re: [Queue-Script] FakeDetector - early detection of fake vi

Posted: 23 Aug 2014, 10:19
by get39678
Yes there is something not quite correct as I get similar things with the PasswordDetector also. I have 0 byte temporary files being left behind only under certain conditions.

Re: [Queue-Script] FakeDetector - early detection of fake vi

Posted: 24 Aug 2014, 03:33
by JVM
FakeDetector doesn't delete temp files of bad downloads because it exits in post processing mode due to os.environ.get('NZBPP_STATUS') == 'FAILURE/BAD' or os.environ.get('NZBPP_TOTALSTATUS') == 'FAILURE' etc before clean_up() can be called. Need to check if the script is being used as a pp script and delete temp file before other mark bad or failure checks.

Re: [Queue-Script] FakeDetector - early detection of fake vi

Posted: 24 Aug 2014, 15:50
by prinz2311
I think the problem is also something else: If I'm not mistaken pp-scripts aren't called when Deleted manually or as Dupe, only Delete Health calls them. So there is no way for the script to delete the tmp file since it's not called. So a new event "NZBNA_EVENT" for queue scripts like NZB_DELETED that just like NZB_DOWNLOADED will always be called in case the nzb is deleted from the queue could fix this.