[Queue-Script] FakeDetector - early detection of fake videos

Share your scripts or request scripts with specific features.
Forum rules
Please keep the forum clean - one topic per script. Questions not related to a specific script should be posted in Support forum.
Post Reply
hugbug
Developer & Admin
Posts: 6700
Joined: 09 Sep 2008, 11:58
Location: Germany (NRW)

[Queue-Script] FakeDetector - early detection of fake videos

Post by hugbug » 17 Aug 2014, 23:25

This is a queue-script which is executed during download, after every downloaded file containing in nzb-file (typically a rar-file). The script lists content of download rar-files and tries to detect fake nzbs.

What is considered fake
As a fake is considered an nzb containing both a media file (typically video) and an executable (typically codec.exe or vlc.exe).

Action
If a fake is detected the download is aborted and marked as bad and processed by duplicate handling (if active). All pp-scripts are executed as well. If option DeleteCleanupDisk is active in NZBGet the downloaded files are deleted.

How to use
You need NZBGet 14.0 or newer.
Select the script as a post-processing script in option PostScript (global or for a category).

Authors
This script is written by a joined force of forum members including:
  • Clinton Hall;
  • JVM;
  • hugbug (me).
License
GPLv2 as usual. Feel free to enhance and share.

Info
Home page and source code: https://github.com/nzbget/FakeDetector.

Download
See FakeDetector releases page.

More info on fake detection
See discussion in topic Fake detection.

hugbug
Developer & Admin
Posts: 6700
Joined: 09 Sep 2008, 11:58
Location: Germany (NRW)

Re: [Queue-Script] FakeDetector - early detection of fake vi

Post by hugbug » 19 Aug 2014, 20:41

Version 1.3:
A big speed improvement when processing NZB_ADDED-event. That's where the script reorders files for sooner detection.

Download
See the first post of the topic.

The whole story
Tests on a slow linux box showed a slow processing of the event "NZB_ADDED". This was strengthen by the fact that the queue-script is executed simultaneously with download, which alone takes all CPU time on the machine. As a result the script required 15 seconds to process the event - this is for a relatively small nzb-file (1GB, about 60 inner files). Then I did a test with a season pack nzb containing over 1000 files. The script were working 10 minutes. Even if download was paused during script execution the script still required one minute. Unacceptable.

The problem was in xmlrpc-module, which apparently is very slow in python.

Reworked the code to use JSON instead - 1m40s - much better.

But still not good enough. Rewritten the code using low level string parsing - 10 seconds, from which 4 seconds is the script boot time. That's on a big nzb. Small nzbs are processed almost immediately. If only python could boot faster.

Summary: execution time reduced from 10 minutes to 10 seconds.

prinz2311
Posts: 460
Joined: 08 Dec 2012, 00:03

Re: [Queue-Script] FakeDetector - early detection of fake vi

Post by prinz2311 » 19 Aug 2014, 21:39

Did I overlook something or does the sorting algo only sort rar's with the new naming and not the old ones (.rar, .r00, r01, ...,.s00,....) ?

hugbug
Developer & Admin
Posts: 6700
Joined: 09 Sep 2008, 11:58
Location: Germany (NRW)

Re: [Queue-Script] FakeDetector - early detection of fake vi

Post by hugbug » 19 Aug 2014, 22:11

Yes, it sorts only the new scheme. All my example fakes (about 80) fall into this category. If you saw fakes with the old naming scheme the script can be extended for it as well.

get39678
Posts: 193
Joined: 09 Jun 2014, 10:49

Re: [Queue-Script] FakeDetector - early detection of fake vi

Post by get39678 » 20 Aug 2014, 11:36

I had to upgrade to r1098 (from r1094) before the new 1.3 script would work. If you do not NZBGet displays communication errors and finally becomes unresponsive.

hugbug
Developer & Admin
Posts: 6700
Joined: 09 Sep 2008, 11:58
Location: Germany (NRW)

Re: [Queue-Script] FakeDetector - early detection of fake vi

Post by hugbug » 20 Aug 2014, 11:42

get39678 wrote:If you do not NZBGet displays communication errors and finally becomes unresponsive.
It's not because of the script but because of an unrelated bug fixed in r1096.

prinz2311
Posts: 460
Joined: 08 Dec 2012, 00:03

Re: [Queue-Script] FakeDetector - early detection of fake vi

Post by prinz2311 » 22 Aug 2014, 21:27

BTW if a download gets deleted from the queue (including as dupe) the tmp file stays in the tmp folder forever if I'm not mistaken.

get39678
Posts: 193
Joined: 09 Jun 2014, 10:49

Re: [Queue-Script] FakeDetector - early detection of fake vi

Post by get39678 » 23 Aug 2014, 10:19

Yes there is something not quite correct as I get similar things with the PasswordDetector also. I have 0 byte temporary files being left behind only under certain conditions.

JVM
Posts: 83
Joined: 21 Oct 2013, 05:11

Re: [Queue-Script] FakeDetector - early detection of fake vi

Post by JVM » 24 Aug 2014, 03:33

FakeDetector doesn't delete temp files of bad downloads because it exits in post processing mode due to os.environ.get('NZBPP_STATUS') == 'FAILURE/BAD' or os.environ.get('NZBPP_TOTALSTATUS') == 'FAILURE' etc before clean_up() can be called. Need to check if the script is being used as a pp script and delete temp file before other mark bad or failure checks.

prinz2311
Posts: 460
Joined: 08 Dec 2012, 00:03

Re: [Queue-Script] FakeDetector - early detection of fake vi

Post by prinz2311 » 24 Aug 2014, 15:50

I think the problem is also something else: If I'm not mistaken pp-scripts aren't called when Deleted manually or as Dupe, only Delete Health calls them. So there is no way for the script to delete the tmp file since it's not called. So a new event "NZBNA_EVENT" for queue scripts like NZB_DELETED that just like NZB_DOWNLOADED will always be called in case the nzb is deleted from the queue could fix this.

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests