RSS Dupe Help?

Get help, report and discuss bugs.
douche
Posts: 47
Joined: 08 Feb 2014, 23:50

RSS Dupe Help?

Post by douche » 12 Jul 2018, 19:35

So with the help of other posts here on the boards, I created this RSS filter, that works, for the most part:

Code: Select all

# Propers
Options(dupescore+:1000):   internal
Options(dupescore+:2000):   repack
Options(dupescore+:4000):   proper
Options(dupescore+:8000):   real

# Dont grab dupes
Reject: dupestatus:success dupescore:<1000 | dupestatus:queued | dupestatus:downloading

# Accept all that passed above
Accept: *specific release groups I want*
But then what happens is I end up with a flood of dupes when NZBGet deletes/hides them from history, so I changed it to the following:

Code: Select all

# Propers
Options(dupescore+:1000):   internal
Options(dupescore+:2000):   repack
Options(dupescore+:4000):   proper
Options(dupescore+:8000):   real

# Dont grab dupes
Reject: dupestatus:success dupescore:<1000 | dupestatus:queued | dupestatus:downloading | age:>28d

# Accept all that passed above
Accept: *specific release groups I want*
and that works

Now that's all well and dandy, but NZBGet or this filter currently treats releases the same, whether they use " " (spaces), "." (periods), "_" (underscores), or whatever other symbol groups are using, as different releases.
For example:
This.One.Release.Year.MediaType.VideoCodec.AudioCodec-Group (1.17GB)
is treated as a completely different release as:
This_One_Release_Year_MediaType_VideoCodec_AudioCodec-Group (1.17GB)
or even:
This One Release Year MediaType VideoCodec AudioCodec-Group (1.17GB)

Is there a way to combat this?

Which also brings me to another related issue.
Currently the only way to deal with those dupe releases is to manual delete them as dupes, but then they can end-up re-downloaded using the filter above, so I have to then mark them as 'success' in history, in order to not download/requeue again.

Hopefully this makes sense, otherwise I can try to clarify.

Thank you!

hugbug
Developer & Admin
Posts: 7070
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: RSS Dupe Help?

Post by hugbug » 12 Jul 2018, 19:51

douche wrote:
12 Jul 2018, 19:35
NZBGet or this filter currently treats releases the same, whether they use " " (spaces), "." (periods), "_" (underscores), or whatever other symbol groups are using, as different releases.
For example:
This.One.Release.Year.MediaType.VideoCodec.AudioCodec-Group (1.17GB)
is treated as a completely different release as:
This_One_Release_Year_MediaType_VideoCodec_AudioCodec-Group (1.17GB)
or even:
This One Release Year MediaType VideoCodec AudioCodec-Group (1.17GB)

Is there a way to combat this?
It's all about dupekey. The items must have the same dupekey. RSS feeds from good indexers have imdbid for movies or tvdbid for series. NZBGet uses them to generate dupekeys automatically. Otherwise you need to set dupekey in your filter manually. For details see documentation.

douche
Posts: 47
Joined: 08 Feb 2014, 23:50

Re: RSS Dupe Help?

Post by douche » 12 Jul 2018, 22:25

yeah, unfortunately not all indexers have the same content, and not all indexers (even 'good' ones) provide imdb info via RSS
'custom' dupekey looks more suited to TV than film, from the looks of it

hugbug
Developer & Admin
Posts: 7070
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: RSS Dupe Help?

Post by hugbug » 12 Jul 2018, 22:54

Most (or maybe even all) indexers run on newznab software or derivatives, which implements imdbid. Check if you include necessary parameters in RSS URL to request extended fields in RSS result, ask indexer admins for further help.

douche
Posts: 47
Joined: 08 Feb 2014, 23:50

Re: RSS Dupe Help?

Post by douche » 13 Jul 2018, 18:25

hugbug wrote:
12 Jul 2018, 22:54
Most (or maybe even all) indexers run on newznab software or derivatives, which implements imdbid. Check if you include necessary parameters in RSS URL to request extended fields in RSS result, ask indexer admins for further help.
There's no RSS parameters required for imdbid, on any of indexers I'm using.
I guess I could try contacting the admins, but I'm not going to get my hopes up.


But with NZBGet, if I delete something as a dupe from queue, it then goes into history as a "dupe", why does the RSS still grab it as 'new' - unless I mark it as a 'success' in history - there's no "dupestatus:dupe" only "dupestatus:success" ?

Thanks.

hugbug
Developer & Admin
Posts: 7070
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: RSS Dupe Help?

Post by hugbug » 13 Jul 2018, 18:40

douche wrote:
13 Jul 2018, 18:25
There's no RSS parameters required for imdbid, on any of indexers I'm using.
What do you mean? Do your feeds provide imdbid?
If they don't then you need a parameter. Try adding "&extended=1" (without quotes) to the feed URL. Test it in a browser first.

hugbug
Developer & Admin
Posts: 7070
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: RSS Dupe Help?

Post by hugbug » 13 Jul 2018, 18:49

Checking dupestatus in RSS filter is a feature for special cases. Most likely you don't need it. Let nzbget enqueue all matching items. If there is an item with success status nzbget will not download another one. It will add the item to history as dupe (which may be used later if necessary though).

The main (or maybe even only) reason for dupestatus check was to avoid fetching of nzbs from indexers with low limit on nzb fetches. This particular issue is addressed in v21 (not yet released) - https://github.com/nzbget/nzbget/issues/541. If you want to try it - send me a message to nzbget@gmail.com and I'll send you current development version, let me know your OS and CPU.

douche
Posts: 47
Joined: 08 Feb 2014, 23:50

Re: RSS Dupe Help?

Post by douche » 13 Jul 2018, 20:20

hugbug wrote:
13 Jul 2018, 18:40
douche wrote:
13 Jul 2018, 18:25
There's no RSS parameters required for imdbid, on any of indexers I'm using.
What do you mean? Do your feeds provide imdbid?
If they don't then you need a parameter. Try adding "&extended=1" (without quotes) to the feed URL. Test it in a browser first.
As is, most of them provide imdbid's on my RSS feeds, some not so reliably.
it's possible that the IMDB matching happens after I grab the nzb off the feed though, which still results in dupes.
I tried adding the "&extended=1" option to my feed urls, there was no obvious change, but we'll see if that makes any difference moving forward.

I also have a spotweb-based 'indexer' (as a catchall, as it SOMETIMES has unique content) that doesn't provide imdbid in their RSS feeds.

regardless, would it not makes sense for NZBGet to also dupecheck regardless of the space/delimiter used? or is there a reason NZBGet should NOT do this, that I'm clearly not getting?
hugbug wrote:
13 Jul 2018, 18:49
Checking dupestatus in RSS filter is a feature for special cases. Most likely you don't need it. Let nzbget enqueue all matching items. If there is an item with success status nzbget will not download another one. It will add the item to history as dupe (which may be used later if necessary though).

The main (or maybe even only) reason for dupestatus check was to avoid fetching of nzbs from indexers with low limit on nzb fetches. This particular issue is addressed in v21 (not yet released) - https://github.com/nzbget/nzbget/issues/541. If you want to try it - send me a message to nzbget@gmail.com and I'll send you current development version, let me know your OS and CPU.
yes, I have dupestatus in there because some of my indexers have low limits on nzb fetches - I had a previous thread here similar to that other user.

my point is that if an NZB job is listed in the history as a 'dupe' - and NZBGet sees it in the RSS again, it currently will grab it again for whatever reason - but if I set the job as a 'success' in history, it ignores it.
Is that addressed in v21?
Last edited by douche on 13 Jul 2018, 20:26, edited 1 time in total.

hugbug
Developer & Admin
Posts: 7070
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: RSS Dupe Help?

Post by hugbug » 13 Jul 2018, 21:27

douche wrote:
13 Jul 2018, 20:20
my point is that if an NZB job is listed in the history as a 'dupe' - and NZBGet sees it in the RSS again, it currently will grab it again for whatever reason - but if I set the job as a 'success' in history, it ignores it.
Is that addressed in v21?
That's how it supposed to work. Since you have not downloaded the title yet nzbget is doing its job trying to grab an nzb for that title. When you delete items from queue that doesn't make them successfully downloaded. However you can mark them as success or good and that status is respected by dupecheck.

hugbug
Developer & Admin
Posts: 7070
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: RSS Dupe Help?

Post by hugbug » 13 Jul 2018, 21:48

douche wrote:
13 Jul 2018, 20:20
regardless, would it not makes sense for NZBGet to also dupecheck regardless of the space/delimiter used?
It could help a little but wouldn't solve the problem because nzb names have a lot of extra words (video quality etc.). That's why nzbget have concept of dupekeys.

Indexers (the proper ones) use additional information to identify titles. They don't rely solely on nzb name, they download nfos and parse them, etc. Then provide movie identification via field imdbid.

If you can code you can try to identify titles from nzb names and build proper dupekeys by writing an RSS feed extension script. We can discuss this further if you are going to write such a script.

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests