[New Feature] Smart duplicates

Discuss newly added features or request new features.
prinz2311
Posts: 466
Joined: 08 Dec 2012, 00:03

Re: [New Feature] Smart duplicates

Post by prinz2311 » 19 Sep 2013, 09:19

hugbug wrote:
  1. Use two filter lines:

    Code: Select all

    A: dexter season:=2 episode:>11
    A: dexter season:>2
    This one can be used right now.
Yes that i overlooked yesterday.
Other two alternatives are possible feature extensions:


[*]add support for OR. This would also require grouping (braces):

Code: Select all

A: dexter (season:=2 episode:>11) | season:>2
[*]add new search field similar to SAB with an appropriate built-in logic:

Code: Select all

A: dexter SxxEyy:>S02E11
[/list]

New field SxxEyy is the easiest alternative to use. "OR and braces" would extend filter capabilities.
This doesn't mean that only one alternative should be implemented. Both make sense. "OR and braces" are probably a lot of work.
I don't think that "OR and braces" are necessary. And if SxxEyy really is need i can't tell, it would maybe for beginners a more intuitive way to do it.
As for dated shows the title parsing is required since newznab fields are very unreliable here. A possible new field could be epdate and it must have built-in logic to allow such things:

Code: Select all

A: daily show epdate:>2013-09-01
Do you have more examples of how date is formatted in title? Is a reliable parsing possible at all?
Most use 2013.09.19 or 2013-09-19 ... Some use 19.09.2013 , also with other chars like: - as delimiter. In rare case I have seen 13.09.19 too. I haven't seen any that had month and day switched, only dates forward and backward.

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Smart duplicates

Post by hugbug » 20 Sep 2013, 21:30

Update: added functions Mark as Bad and Mark as Good for history items:
  • when a history item having success-status is marked as bad:
    • it is considered as failure by any duplicate check performed later;
    • if history has duplicates with dupe-status (dupe-backups) they are all moved (as paused) to download queue and one of them (with the highest duplicate score) is unpaused (downloaded);
  • when a history item is marked as good:
    • it is considered as success by any duplicate check performed later;
    • no other duplicates will be added to history as dupe-backups anymore;
    • if history has duplicates with dupe-status (dupe-backups) they are all removed from recent history (moved to dup-history);

prinz2311
Posts: 466
Joined: 08 Dec 2012, 00:03

Re: [New Feature] Smart duplicates

Post by prinz2311 » 21 Sep 2013, 11:21

hugbug wrote:Update: added functions Mark as Bad and Mark as Good for history items:
  • when a history item having success-status is marked as bad:
    • it is considered as failure by any duplicate check performed later;
    • if history has duplicates with dupe-status (dupe-backups) they are all moved (as paused) to download queue and one of them (with the highest duplicate score) is unpaused (downloaded);
Does this work logically in the following 2 cases:

Filter:

Code: Select all

o(s:100): 720p HDTV
o(s:200): 720p web-dl
o(s:350): 1080p web-dl
A: TV Show
nzbget has downloaded for all 3 quality a release over time.

Case 1:
User marks the 720p HDTV and/or web-dl as Bad. (In this case nzbget should do nothing besides mark it as bad since a download with higher dupescore has already been downloaded)

Case2:
User marks the 1080p web-dl as Bad. (In this case nzbget should only add releases that have a higher dupescore as the already successfully downloaded 720p web-dl)

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Smart duplicates

Post by hugbug » 21 Sep 2013, 12:48

In both cases if there any dupe-backups they all will be moved to queue and one will be downloaded.
There is no check if there are any other duplicates with success status for the marked item.
I'll fix this.

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Smart duplicates

Post by hugbug » 23 Sep 2013, 20:36

Update:
changed option "nodupe (yes, no)" to "dupemode (score, all, force)" in RSS filter commands "Append" and "Options"
  • Score - download duplicates with higher scores. Item is skipped if a duplicate marked as "good" exists in history. This is default.
  • All - download all duplicates regardless of scores. Item is skipped if a duplicate marked as "good" exists in history. This can be useful if you want download all releases and then manually check if there is a good one among them.
  • Force - treat item as unique, ignore all duplicate checks. This is mainly for usage in Add file dialog but can be used in rss filters too.
TODO:
  • problem described in case 1 and case 2 in previous posts regarding "Mark as bad";
  • filtering of daily shows - field epdate or similar.

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Smart duplicates

Post by hugbug » 24 Sep 2013, 09:24

hugbug wrote:filtering of daily shows - field epdate or similar.
I don't think anything is needed for dated (daily) tv shows:
  • They are posted immediately and there are no reposts. Meaning you don't need to filter for air date.
  • The don't have duplicates since they are posted by one poster only.
  • If you concern about duplicates of the same post coming from different rss providers this should be handled by nzb content check.
If I'm wrong, do you have examples (of duplicates or reposts) to look at?

prinz2311
Posts: 466
Joined: 08 Dec 2012, 00:03

Re: [New Feature] Smart duplicates

Post by prinz2311 » 24 Sep 2013, 10:38

Example: https://nzbindex.com/search/?q=jimmy+fa ... m=1&more=1

The same episode (and quality) are posted by two posters. one in a.b.teevee and one in a.b.town

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Smart duplicates

Post by hugbug » 24 Sep 2013, 11:08

Thanks. Then duplicate detection is indeed helpful. I've built this filter:

Code: Select all

A(k:tv=Jay.Leno-${1}): Jay Leno $([0-9]+\.[0-9]+\.[0-9]+)
It produces dupekeys like "tv=Jay.Leno-2013.09.20". Seems to work well. A problem may occur if two posters use different date formats.

Still don't see if a filtering for air dates is necessary.

prinz2311
Posts: 466
Joined: 08 Dec 2012, 00:03

Re: [New Feature] Smart duplicates

Post by prinz2311 » 24 Sep 2013, 11:30

With the reference in dupekey, i don't see at the moment a need for a filter. But maybe someone knows a something that uses dates and is reposted later. I don't have anything with dates in my list.

prinz2311
Posts: 466
Joined: 08 Dec 2012, 00:03

Re: [New Feature] Smart duplicates

Post by prinz2311 » 24 Sep 2013, 11:52

hugbug wrote:TODO:
problem described in case 1 and case 2 in previous posts regarding "Mark as bad";
I have case 3 for this:

Same Filters:

Code: Select all

o(s:100): 720p HDTV
o(s:200): 720p web-dl
o(s:350): 1080p web-dl
A: TV Show
nzbget has downloaded the first two quality's (dupescores) and the third one is in the download queue or downloading/postprocessing.

Now the user marks the 720p web-dl (which is already downloaded) as Bad. Here is the problem that a better quality is already in the download/postprocessing queue but it's not yet known if it will succeed.

Post Reply

Who is online

Users browsing this forum: No registered users and 13 guests