Proper gets marked as a Dupe and then gets deleted

Get help, report and discuss bugs.
Post Reply
kjKmcQN3
Posts: 75
Joined: 29 Aug 2015, 15:09

Proper gets marked as a Dupe and then gets deleted

Post by kjKmcQN3 » 06 Jun 2017, 12:21

I get nzb files pushed via RSS feeds and have set it up so that my provider pushes Propers/Repacks to me in case the need arises. I was surprised to see that when a Proper release for a show that I had successfully downloaded yesterday got pushed to me today it got marked as a Dupe and, even more surprisingly, got automatically deleted instead of going to the History as a Dupe/Good. How can I prevent this from happening in the future? Here are the logs that I pulled up.

Code: Select all

Tue Jun 6 14:49:55 2017 DETAIL Downloading Some.Show.PROPER.720p.HDTV.X264-UAV @ dognzb.cr
Tue Jun 6 14:49:57 2017 DETAIL Download Some.Show.PROPER.720p.HDTV.X264-UAV @ dognzb.cr completed
Tue Jun 6 14:49:57 2017 DETAIL Category TV > HD matched to SickRage for Some.Show.PROPER.720p.HDTV.X264-UAV.nzb
Tue Jun 6 14:49:57 2017 INFO Adding collection Some.Show.PROPER.720p.HDTV.X264-UAV.nzb to queue
Tue Jun 6 14:49:57 2017 WARNING Skipping duplicate Some.Show.PROPER.720p.HDTV.X264-UAV, found in history Some.Show.720p.WEB.h264-TBS with success status
Tue Jun 6 14:49:57 2017 INFO Deleting file /Users/abc/Library/Application Support/NZBGet/nzb/Some.Show.PROPER.720p.HDTV.X264-UAV.nzb.queued

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Proper gets marked as a Dupe and then gets deleted

Post by hugbug » 06 Jun 2017, 12:30

You can disable duplicate check via option DupeCheck=no if you want duplicates to be downloaded.

Or you can setup your RSS filters to handle duplicates in a way you like it. See RSS and duplicate check.

kjKmcQN3
Posts: 75
Joined: 29 Aug 2015, 15:09

Re: Proper gets marked as a Dupe and then gets deleted

Post by kjKmcQN3 » 06 Jun 2017, 15:21

I would like to keep duplicate check enabled. However, I would expect it to send items marked as duplicates to be sent to History as either Dupes/Good/Copy (as an aside, could you please point me to a page that explains these different statuses?), as described on the RSS and duplicate check page, instead of getting deleted with no indication. I happened to notice the file as I had seen it in my Bookmarks section on the website that it gets pushed from but I could have easily not known otherwise.


"Before an item is added to queue a check is performed whether a similar item is already queued or was downloaded before. If download queue contains the same title or history contains the title with status “success” the duplicate is not added to queue but instead put directly to history as backup (see later)."

The thoroughly detailed documentation pages are indicative of the amount of hard work that you have put into the project and the results speak for themselves. NZBGet is a real pleasure to use. Thank you.

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Proper gets marked as a Dupe and then gets deleted

Post by hugbug » 06 Jun 2017, 15:58

If a duplicate is found in history the new nzb is added to history with status DUPE or COPY. However if a duplicate exist in hidden history only (and not in visible history) the duplicate isn't added to history at all.

The reason is: let's assume you have an RSS for certain show. The same episodes are reposted again all the time. You would have your history constantly filled with DUPE-items for episodes you downloaded months or years before for which you don't really care anymore (they are long in your library or are watched).

With current behaviour the new aired episodes are added to queue and downloaded. Duplicates are added to history with DUPE status. After KeepHistory days (30 by default) the successfully downloaded nzb transits into hidden state. After that all new nzbs for the same episode will be ignored. Isn't that nice? ;) There is also a possibility to force ignoring to happen sooner than KeepHistory days. For that user can mark item in history as GOOD which for NZBGet means that the title isn't needed anymore and shouldn't be watched.

In your case I think it would be better to disable duplicate check for items coming from Bookmark feed. You can do that by setting dupemode=force in the filter of that feed:

Code: Select all

Accept(dupemode:force): *
Statuses:
  • Dupe - the nzb was deleted by duplicate check. Because there is another item in queue. The item with status "Dupe" maybe used automatically if the item which is in queue fails.
  • Good - the nzb was marked as good by user using command "Mark as good" in history dialog. Any new arrived duplicates will be ignored (not added to queue nor to history).
  • Copy - the nzb was deleted by duplicate check because this nzb-file already exists in download queue or in history. Here the nzb is the exactly same copy of another nzb. This status can occur only if you try to add the same nzb again. The exact content of nzb is compared here (via article IDs).
I have to review and update the documentation page for RSS as it seems to miss some infos.

kjKmcQN3
Posts: 75
Joined: 29 Aug 2015, 15:09

Re: Proper gets marked as a Dupe and then gets deleted

Post by kjKmcQN3 » 06 Jun 2017, 18:16

Thanks for the very thorough reply. I now understand why certain items get deleted instead of being marked as Dupe or Copy and it makes a lot of sense. However, in this particular instance the file that got deleted was a Proper release of an already downloaded item (the original item was a regular release and did not have Proper in its name). Shouldn't it have instead been identified as a separate item and been added to the queue?

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Proper gets marked as a Dupe and then gets deleted

Post by hugbug » 06 Jun 2017, 18:30

Duplicate check identifies duplicates via duplicate key. Only if duplicate key isn't set the titles are used for identification. Feed items coming from newznab feeds have unique identifier for tv show or movie. This allows nzbget to generate duplicate keys automatically.

Click on an item in history or queue, then on button Log and you'll see its duplicate key.

As a result "proper" doesn't matter as both items have the same duplicate key. However you can make it matter by using duplicate scores as explained in the docs.

kloaknet
Posts: 337
Joined: 23 Jul 2014, 08:52

Re: Proper gets marked as a Dupe and then gets deleted

Post by kloaknet » 07 Jun 2017, 16:05

maybe this helps you a bit to get an idea:

Code: Select all

# don't want:
Reject: 1080p | DD5.1 | KORSUB | HC | HDrip | hdcam |  BRRip | HDTC | HDTS | TC | TS

# retention limit
Require: age:<1200

# raise dupe for better stuff
Options(dupescore+:1): .bluray.
Options(dupescore+:2): .720p.

# propers
Options(dupescore+:10):   internal
Options(dupescore+:20):   repack
Options(dupescore+:40):   proper
Options(dupescore+:80):   real

# Dont grab non scene stuff again
Options(dupescore+:1000): GR0UP1 | GR0UP2 | SC3NE
Reject: dupestatus:success dupescore:<1000

# Accept all that passed above
Accept: *

kjKmcQN3
Posts: 75
Joined: 29 Aug 2015, 15:09

Re: Proper gets marked as a Dupe and then gets deleted

Post by kjKmcQN3 » 13 Jun 2017, 18:55

kloaknet wrote:
07 Jun 2017, 16:05
maybe this helps you a bit to get an idea:

Code: Select all

# don't want:
Reject: 1080p | DD5.1 | KORSUB | HC | HDrip | hdcam |  BRRip | HDTC | HDTS | TC | TS

# retention limit
Require: age:<1200

# raise dupe for better stuff
Options(dupescore+:1): .bluray.
Options(dupescore+:2): .720p.

# propers
Options(dupescore+:10):   internal
Options(dupescore+:20):   repack
Options(dupescore+:40):   proper
Options(dupescore+:80):   real

# Dont grab non scene stuff again
Options(dupescore+:1000): GR0UP1 | GR0UP2 | SC3NE
Reject: dupestatus:success dupescore:<1000

# Accept all that passed above
Accept: *
Thanks for the very helpful clarifications guys. I think I understand the issue a lot better now!

@kloaknet
Thanks for the excellent list! Could you please show me how I can add some of the filters mentioned? Do I add them in the filter section under RSS Feeds? If so, how do I add multi-lines such as:

# propers
Options(dupescore+:10): internal
Options(dupescore+:20): repack
Options(dupescore+:40): proper
Options(dupescore+:80): real

Could you please share a screenshot if possible? Many thanks!

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Proper gets marked as a Dupe and then gets deleted

Post by hugbug » 13 Jun 2017, 19:03

Click the button right of the filter edit box which brings an extra dialog.

kjKmcQN3
Posts: 75
Joined: 29 Aug 2015, 15:09

Re: Proper gets marked as a Dupe and then gets deleted

Post by kjKmcQN3 » 14 Jun 2017, 04:20

Ah. That was sloppy of me. Thanks again!

Post Reply

Who is online

Users browsing this forum: No registered users and 51 guests