[New Feature] Smart duplicates

Discuss newly added features or request new features.
hugbug
Developer & Admin
Posts: 7514
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Smart duplicates

Post by hugbug » 30 Oct 2013, 07:56

A problem indeed. Thanks to HealthCheck the par-repair is now made before trying to unpack (in opposite to how it worked in v11, where it was trying to unpack first and then requested par-repair). In most cases if download comes to unpack it is not damaged (exception: one or more rar-files were missing in nzb-file). Moreover unrar returns a special error code for "not enough disk space". I need to save this result code as a separate Unpack-Status.

Regarding duplicate handling the status "space-problem" should be treat as success - all dupes should be moved to history. After freeing space the user can "Post-process again" from history. If unpack really fails the duplicate handling should work as in function "Mark as Bad" moving dupes back from history to download queue.

A lot of todo actually :shock:

hugbug
Developer & Admin
Posts: 7514
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Smart duplicates

Post by hugbug » 30 Oct 2013, 12:47

Not related to rar:

I'm evaluating a possible change in duplicate handling.
Currently all duplicates are in download queue: one is active and others are paused. This makes the managing of queue more difficult. If queue is big and you want do download only one or few items you could (before duplicate handling existed) pause all items which you don't want to download now. With duplicate handling a lot of care is needed when pausing/unpausing items.

Solution:
  • download queue contains only one of duplicates, all other must be kept outside of queue - we already have a place for that - it's history;
  • when a new item is added to queue and there is a duplicate with the same or higher score in queue, the new item is moved to history with DUPE-status;
  • when a new item is added to queue and there is a duplicate with lower score in queue, the existing item is moved to history with DUPE-status. The new item is added to queue;
  • if download fails the best of DUPE-items is moved back from history to queue for download;
  • if download completes successfully nothing extra needs to be made since all dupes are in history already;
  • dupekey-badges can be removed from the list of downloads making it cleaner. Badges will be shown in history still (no changes).

prinz2311
Posts: 464
Joined: 08 Dec 2012, 00:03

Re: [New Feature] Smart duplicates

Post by prinz2311 » 30 Oct 2013, 12:59

hugbug wrote:
  • if download fails the best of DUPE-items is moved back from history to queue for download;
  • if download completes successfully nothing extra needs to be made since all dupes are in history already;
Only true for automatically added items.

Not true when the user adds a new download and edits the dupekey, score for it. Then a dupe can be in the queue that needs to be removed or unpaused depending of result.

Or alternatively when the user edits the dupekey/score the item has to be checked against the existing items in queue and moved to history or the existing lower score has to be moved to the history.

hugbug
Developer & Admin
Posts: 7514
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Smart duplicates

Post by hugbug » 30 Oct 2013, 13:11

prinz2311 wrote:Or alternatively when the user edits the dupekey/score the item has to be checked against the existing items in queue and moved to history or the existing lower score has to be moved to the history.
Yes, something is needed for that case. Automatically moving to history is bad, because all relevant items need to be edited first, before it can be decided which of them to keep and which to move to history.

Possible solution: after editing of dupe properties the user should delete other duplicates from queue, they then should become delete-status=dupe in the history. This is very much similar to how user currently handles such a case - currently it has to pause duplicates, after the change - it has to delete the items.

hugbug
Developer & Admin
Posts: 7514
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Smart duplicates

Post by hugbug » 04 Nov 2013, 21:10

Update:
  • when a duplicate is added to queue it is now moved to history as dupe-backup instead of being paused in download queue;
  • if download fails the best duplicate is moved from history back to queue for download (if there are no duplicates in queue); this makes it easier to manage download queue without worrying about properly pausing/unpausing duplicates);
  • badges with duplicate info are not shown in the list of downloads and history anymore; if necessary they can be activated by manually editing setting "dupeBadges" in index.js;
  • when deleting downloads from queue there are three options now: "move to history", "move to history as duplicate" and "delete without history tracking";
  • DUPE-mark for history records can now be set or cleared via history dialog;
  • when deleting downloads from queue the messages about deleted individual files are now printed as "detail" instead of "info";
  • removed command "Mark as duplicates" from edit dialog for multiple selected downloads; the command is not needed anymore since all duplicate properties are now changeable'
When updating from previous testing versions: if you have paused duplicates in download queue you need to delete them from queue and choose option "Move to history as duplicates" in the delete confirmation dialog. If you don't do this the paused duplicates will not be used as duplicate backups.

hugbug
Developer & Admin
Posts: 7514
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Smart duplicates

Post by hugbug » 08 Nov 2013, 21:56

prinz2311 wrote:If a download fails because of Low disk space during unpack, a dupe will be unpaused. That doesn't make much sense, because the reason can only be fixed by freeing space on the device and the download has not really failed in most cases (and the unpaused will fail too in most cases if it's not much smaller).
Fixed in r905:
  • if unpack fails with write error (usually because of not enough space on disk) this is shown as status "Unpack: space" in web-interface;
  • this unpack-status is handled as "success" by duplicate handling (no download of other duplicate);
  • also added new unpack-status "wrong password" (only for rar5-archives);
  • env.var. NZBPP_UNPACKSTATUS has two new possible values: 3 (write error) and 4 (wrong password);
  • updated pp-script "EMail.py" to support new unpack-statuses

hugbug
Developer & Admin
Posts: 7514
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [New Feature] Smart duplicates

Post by hugbug » 21 Dec 2013, 21:44

Added new action "Download again" for history items. It's not limited to duplicate handling but is in particular useful with duplicates. It allows to download a history-item marked as "Dupe" without marking another item as "Bad". For example if you want compare qualities of two (or more) releases.

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests