Why does NZBGet do post-processing in the DestDir?

Get help, report and discuss bugs.
mmayhew
Posts: 5
Joined: 14 Jun 2017, 16:29

Why does NZBGet do post-processing in the DestDir?

Post by mmayhew » 14 Jun 2017, 16:34

I just switched from SAB to NZBGet and I love it so far but it bugs me that it extracts directly to the DestDir directory. I have NZBGet running on a very fast machine and my DestDir is my NAS. Not only does post-processing take longer now that it's happening over the network via a UNC path but I've also been dealing with file locks causing post-processing to fail. I already have all of my temp directories excluded from things like anti-virus, plex transcoding, indexing, etc but now I have to do the same thing on my NAS for this DestDir. That doesn't make sense.

PLEASE do the extraction, renaming, etc in the InterDir or even the MainDir but not the DestDir. Only move the final file there.

mmayhew
Posts: 5
Joined: 14 Jun 2017, 16:29

Re: Why does NZBGet do post-processing in the DestDir?

Post by mmayhew » 14 Jun 2017, 16:44

To further explain, see the attached screenshot. Process Explorer shows NZBGet unraring directly to the UNC path. The only UNC path in my entire NZBGet configuration is the DestDir. Everything else is local, I have the temp dir and inter dir specified but NZBGet unrars directly to the DestDir.
Attachments
unrar.png

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Why does NZBGet do post-processing in the DestDir?

Post by hugbug » 14 Jun 2017, 17:57

Thanks for your feedback. Let me clarify.
mmayhew wrote:
14 Jun 2017, 16:34
Not only does post-processing take longer now that it's happening over the network via a UNC path
Your assumption that unpacking to local drive and then moving to NAS is faster than unpacking to NAS is plain wrong. NZBGet unpacks directly to DestDir for the reason of making the whole process faster and more performant. And in a case you didn't know - SAB works the same way.

Renaming (par-renaming, rar-renaming) and repair are performed in InterDir (before unpack).

If you still wants unpack locally set DestDir to local drive, then use a post-processing script to move files to NAS, for example VideoSort or EasySort. That can make sense if your post-processing needs to read the whole extracted files, for example if you do transcoding with post-processing scripts.
mmayhew wrote:
14 Jun 2017, 16:34
I already have all of my temp directories excluded from things like anti-virus, plex transcoding, indexing, etc but now I have to do the same thing on my NAS for this DestDir.
You will need to do that even if files will be moved after local unpack because moving to NAS requires more than a millisecond and will take roughly the same time as unpack.



BTW, instead of downloading on PC to NAS why not install NZBGet directly on NAS?

mmayhew
Posts: 5
Joined: 14 Jun 2017, 16:29

Re: Why does NZBGet do post-processing in the DestDir?

Post by mmayhew » 14 Jun 2017, 18:09

You are right, I made a few assumptions. I didn't actually test to see how SAB was working but I just find it hard to believe it's unpacking to the same destination directory because I had zero file lock issues with SAB and I'm using the same directories. The reason I say unpacking to the NAS is slower is because it's disks with platters and heads in a raid 5 config. My NAS is also low on resources because of what all it does (hypervisor, camera feed recorder, etc) so writing to it is not meant to be fast. That's why I would prefer to extract locally using zero NAS resources and using my local SSD and i7. Either way, This isn't a show stopper for me at all, I just had to get plex and my anti-virus to stop messing with that directory... the same one SAB used. But with NZBGet, 3 out of 5 downloads fail in post-processing because of "permission denied". Both running as a service configured to run as a local user with permissions to the UNC path as well as being launched locally by the same user that has permissions. Once I excluded the folder from Plex and AV, I have 100% success rate with post-processing.

I used to run everything on my NAS or on a VM on my NAS but it's just underpowered. I couldn't download faster than 15Mbps due to disk writing contention. If you look up 7200rpm disks in raid 5, unfortunately you can't expect much more.

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Why does NZBGet do post-processing in the DestDir?

Post by hugbug » 14 Jun 2017, 18:42

mmayhew wrote:
14 Jun 2017, 18:09
My NAS is also low on resources because of what all it does (hypervisor, camera feed recorder, etc) so writing to it is not meant to be fast. That's why I would prefer to extract locally using zero NAS resources and using my local SSD and i7.
When you unpack to NAS the only thing NAS has to do is to write the data it gets from PC. When you unpack locally and when move to NAS the only thing NAS has to do is to write the data it gets from PC. As you see for your NAS there is no difference whether you unpack locally or directly to NAS.
mmayhew wrote:
14 Jun 2017, 18:09
I just had to get plex and my anti-virus to stop messing with that directory... the same one SAB used. But with NZBGet, 3 out of 5 downloads fail in post-processing because of "permission denied".
Can you please provide a log for one such case in order for me to see at what stage the error occurs? Click on item in history, then on button "Log", when button "Save to file" or send me the global log-file (that's even better as this file may contain more info) - nzbget@gmail.com.

SAB unpacks into subdirectory "__UNPACK" (or something like this) which it creates as temporary directory inside destination, after unpacking it moves files from "__UNPACK" into destination directory. NZBGet uses for the same purposes directory "_unpack" as well. It's possible that your anti-virus and plex are configured to ignore files in directory "__UNPACK". You would need to configure them to ignore files in directory "_unpack". If you want I can provide you with a special build which uses subdirectory "__UNPACK" to verify that theory.
mmayhew wrote:
14 Jun 2017, 18:09
I used to run everything on my NAS or on a VM on my NAS but it's just underpowered. I couldn't download faster than 15Mbps due to disk writing contention. If you look up 7200rpm disks in raid 5, unfortunately you can't expect much more.
So you tried NZBGet on your NAS already or have your tried SAB only?

Do your really mean 15Mbps (2MB/s = Megabytes/s)? I can't imagine NAS being that slow. I personally use a SAT receiver as NZBGet download box. It has one 5400rpm 2.5" drive from an old notebook and ARM CPU with 2 cores @1700MHz. It achieves 11 MB/s maximising my 100Mbps line and uses about 60% CPU. I also used to have an older SAT receiver with MIPS CPU 2 cores @ 400 MHz which could download at 5 MB/s with 100% CPU load.

Every NAS should be able to download at 11 MB/s (100Mbps) easy. Use article cache (option ArticleCache) in NZBGet to minimize disk activity during downloading.

Installation on Linux provides instructions if you want to give it a try. The installation process will take about five minutes and you could quickly try the program. I recommend installing directly on system (not in VM) as this is easier on resources. The installers puts all files into one directory which you can delete after the test and nothing remains on the system.

mmayhew
Posts: 5
Joined: 14 Jun 2017, 16:29

Re: Why does NZBGet do post-processing in the DestDir?

Post by mmayhew » 14 Jun 2017, 20:12

I apologize I've been replying from my phone and I completely meant I download at 15MB on my NAS. I have a 300Mbps connection so I appreciate being able to use it, mainly because I'm a nerd. And I see your point now about unpacking to the NAS. I wasn't thinking through it.

On the disk locks, once I started monitoring with Process Explorer, it was Plex Media Scanner or Plex Transcoder every single time.

Plex is told to monitor for new content here: /a/b/c/Movies
My final directory (for NZBget) is /a/b/c/Movies/Download/

So when NZBget goes to move the file, it can't because Plex is still scanning it, generating thumbnails, importing, etc.

I've fixed this by moving my final directory to a/b/c/Movie_Downloads/ which Plex does not monitor. I tried using Plex's .plexignore files but like many others in their forums, it ignored the proper syntax and scanned anyway.

I've sent a few of the logs that did have the file lock issue. Sorry my first message was a request and not a thanks. I installed NZBget for the first time last night (12 hours ago) and have downloaded 871gb since.The only issue I had at all was the file lock issue but once solved, I re-ran post processing on all the failed downloads and they all were successful. I've already found it a much nicer experience than SAB. If you don't have plans to implement notification integration (growl, telegram, pushover, etc), I'd appreciate your consideration. I do like being able to push failed downloads to a notifier because it's normally indicative of a problem I need to be aware of.

Thanks so much for responding so quickly and thorugly. I know you are the sole maintainer. Thanks for the project.

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Why does NZBGet do post-processing in the DestDir?

Post by hugbug » 14 Jun 2017, 21:01

mmayhew wrote:
14 Jun 2017, 20:12
I download at 15MB on my NAS.
With NZBGet or SAB? If you didn't try NZBGet there it would be interesting to know how it performs, even if you are not gonna use it there after the test. 300Mbps (approx. 32-36 MB/s) should be possible with two cores Intel CPUs. If you can use a separate physical drive (HDD or SSD) for InterDir, that would decrease load on RAID.
mmayhew wrote:
14 Jun 2017, 20:12
I've fixed this by moving my final directory to a/b/c/Movie_Downloads/ which Plex does not monitor.
At the end you probably want Plex to scan new files. You can achieve this by using pp-script VideoSort
to move downloaded and unpacked files from Movie_Downloads in Movies. Since both dirsctories are on the same partition the move will be instantaneous.

I'm still interested to know what SAB does different to avoid access errors. From what I read SAB devs also recommend to exlude the destination directory from scanning; so it must have the same issues.

If you still have SAB, can you check with Process Monitor if Plex accesses the files during unpack by SAB? May be you could even see if SAB makes mutliple attempts when moving files from __UNPACK into the destination. The reason NZBGet doesn't do multiple attempts is because it doesn't know how long should it try before reporting errors.

I'll take a look at the log-files later, probably tomorrow (it's late evening here).
mmayhew wrote:
14 Jun 2017, 20:12
growl, telegram, pushover,
They all are supported. Take a look at Notifications section of Catalog of extension scripts. In particular, as a nerd, you should fall in love with pp-script Notify - notifications (All In One) which supports just any notification service in the world. And if you manage to find a service it doesn't support - let the author know and he adds support for it, probably the same day you ask.

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Why does NZBGet do post-processing in the DestDir?

Post by hugbug » 16 Jun 2017, 09:01

mmayhew wrote:
14 Jun 2017, 20:12
I've sent a few of the logs that did have the file lock issue.
Having read the logs I can only confirm your observation that the file locking issue appears when nzbget tries to move extracted files from _unpack into destination directory.

mmayhew
Posts: 5
Joined: 14 Jun 2017, 16:29

Re: Why does NZBGet do post-processing in the DestDir?

Post by mmayhew » 17 Jun 2017, 03:45

hugbug wrote:
16 Jun 2017, 09:01
Having read the logs....
Thanks so much for looking. Like I said before, once I got past this hiccup it was smooth sailing. I've successfully downloaded over 225 nzbs since the file locking issue with zero problems other than deleted articles. I love how logging events have a relationship with specific downloads. I also love how logging failures clear from the web console if the failure is corrected. Thanks again, sending some beer money your way.

habskilla
Posts: 13
Joined: 29 Oct 2014, 13:50

Re: Why does NZBGet do post-processing in the DestDir?

Post by habskilla » 30 Jun 2017, 12:56

hugbug wrote:
14 Jun 2017, 21:01
At the end you probably want Plex to scan new files. You can achieve this by using pp-script VideoSort
to move downloaded and unpacked files from Movie_Downloads in Movies. Since both dirsctories are on the same partition the move will be instantaneous.
instantaneous you say. That got me thinking. It makes a lot of sense to have the _unpack on the destination and then Videosort will do a simple move. So I changed the Movie destination in categories to a tmp dir on my NAS. Movie downloaded and unpacked successfully into the tmp dir. Then Videosort kicked off. Found the new NAS tmp dir and proceeded to copy, not move the file.

Am I missing a Videosort setting somewhere that says instructs Videosort to do a move instead of copy?

Post Reply

Who is online

Users browsing this forum: No registered users and 55 guests