NZBGet on host1, other apps on host2 - Post processing over NFS?

Get help, report and discuss bugs.
Post Reply
guitarlp
Posts: 2
Joined: 25 Jul 2017, 10:08

NZBGet on host1, other apps on host2 - Post processing over NFS?

Post by guitarlp » 25 Jul 2017, 10:29

I'm looking to replace my current host that runs sab with all my other applications with a more modular server setup. Since I'm running under ESXi, I was thinking of running NZBGet on its own host, and then running other applications like CouchPotato, Sickbeard, etc. on a different host.

I have NZBGet running on a new CentOS 7 host, and all is good. But now that I'm starting to experiment with pointing my existing apps like CouchPotato to this new host I'm starting to wonder if this is the right move.

When NZBGet and CouchPotato are on the same host, they can access the same local hard drive. So when a download is finished, CouchPotato has direct access to that disk and can run the needed post processing tasks before moving the file to the final destination (my NAS).

But when these services are on different hosts, am I making things harder than they need to be?

I believe in my case I need to setup NFS on both servers. The nzbget server becomes the NFS server, and the CouchPotato server becomes the NFS client. After sharing the movie download directory over NFS from the NZBGet server, I need to mount that on the CouchPotato server (using autofs), and update the couchpotato settings to use that mount point as the FROM folder. I also need to update the CouchPotato server to also mount the NAS share point, as that will be the final destination CouchPotato will send to.

Does that sound right? If it does, then doesn't this seem a bit messy?

NZBGet will download locally, then CouchPotato will re-name and (I believe) download the file locally over NFS so that it can then send it over NFS again to the final destination.

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: NZBGet on host1, other apps on host2 - Post processing over NFS?

Post by hugbug » 25 Jul 2017, 10:46

Efficiency is about (not) moving data between hosts and volumes. Since NZBGet downloads and unpacks it's important to minimise the network traffic and moving between volumes. On the other side for unpacking it's better to unpack to other physical drive.

The best setup for NZBGet is to have two separate local physical drives (SSD or HDD) - one for InterDir and another one for DestDir. Downloading works into InterDir. Unpacking works from InterDir (read data) to DestDir (write data). With two drives it works the best. If your destination is your NAS it's best to run NZBGet on NAS directly.

Now about CouchPotato. What it does with the files? If it only renames them (moving within same volume counts as renaming) then it doesn't matter if the files are local or on a remote drive, renaming works fast in both cases. If however your post-processing in CouchPotato involves reading of total file data, for example if it does transcoding, then the data should be on local drive, for best efficiency.

We are talking about "best efficiency" here. In many cases however "enough efficiency" is also acceptable if the setup has other advantages. You can for example run NZBGet on one machine and InterDir on that machine locally, then let NZBGet unpack into NAS (DestDir on NAS). That's not as fast as unpacking to local drive but removes the downloading burden from NAS.

I wouldn't bother putting NZBGet and CouchPotato in two different VMs if they both run on the same physical machine as I think that would be a waste of resources. But again there are other factors such as easier backup/restore/updates of apps independently.

guitarlp
Posts: 2
Joined: 25 Jul 2017, 10:08

Re: NZBGet on host1, other apps on host2 - Post processing over NFS?

Post by guitarlp » 25 Jul 2017, 11:22

Thank you so much for the reply. What you're saying makes perfect sense.

On my current server, I have the host OS running on a .vmdk (again, ESXi), and another hard drive attached directly to that host. I mount that drive in /mnt/data/ and I have Sab process my temporary files to /mnt/data/temp and finished downloads to /mnt/data/downloads. From there, if it's a movie, CouchPotato will then copy the file over NFS to my NAS (unRAID).

I think you pointed out one option with the setup I'm thinking about pursuing though. I can still download to a disk on the nzbget server (InterDir) and then have the final destination (DestDir) be my NAS. In this case, I'm still transferring the file over NFS to the NAS like I always have, and the transfer over NFS only happens once.

CouchPotato would then watch that folder on the NAS and move/rename as needed. Like you mentioned, that move is near instant since the file(s) already exist on the NAS. I don't do any transcoding with CouchPotato, so that performance bottleneck isn't a concern with this setup.

This is a bit of waste of resources because each machine needs the roughly 500MB of ram just to run the host OS. So by running 2 hosts, I'm wasting 500MB (roughly) of ram compared to running everything on one host. I'll have to think about this a bit more, but my initial thought was as you mentioned "easier backup/restore/updates of apps independently."

In my use case, would there be any benefit in having 2 additional local disks attached to the NZBGet server (one for InterDir and one for DestDir), when CouchPotato would still be moving the files from DestDir to the NAS during post processing?

hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: NZBGet on host1, other apps on host2 - Post processing over NFS?

Post by hugbug » 25 Jul 2017, 12:01

guitarlp wrote:
25 Jul 2017, 11:22
would there be any benefit in having 2 additional local disks attached to the NZBGet server (one for InterDir and one for DestDir), when CouchPotato would still be moving the files from DestDir to the NAS during post processing?
It's better to unpack with NZBGet directly to NAS and let CouchPotato just rename. It should be faster than "unpack locally, then move" and saves an extra drive.

Whether to use an extra drive for InterDir depends on if the existing drives are a bottleneck. The tips are about achieving maximum performance but you should of course consider costs too. In other words, if you have separate physical drives - use them to improve performance. Putting an extra drive makes sense if the performance benefit is important for you.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 51 guests