[Schd-Script] DirWatch - Multiple NZB-File Drop Support

Share your scripts or request scripts with specific features.
Forum rules
Please keep the forum clean - one topic per script. Questions not related to a specific script should be posted in Support forum.
hugbug
Developer & Admin
Posts: 7645
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: [Schd-Script] DirWatch - Multiple NZB-File Drop Support

Post by hugbug » 01 Mar 2017, 09:29

It can be that the wrong python is used to exec the script.
Try option ShellOverride in nzbget to associate .py-files with python 2.x.

l2g
Posts: 228
Joined: 27 Jun 2014, 22:13
Contact:

Re: [Schd-Script] DirWatch - Multiple NZB-File Drop Support

Post by l2g » 01 Mar 2017, 14:39

nzbget_fan wrote: info Wed Mar 01 2017 09:16:31 DirWatch: 2017-03-01 09:16:31,229 - 4196 - WARNING - Source directory C:\Windows\system32\* was not found.
Off hand, i can see a flaw with the line above; 2 things worth mentioning:
  • Wildcards (*) work from the command line (but not NZBGet), so C:\Windows\system32\* should become C:\Windows\system32
  • The other thing is just to question if this is what you actually want? Do you really want to search C:\Windows\system32\ for nzb-files? Seems like a rather precarious spot to drop NZB-Files; I mean... right in with your operating systems core libraries?. But hey... to each their own right? :)
nzbget_fan wrote: My nzbget root-directory (where nzbget.exe is located) is "F:\!Downloads\Anwendungen\Netzwerk Tools\nzbget".
This part of the script is working as it should, you can see within the logs that it detected that NZB-Files should be moved here:

Code: Select all

info	Wed Mar 01 2017 09:16:31	DirWatch: 2017-03-01 09:16:31,165 - 4196 - INFO - Target directory set to: F:\!Downloads\Anwendungen\Netzwerk Tools\nzbget\downloads\nzb
I think you're close to having it all working; just have another look at your WatchDir entries. In the meantime, when i get home tonight, i'll see if i can spot anything i may have overlooked.

nzbget_fan
Posts: 6
Joined: 28 Feb 2017, 19:43

Re: [Schd-Script] DirWatch - Multiple NZB-File Drop Support

Post by nzbget_fan » 01 Mar 2017, 20:41

Thank you very much for your support.

This hint from hugbug helped me.
Try option ShellOverride in nzbget to associate .py-files with python 2.x.
The script runs now.The search path in the script options didn't contain any system paths. My next problem was that another script which took the passwords from filenames didn't run anymore when the ShellOverride option was set. I'm avoiding the problem with the two python versions by running the script as a windows task.

But there's another "bug" which I underestimated. I accidently ignored this behaviour in my first test:
Either it is run as nzbget oder standalone-script it doesn't move/delete the original zip-file. So every time I run the script it imports the old an new files into nzbget (cause the old files are still in the source directroy). From command line I'm running it by starting cmd.exe with admin rights and executing "py dirwatch.py ...." from there. I tried it even with several source folders and also reinstalled the latest Python 2.x-Version 2.7.13 for windows.
2017-03-01 21:25:52,030 - 15756 - INFO - Target directory set to: F:\!Downloads\Anwendungen\Netzwerk Tools\nzbget\downloads\nzb
2017-03-01 21:25:53,246 - 15756 - INFO - Loaded NZB-File: Test_{{PW}}.nzb.zip, category='Movies'
2017-03-01 21:25:53,246 - 15756 - WARNING - Failed to remove NZB-File D:\Test\Test_{{PW}}.nzb.zip after loading it into NZBGet

l2g
Posts: 228
Joined: 27 Jun 2014, 22:13
Contact:

Re: [Schd-Script] DirWatch - Multiple NZB-File Drop Support

Post by l2g » 01 Mar 2017, 21:52

nzbget_fan wrote:Thank you very much for your support.

This hint from hugbug helped me.
Yeah... he's pretty awesome; I'm glad you're closer to a working solution :)
nzbget_fan wrote:But there's another "bug" which I underestimated. I accidentally ignored this behaviour in my first test:
Either it is run as nzbget oder standalone-script it doesn't move/delete the original zip-file. So every time I run the script it imports the old an new files into nzbget (cause the old files are still in the source directroy). From command line I'm running it by starting cmd.exe with admin rights and executing "py dirwatch.py ...." from there. I tried it even with several source folders and also reinstalled the latest Python 2.x-Version 2.7.13 for windows.

Code: Select all

2017-03-01 21:25:52,030 - 15756 - INFO - Target directory set to: F:\!Downloads\Anwendungen\Netzwerk Tools\nzbget\downloads\nzb
2017-03-01 21:25:53,246 - 15756 - INFO - Loaded NZB-File: Test_{{PW}}.nzb.zip, category='Movies'
2017-03-01 21:25:53,246 - 15756 - WARNING - Failed to remove NZB-File D:\Test\Test_{{PW}}.nzb.zip after loading it into NZBGet
After i process the NZB-File, i attempt to delete it (for this very reason because I'd hate to process it a second time and get stuck in a loop. But maybe i need to re-think this process a bit better? Maybe I should try renaming it to something else... (anything at all). I'd do this just to see if i have write permissions on the directory (where the NZB-Filee sits)... Only if these permissions are present would i preform the file handling.... make sense?

Thoughts?

In the meantime, you can definitely fix this problem by making sure the same user that NZBGet runs as has write permissions to the directory the NZB-Files are found in.

I'll have a look at this tonight if i can; otherwise worst case tommorow night.

nzbget_fan
Posts: 6
Joined: 28 Feb 2017, 19:43

Re: [Schd-Script] DirWatch - Multiple NZB-File Drop Support

Post by nzbget_fan » 02 Mar 2017, 20:46

After i process the NZB-File, i attempt to delete it (for this very reason because I'd hate to process it a second time and get stuck in a loop. But maybe i need to re-think this process a bit better? Maybe I should try renaming it to something else... (anything at all). I'd do this just to see if i have write permissions on the directory (where the NZB-Filee sits)... Only if these permissions are present would i preform the file handling.... make sense?

Thoughts?

In the meantime, you can definitely fix this problem by making sure the same user that NZBGet runs as has write permissions to the directory the NZB-Files are found in.

I'll have a look at this tonight if i can; otherwise worst case tommorow night.
Unfortunately I'm not good in programming :( . So I have no opinion about how solving file handling the best way. I'm even not sure if it is a problem with the script or Python. I''m sure that my windows user (admin) has all rights in the source directory. So when I start your script from windows command line I get the error mentioned above, that it was unable to remove the Zip-File. After running your script I did a DOS

Code: Select all

del *.*
in the source directory within the same command line box and it deleted the file as expected. So it could hardly be a windows rights problem in the folder. When I start your script with nzbget it can't remove the file neither.

Code: Select all

warning	Thu Mar 02 2017 21:05:24	DirWatch: Failed to remove NZB-File D:\Test\Testfile_{{pw}}.nzb.zip after loading it into NZBGet
When I was just reading my text again I recognized that it was a zip-file containing the nzb-file that couldn't be deleted. So I just extracted the nzb file to the folder

Code: Select all

D:\Test
manually and tested it again. Now it was able to delete the remaining nzb-file in the folder. So I guess it might be a problem with the unzip-process which locks the zip-file and makes it undeleteable this way.

I hope this information helps you.

l2g
Posts: 228
Joined: 27 Jun 2014, 22:13
Contact:

Re: [Schd-Script] DirWatch - Multiple NZB-File Drop Support

Post by l2g » 03 Mar 2017, 02:12

nzbget_fan wrote: When I was just reading my text again I recognized that it was a zip-file containing the nzb-file that couldn't be deleted. So I just extracted the nzb file to the folder

Code: Select all

D:\Test
manually and tested it again. Now it was able to delete the remaining nzb-file in the folder. So I guess it might be a problem with the unzip-process which locks the zip-file and makes it undeleteable this way.

I hope this information helps you.
Thank you for your feedback! Anything you have to share helps regardless of your programming experience!

So... I did quite a few (moderate) code changes. I may or may not have solved your problem, but I certainly eliminated the issue you got into where you were processing the same file over and over again. Grab the latest master copy (shortcut) and give'r another shot! Please let me know your feedback!

nzbget_fan
Posts: 6
Joined: 28 Feb 2017, 19:43

Re: [Schd-Script] DirWatch - Multiple NZB-File Drop Support

Post by nzbget_fan » 03 Mar 2017, 10:32

So... I did quite a few (moderate) code changes. I may or may not have solved your problem, but I certainly eliminated the issue you got into where you were processing the same file over and over again. Grab the latest master copy (shortcut) and give'r another shot! Please let me know your feedback!
Hey, good job so far :) The script has almost stopped now from processing the same file over and over again. But it lets the processed file remain in the directory with a .dw ending and an additional copy with the ending .dw.dw. I would prefer a complete cleaning of the directory as I get a error message in my nzgbet protocol every minute / with each script cylcle that the prep of the remaining .dw file has failed.

Code: Select all

DirWatch: Could not prep FILE: NZB-File D:\Test\Testfile_{{pw}}.nzb.zip.dw (NZB-File D:\Test\Testfile_{{pw}}.nzb.zip.dw.dw)

nzbget_fan
Posts: 6
Joined: 28 Feb 2017, 19:43

Re: [Schd-Script] DirWatch - Multiple NZB-File Drop Support

Post by nzbget_fan » 03 Mar 2017, 12:23

PS: An easy solution might be deleting all .dw-files in the source directories at the beginning of each cycle.

l2g
Posts: 228
Joined: 27 Jun 2014, 22:13
Contact:

Re: [Schd-Script] DirWatch - Multiple NZB-File Drop Support

Post by l2g » 03 Mar 2017, 19:11

nzbget_fan wrote:PS: An easy solution might be deleting all .dw-files in the source directories at the beginning of each cycle.
Done! I added an option to handle .dw files if they're detected when the script starts (but it's a configurable option from the Settings, or -c (for --auto-clean) from the command line), then they will be automatically deleted (if permissions allow for it).

For the second issue your having with the files not being deleted, I can't reproduce this on my side unfortunately (but i also don't have a windows machine at this time to test with).

I have a hunch that your copy of NZBGet (when you start it up and run it as a daemon in the background) may not necessarily be running with Administration privileges. I only say this because otherwise it would be able to delete the files just as you suspect it should. I could be wrong here though; it's just a theory.

That all said... grab the master branch again (link in the previous comment). Put on the DebugMode too if you don't mind (accessible from the NZBGet Settings page associated with DirWatch). I've added some extra (debugging) output that will spit out the reason for deletion failures. Hopefully it will help us identify the issue you're having. That tied in with the new suggestion you recommended, well... we just might be good to go. :)

From my standpoint: as long as i don't process a file a second time (which by the way was a huge bug you pointed out - thank you), then I'm in good shape; I can't control file system permissions. As of now, the way the script works has become a little more convoluted (but much safer). In a nutshell (for each WatchPath specified):
  • Look for NZB-Files (.nzb) and Zip-Files (.zip) including those with a .dw extension
  • Check if the user wants me to clean up .dw files
    • if so: go ahead and delete them (if i can). If i can't, i don't panic... i just treat the file as if not identified on the next step.
    • if not: just take them out of the list of files we want to process (silently ignore them under the assumption they've already been handled)
  • Ignore any files found in this list that are less then X seconds old (default value is 30). Basically we don't want to touch files that have JUST arrived incase they're still being written to by another application. This is just a minor safeguard. The file will just get processed in the next iteration anyway when it's older then this time.
  • Begin processing each file we haven't filtered out yet:
    • Rename it to the same name but with a .dw extension (short for DirWatch).
    • Process it.... this might involve moving it... it might involve remotely connecting to the API and sending our information remotely... It might involve opening a Zip-File and processing each entry inside...etc
    • If... and only if we successfully processed the content do we remove the file (if AutoCleanup is specified - otherwise file is left with the .dw extension).
    • If i fail, i rename the file back (without the .dw extension).
Rinse and repeat for each file in each directory, and for each time the script's scheduled to wake up.

By default the AutoCleanup switch is set to No... you have to change it manually if you want to change this.

Let me know your thoughts and results! Good luck!

nzbget_fan
Posts: 6
Joined: 28 Feb 2017, 19:43

Re: [Schd-Script] DirWatch - Multiple NZB-File Drop Support

Post by nzbget_fan » 06 Mar 2017, 19:38

I used the weekend for some testing and am returning now with bad news.

The script is still not able to delete the .dw-files in the directory. I guess it has something to do how python accesses the windows file system. With Google's help I found this code

Code: Select all

import glob,os ; [os.remove(x) for x in glob.glob("D:\Test\*.dw")]
with which the script was able the .dw files. So I'm sure it's not because of folder or user rights (unless it has something to do with the default rights given to folders/files by windows).

But I ran in another problem where nzbget tells me after a soft start that the script is running in another session altough there's no other session of nzbget running. After stopping and starting the nzbget service the script is running, but I can't say for how long.

I understand that's difficult for you to comprehend my issues without a windows test plttform. Even I'm not sure if you could comprehend the issues with a suitable test platform. With this in mind, I'm taking your idea for creating an own solution with a windows folder watch tool and a batch script. I still say thank you very much for your work, your support and the idea for the script.

Post Reply

Who is online

Users browsing this forum: No registered users and 39 guests