Queue lost after system crash

Get help, report and discuss bugs.
Post Reply
ACiD GRiM
Posts: 15
Joined: 06 Aug 2014, 09:24

Queue lost after system crash

Post by ACiD GRiM » 16 Aug 2014, 03:31

I send an email with my queue folder attatched after my nzbget system halted and upon restarting all 7 pages in my queue were unfortunately gone.

This is after a number of times the same thing happening, and I recently started fresh by deleting the queue folder and all nzb from the watch directory. I did find that if I delete the queue directory and restart nzbget before deleting the nzb watch dir I can rename all of the .nzb.queue to .nzb to recover most of my queue, but some movies were redownloaded.

Maybe in the mean time it might be possible to rename queue'd and downloaded nzbs .nzb.<category>.queued and nzb.<category>.deleted or .nzb.<category>.ppqueued and parse them as named in the case of queue curruption.

hugbug
Developer & Admin
Posts: 7529
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Queue lost after system crash

Post by hugbug » 16 Aug 2014, 09:49

The file "queue" which holds the queue is empty (zero bytes). The system crash is most likely responsible for that.
When updating queue on disk NZBGet tries to do that safely:
  • queue saved into temporary file queue.new;
  • then existing file queue is renamed to queue.old;
  • then queue.new is renamed to queue;
  • then queue.old is deleted;
In your queue directory there is only "queue". Neither queue.old nor queue.new are there meaning the saving was completed without error.

What could cause the lost of queue file? May be the system buffers or hard drive buffers were not flushed? Or the file system got corrupted and were repaired after reboot and some files were lost.

What NZBGet can do about this? I can offer keeping of older queue-files (one, two or more copies). But I don't know if it will really help. During download the queue file is updated frequently (after every downloaded inside nzb). All backup files may be very fresh and all may still be cashed and then get lost. Having a backup of queue-file only is also bad because there are other important files in the directory and all files are related.

My understanding is that the system must flush files on closing. How can programs be sure about writing if the system writes data when it wants. I'm still not sure if this is what has happened. What is the file system used? How about putting queue-directory on another partition with another (better) file system?

I do have crashes now and then during development but never lost queue.

What you can do:
  • in your NZBGet startup script add code to make a backup of queue-directory:

    Code: Select all

    #!/bin/sh
    zip -r queue-$(date +%Y%m%d_%H%M%S).zip /path/to/queuedir
    nzbget -D
  • write a scheduler script, which will make backup of queue-directory. The script should pause NZBGet before doing backup.

ACiD GRiM
Posts: 15
Joined: 06 Aug 2014, 09:24

Re: Queue lost after system crash

Post by ACiD GRiM » 19 Aug 2014, 06:35

Thank you, that's a really good idea, I've had my nose in category scripts I forgot about the global scripts.

hugbug
Developer & Admin
Posts: 7529
Joined: 09 Sep 2008, 11:58
Location: Germany

Re: Queue lost after system crash

Post by hugbug » 19 Aug 2014, 07:59

I'm still curious why the queue-file got lost.
What file system is on the partition where queue is stored?
Is it EXT3/EXT4 a more exotic one?
Anything interesting about the drive: SSD, RAID, USB-attached, etc.?

Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests