[Dirvish] Vault Unsynced

Keith Lofstrom keithl at kl-ic.com
Mon Dec 27 13:00:03 PST 2004


On Mon, Dec 27, 2004 at 09:54:42AM -0800, Steve Ramage wrote:
> 
> I've noticed a HUGE jump in disk consumption accross all my backups, 
> they seem to be double each of there original size. I don't know when 
> this happened, but it was recent. file.sh is just something that does it 
> for every folder (before I found out I could do du -s -si 20041029/ 
> 20041030/


Well, one way to see what happened is to examine the per-vault dirvish
log files.  For example, I have a log file for the vault of my firewall
root, from last night, at:

/backup/dirvish/fw/fwroot/2004-1227-0300/log

In that file, it lists the output of the pre-server script, then the
pre-client script, and then the most important part:

*   ACTION: rsync -vvrltH --delete -pgo --stats -D --numeric-ids -x   \
*   --exclude-from=/backup/dirvish/fw/fwroot/2004-1227-0300/exclude   \
*   --link-dest=/backup/dirvish/fw/fwroot/2004-1226-0300/tree fw:/    \
*   /backup/dirvish/fw/fwroot/2004-1227-0300/tree
                                                                                
*   opening connection using ssh fw rsync --server --sender -vvlHogDtprx \
*    --delete --numeric-ids . /
*   receiving file list ...
*   expand file_list to 4000 bytes, did move
...
(lines broken for clarity)

The action tells us the actual command we are feeding rsync;  most important
thing that might concern your problem is the --link-dest, which points to the
previous day's file tree.

What follows is a verbose description of what happened;  most importantly, 
there is a very long list of the files traversed.  Near the tail end of the
log file are the rsync statistics:

...
*   Number of files: 24095
*   Number of files transferred: 1
*   Total file size: 76242237 bytes
*   Total transferred file size: 628 bytes
*   Literal data: 628 bytes
*   Matched data: 0 bytes
*   File list size: 383116
*   Total bytes written: 131
*   Total bytes read: 384305
*
*   wrote 131 bytes  read 384305 bytes  13488.98 bytes/sec
*   total size is 76242237  speedup is 198.32
...

Note that last night, not very many files changed on the root partition of
my firewall, which is a GOOD thing.  Sounds like you were not so lucky.

If you look at all the log files, you might be able to find the one 
associated with the move, and from that (and "man rsync") determine why
rsync decided to move all the files again.

If I had to guess, I would guess that rsync got stopped near the beginning
of a transfer, and that the next night, dirvish did not detect that the
previous image was incomplete.  But the log will tell the tail.  There
may be some problem in dirvish, or it may be operator error;  the first is
a coding fix, and the second is a documentation fix, so either way what
you learn is important.  

As far as merging two sets of images, before and after the problem, there
is not a good tool for that.  You have two sets of files and inodes, and
tranferring the pointers from one set of directories to another is a
very complicated procedure that would involve fooling with raw bytes in
the file system, over and over again for each file in the "old" set and
in the "new" set.  

The management answer is that Fry's is advertising 250GB Western Digital
Hard drives for $100 US.  So when you run out of disk space, buy another.
It isn't worth a lot of grief to save a dollar's worth of disk space, and 
having two separate copies might be advantageous some day.  When I worked
at Tektronix, years ago, we called this "if you can't fix it, feature it!"

Keith

-- 
Keith Lofstrom          keithl at keithl.com         Voice (503)-520-1993
KLIC --- Keith Lofstrom Integrated Circuits --- "Your Ideas in Silicon"
Design Contracting in Bipolar and CMOS - Analog, Digital, and Scan ICs


More information about the Dirvish mailing list