[Dirvish] Excessively Long backups on a Postfix Mailder vault with Dovecot as the IMAP server
paul at debian.org
Wed Mar 19 14:57:42 UTC 2008
On Wed 19 Mar 2008, Damian Cunniff wrote:
> When we first implemented the system everything seemed to be running
> correctly. The vault took somewhere between 20 minutes to 1 hour to do
> its backups. Gradually we added new users to the new mail system and
> this time window increased but then subsided as would be expected.
> Recently however it continues to increase in the amount of time that it
> is taking to backup. It has reached the point of taking as much as 8-10
> hours to complete.
> Initially I thought we might be collecting enough mail on a daily basis
> to create this situation, however that doesn't really make sense when
> you look at how much mail actually comes into the server. Additionally
The important thing is how much mail is _stored_ on the system, not how
much is added every day. A very large number of files will cause a long
backup time, even though not much is actually transferred.
> I'm stumped. What could possibly be causing this problem? Why would an
> otherwise seemingly healthy setup be malfunctioning here? Should I fsck
> the RAID array in hopes that that may be the trouble?
fsck is probably useless if there aren't actual errors.
What's more important is what sort of filesystem you're using.
Assuming you're using linux (you don't say...):
reiser is bad at sequentially scanning very large directories, as it's
optimized for fast access to a random file (i.e. where you know the name
of the file, so it can use the index to find the file).
jfs is also not so good at random access.
I usually use either xfs or ext3 with the dirindex option.
Most importantly, if you haven't upgraded rsync to 3.0.0 yet, I'd advise
that, as the incremental transfer protocol can speed things up a lot.
(You need to upgrade both sides to make use of that protocol.)
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
Such a footer is usually worse than useless, as a virus will usually
claim something similar...
More information about the Dirvish