[Dirvish] implementing new dirvish features

Dave Howorth dhoworth at mrc-lmb.cam.ac.uk
Thu Nov 15 10:34:21 UTC 2007


Jacki wrote:
> hello dirvish users and developers
> 
> i am an excessive dirvish user and think about implementing some improvements. please let me known your opinion about the following ideas:

Hello Jacki,

Lots of good ideas for improvements and I've made some individual
comments below. There's been a lot of discussion of possible
imporovements on the list and it would be worth your while to go back
and read those discussions. It turns out that ideas are not a problem
and nor are people willing to implement them.

What hampers acceptance is that dirvish is a backup program and so
reliability is of paramount importance. Making changes can introduce
bugs, so we need to have a plan to avoid that. The basis of any such
plan is testing and sadly dirvish is lacking a full test suite. Keith is
also working on restructuring dirvish to make it easier to modify and
test, so that should be factored in to any plans.

> ###################
> # mail generation #
> ###################
> 
> #Target state#
> dirvish generates an configurable mail content, listing all vaults and vault-details like the table below:
> 
> #implementation#
> i think this feature can only work with dirvish-runall:
>   - dirvish-runall has to create a file at a configured place, with a configured name (configurable by master.conf)
>   - dirvish-runall generates the header of the table and passes the filename of the report file to the dirvish command
>   - dirvish-runall invokes the mail hook, which sends the mail

Excellent idea. There's a possible alternative approach to
implementation though, which is to write a script that is run after
dirvish and which analyses the logs to produce the report. Which is a
generalized version of what everybody does now. This approach has the
advantage that it doesn't change the dirvish code, so there's no risk of
introducing bugs.

> ########################################
> # dirvish-runall runs dirvish parallel #
> ########################################
> 
> #current state#
> dirvish-runall process all vaults like listed in master.conf under
"runall:". this not fine, if you backup over a speed-limited and not so
reliable connection. a "big" backup in the middle of the list blocks
"small" backups at the end. especially if the connection is lost.

A nice enhancement but it's already possible to achieve most of the
effect by having multiple conf files - this has been discussed before.
And of course if you put the small backups before the big one, you avoid
that specific issue. Also, if your backup network speed is significantly
constrained, it may be better to make a backup to a local drive first so
it's less of an issue how long the network transmission takes.

> ####################################
> # selectable interpreter for hooks #
> ####################################
> 
> #current state#
> there are 4 hooks "pre-server", "pre-client", "post-client" and "post-server"
> these commands are executed by /bin/sh

I don't really see the need for change here, unless there's some
specific problem that I haven't understood.

In general if you want to invoke say a perl script, you just put
#!/usr/bin/perl on the first line in the normal way, I believe. Sure
there's a perhaps unnecessary /bin/sh instance as its parent but that's
not really important is it?

The scripts on the client are executed via rsh and exactly what happens
can be controlled via the 'rsh' config option and by configuration on
the client. There also seems to be an undocumented 'pre-post-rsh' option
that may help.

> #########################################
> # dirvish-runall vaults on command line #
> #########################################
> 
> #current state#
> dirvish-runall executes dirvish on all vaults within master.conf and
"runall:" if you split your backup, for instance in daily backups and weekly
backups, you have to do one the invokations by hand
>
> #target state#
> a command like "dirvish-runall /srv/backup/daily/*" executes dirvish on all vaults in "/srv/backup/daily/"
> a command like "dirvish-runall /srv/backup/weekly/*" executes dirvish on all vaults in "/srv/backup/weekly/"
> a command like "dirvish-runall" executes dirvish on all vaults specified in master.conf

Again, the ability to do this already exists. Search the archive for
more discussion. For example I have:

 /usr/local/sbin/dirvish-runall # defaults to master.conf
 /usr/local/sbin/dirvish-runall --config /etc/dirvish/wwpdb.conf

Specifically if you want to run things at different frequencies, put the
runall commands in the appropriate cron files!

> #####################
> # summary           #
> #####################
> i am willing to implement these features - who whats to update the documentation
> 
> thanks for your feedback
> jacki

Thanks for these suggestions. As I said, IMHO it's not a question of who
wants to implement the improvements or even the docs, it's who's
prepared to implement a full regression test suite?

Cheers, Dave


More information about the Dirvish mailing list