Migrating vpopmail domains between servers

I often see questions about moving existing mailboxes from one machine to another. Sometimes it's a case of moving a single vpopmail domain from one server to another, sometimes it's a case of moving every mailbox in every domain on the machine to a new server, and sometimes it's a case of converting "system" domains (i.e. where the mailboxes correspond to system accounts from /etc/passwd) to vpopmail.

I've also seen a variety of different answers to these questions- some of which are more or less correct, and some of which will leave you with a non-working server. Hopefully the information on this page will clear up the confusion.

Hashing

One problem that people run into with vpopmail is that, even if the vpopmail user has the same physical home directory on both servers, the vadddomain and vadduser commands don't always use the same physical directories for a given domain or mailbox name.

On a server with many domains, or in a domain with many mailboxes, vpopmail uses hashing to make the server faster. This is because directories are physically not sorted on the disk- if a single directory contains 1000 entries and you're trying to find a file or directory within that directory, it has to do a linear search and may need to check up to a thousand entries to find what you're looking for. If you create a set of sub-directories, each containing no more than 50-100 entries, then finding one of a thousand entries involves one search through the top level directory (up to 60 entries) followed by a search through a child directory (up to 50 entries)... which ends up being much faster than searching through a thousand entries.

This is the same idea as the hashing done within qmail's queue, although the implementation is different- instead of the "bucket" being chosen as a numeric hash of the message number, the "bucket" is chosen as the first bucket which isn't full at the time the domain or mailbox is created.

You may think it makes sense to just compile vpopmail without support for hashing... and if you are absolutely sure that the server will never have any major load (i.e. it will never have more than about 40 domains, AND it will never have more than 40 mailboxes in any one domain) then you're probably right. However, even on a "small" server like this, the extra hashing code only adds a few micro-seconds to the time it takes to create a domain or mailbox, and does not affect the time it takes to authenticate or access the contents of a mailbox at all. To me it makes sense to just build in support for hashing (which is the default) and not worry about it.

The directions below will work regardless of whether or not each server supports hashing.

Migration Methods

There are two primary ways to do a vpopmail-to-vpopmail migration: moving ALL domains from the old server to a new server which has no domains yet, or moving one domain at a time. Moving all domains at once is a lot easier, however you can only do it this way if ALL of the following are true:

• You need to move ALL mailboxes in ALL domains from the old server to the new server, rather than just a subset of them.

• The new server has the same software (i.e. qmail, vpopmail, courier-imap, etc.) as the old server.

• The physical directories used by qmail and vpopmail (i.e. /var/qmail directory and the home directory of the vpopmail user) are in the same locations (i.e. if the vpopmail user's home directory is "/home/vpopmail" on the old server, it should also be "/home/vpopmail" on the new server.)

• There are currently NO mailboxes on the new server.

If ALL of these items are true, you can, and should, move everything at once. Otherwise, you will need to move the mailboxes one domain at a time, even if you will end up moving all of the mailboxes by the time you're finished.

Of course, the other option is to do it all by hand- move the mailbox files and manually edit the various qmail and vpopmail control files by hand. However, this requires you to fully understand how both qmail and vpopmail work- which is a lot more detail than I'm going to put up on a web page. If you have that kind of knowledge, it's an honour to know that you are reading my web site, Dr. Bernstein.

Full Server Migration (aka Backup and Restore)

This method copies the entire qmail and vpopmail configuration of the old server to the new server. This will totally replace the qmail and vpopmail configuration on the new server, so the new server should not have any kind of existing configuration (i.e. no existing domains.)

Doing the migration is actually rather simple. In fact, I've already written a web page explaining what needs to be backed up on a qmail system. Basically, do a backup, restore the backup to the new machine, and you're done.

Single Domain Migration

This method is a bit more complicated, because you have to deal with hashing and the fact that the directories will probably not be exactly the same on the old and new machines.

This kind of migration could be done by just plain backing up the domain's directory and then restoring it on the new machine, but then you also have to manually adjust qmail's virtualdomains, rcpthosts, and users/assign files, remember to run qmail-newu, and send a HUP to qmail-send... and unless you know exactly what you're doing while editing these files, you can very easily break your server.

When I've had to do this in the past, I ended up writing a script which writes a script, while making tarballs of each individual mailbox. The generated script and the tarballs are then copied to the new server, and when you run that generated script, it creates the domain and each mailbox using the same encrypted password and quota from the old server, and then expands the mailbox contents into place.

I don't have any of the actual scripts I wrote in the past- they were always specific things for that one client. However, I do remember the basic ideas behind the process, which allowed me to write...

The migrate-domain script

This is a newly written script which implements the same basic idea as the custom scripts I've written for clients in the past. I'm pretty sure this works- I've tested the script itself, but I haven't had a chance to test the script it generates, which would restore the files- and that's only because I don't have a test machine to do it on at the moment.

The script creates the tarballs for each mailbox, and generates the shell script for the new server. The script expects to be called with one domain name on the command line.

Limitations

The script will not work correctly with vpopmail's mysql back-end, for the following reasons:

• There is currently no way to "feed" alias lines into the database, other than the "valias -i" option. In theory this should work, however my own testing ran into problems with alias lines containing single quotes, because

• The mysql back-end has no provision to store a proper sequence of lines within an alias. For example, you might have an alias which, if stored in a .qmail file, looks like this:

  |condredirect nospam@domain.zzz /usr/local/bin/spamcheck
  ./userid/Maildir/

  The idea is this:

  • The "spamcheck" program examines the message and returns 0 if it's spam, or 1 if it's clean. (This is an example, I haven't actually written such a script.)
  • The "condredirect" program forwards the message to the "nospam@domain.zzz" address if it's spam, or does nothing (and allows qmail-local to process the next line of the .qmail file) if it's clean.
  • The "./userid/Maildir/" line causes the message to be delivered to that user's normal inbox.

  Because there is no explicit sequencing information stored in the database, the mysql server may return these two lines in either order- and if the "./userid/Maildir/" line happens to be returned first, the desired filtering never happens.

Because of these limitations, the script cannot be made to work with the mysql back-end under current versions of vpopmail. These problems will be resolved in a future vpopmail version 6.

Another problem

Support for autoresponders and mailing lists should be working, but has not been fully tested yet. Much as I hate to admit it, I discovered the hard way that it didn't work, when I migrated my own domains from one server to another and lost an autoresponder and a mailing list from my own "jms1.net" domain, as well as two mailing lists from a client's domain.

If anybody is able to fully test this with autoresponders, mailing lists, or both, I would appreciate hearing about your results.

2008-06-18 I was moving some domains from one server to another for a client. The machines had different home directories for the vpopmail user, and I happened to catch a bug I hadn't seen before- the home directory from the old machine somehow ended up being written into the script and used on the new machine. Turns out I just forgot to add a "\" before one of the variables in the code which creates aliases.

If you have a copy of the script, you should download the newer version and start using it instead.

Download

Using the script

• Create an empty directory, and "cd" into it. This directory should be on a filesystem which has enough free space to hold the contents of all of the mailboxes you will be migrating.

• Run the script, with the domain you wish to back up on the command line.

  # migrate-domain domain.zzz
  Writing script header
  Backing up postmaster@domain.zzz
  Backing up mailbox alan@domain.zzz
  Backing up mailbox carla@domain.zzz
  Backing up mailbox info@domain.zzz
  Backing up mailbox sales@domain.zzz
  Backing up mailbox tony@domain.zzz
  Backing up .qmail files

  This creates a script for the domain and a tarball for each mailbox.

  # ls -1
  domain.zzz-alan.tar.gz
  domain.zzz-carla.tar.gz
  domain.zzz-info.tar.gz
  domain.zzz-postmaster.tar.gz
  domain.zzz-sales.tar.gz
  domain.zzz-tony.tar.gz
  domain.zzz.qmail.tar.gz
  domain.zzz.sh

  Feel free to look at the script it generated- it's just a normal shell script, nothing overly complicated about it.

• Copy the files to a directory on the new machine.

• On the new machine, run the script.

  # sh domain.zzz.sh
  Creating domain and postmaster@domain.zzz
  Creating mailbox alan@domain.zzz
  Creating mailbox carla@domain.zzz
  Creating mailbox info@domain.zzz
  Creating mailbox sales@domain.zzz
  Creating mailbox tony@domain.zzz
  Creating .qmail files

Copyright 1995-2024 John M. Simpson KG4ZOW (email) (PGP key)
Last updated 2008-08-08 03:43:26 +0000