MIX and NFS

[Andrew Daviel]

Now that Mark sadly is no longer with us, maybe I can raise this without
getting my head bitten off.

We are using UW imapd 2007 with MIX format mailboxes.
Our storage expert would really like to use an NFS-mounted volume on a
native ZFS appliance, saying it's much better for snapshots and
block-oriented backups than iSCSI.

Given that we are only trying to support one flavour of Unix (Linux), and
possibly using NFS4 on Centos6/Linux 2.6, is there still a problem with
NFS, if the only processes opening mailboxes are using c-client libraries
(dmail, imapd, mailutin, alpine) ?

I have read some of the copious texts about Sys V and flock() and am not
sure what applies to Linux.

[Oscar del Rio]

Post by Andrew Daviel
We are using UW imapd 2007 with MIX format mailboxes.
Our storage expert would really like to use an NFS-mounted volume on a
native ZFS appliance, saying it's much better for snapshots and
block-oriented backups than iSCSI.
Given that we are only trying to support one flavour of Unix (Linux),
and possibly using NFS4 on Centos6/Linux 2.6, is there still a problem
with
NFS, if the only processes opening mailboxes are using c-client
libraries (dmail, imapd, mailutin, alpine) ?

We had some users with NFS-mounted mailboxes (~/Mail) and MIX format.
Every once in a while there were some corrupted folders even though the
only access was through one imapd server.
We never got corrupted folders on local storage.

If you are planning to change storage, maybe it is also time to consider
switching to another server.
We recently moved to dovecot with mdbox format, which is similar to
Mark's MIX format (but not compatible).

[Andrew Daviel]

Post by Oscar del Rio

Post by Andrew Daviel
We are using UW imapd 2007 with MIX format mailboxes.
Our storage expert would really like to use an NFS-mounted volume on a
native ZFS appliance, saying it's much better for snapshots and
block-oriented backups than iSCSI.

Thanks Oscar for your answers

I have been looking at this again, as our storage guy said "Oh but Linux
supports the local_lock switch in NFS mount".
Yes it does - from Linux 2.6.37. But I found a patch to backport that to
2.6.18, which is coincidentally what I am running on CentOS 5.

So I've been playing. I ran mailutil to repeatedly append and prune a
mailbox on NFS, while dmail keeps delivering mail. Sure enough, after a
few minutes, I get mailbox corruptions.
Then I built a patched nfs_utils and nfs.ko module, and tried mounting the
disk with e.g. mount -t nfs -o rw,nosuid,local_lock=all

UW imap defines safe_flock in c-client/flocklnx.c, which makes flock()
a no-op, citing https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=123415
But that's from 2005, talking about Linux 2.4.21.
http://en.wikipedia.org/wiki/File_locking says that
"Kernel 2.6.12 and above implement flock calls on NFS files using POSIX
byte-range locks" and the kernel code fs/nfs/file.c says
"We're simulating flock() locks using posix locks on the server. No BSD
flocks over NFS allowed.' Seems to me flock() works now. The flock shell
command works fine on an NFS volume (flock lockfile bash). In spite of the
fact the manpage for flock() (2) still says it doesn't.

So I commented out the no-op in safe_flock, and my test setup works fine -
no mailbox corruption - even with local_lock=none. However, it does
occasionally hang for a bit - it may be triggering the 5-second sleep in
safe_flock() although I don't see any syslog messages. Or it may be the
NFS server itself.

If I set local_lock=all, again I get no corruption, and I don't get any
hangs either.

I haven't tried exhaustively yet - running the test for hours - or a
proper mailtest suite.

Post by Oscar del Rio
If you are planning to change storage, maybe it is also time to consider
switching to another server.
We recently moved to dovecot with mdbox format, which is similar to Mark's
MIX format (but not compatible).

I found e.g.
http://www.dovecot.org/list/dovecot/2012-June/066291.html
http://wiki.dovecot.org/NFS
which implies Dovecot may not be 100% robust on NFS.