thoughts from two guys

So I'm attempting to prep a build box for Centos 5.5 and ZFS+ZPL testing.

First off we are going to run the "kernel-ml" or mainline kernel on the system which is part of the fine EPEL repository.

rpm -Uvh http://elrepo.org/elrepo-release-5-2.el5.elrepo.noarch.rpm
yum --enablerepo=elrepo-kernel install kernel-ml kernel-ml-devel

Reboot and we're in 2.6.37 land. HOLD UP: THAT MAY BE TOO NEW.

Linux localhost.localdomain 2.6.37-0.3.el5.elrepo #1 SMP PREEMPT Mon Feb 7 17:31:48 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

Then we are downloading and installing the development versions from the ZFS site.

Then we build them.

mkdir zfs
cd zfs
(download from zfs developer version)
tar zxvf behlendorf-spl-spl-0.5.2-23-g22ccfaa.tar.gz
cd behlendorf-spl-22ccfaa/
./configure 
make install
modprobe spl

So far so good:

SPL: Loaded Solaris Porting Layer v0.6.0

On to the ZFS module itself.

tar zxvf behlendorf-zfs-zfs-0.5.2-69-g2c395de.tar.gz 
cd behlendorf-zfs-2c395de/
./configure

We hit a few issues with missing defines for the zfs build so we hack around it with these defines.

Added to deal with undefined items on line 310 of libzfs_mount.c:

#define MNT_FORCE       0x00000001      /* Attempt to forcibily umount */
#define MNT_DETACH      0x00000002      /* Just detach from the tree */
#define MNT_EXPIRE      0x00000004      /* Mark for expiry */

Later in modules/zfs we seem to hit a missing kernel API routine invalidate_inodes. We work around it and vow to do further research

modprobe zfs
ZFS: Loaded ZFS Filesystem v0.6.0

OK, now attaching a disk.

zpool create data01 sdb1
# zpool list
NAME     SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
data01    37G    81K  37.0G     0%  1.00x  ONLINE  -
# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             37690316   2006484  33738364   6% /
tmpfs                   511360         0    511360   0% /dev/shm
data01                38190976         0  38190976   0% /data01
# zpool upgrade -v
This system is currently running ZFS pool version 28.

The following versions are supported:

VER  DESCRIPTION
---  --------------------------------------------------------
 1   Initial ZFS version
 2   Ditto blocks (replicated metadata)
 3   Hot spares and double parity RAID-Z
 4   zpool history
 5   Compression using the gzip algorithm
 6   bootfs pool property
 7   Separate intent log devices
 8   Delegated administration
 9   refquota and refreservation properties
 10  Cache devices
 11  Improved scrub performance
 12  Snapshot properties
 13  snapused property
 14  passthrough-x aclinherit
 15  user/group space accounting
 16  stmf property support
 17  Triple-parity RAID-Z
 18  Snapshot user holds
 19  Log device removal
 20  Compression using zle (zero-length encoding)
 21  Deduplication
 22  Received properties
 23  Slim ZIL
 24  System attributes
 25  Improved scrub stats
 26  Improved snapshot deletion performance
 27  Improved snapshot creation performance
 28  Multiple vdev replacements
#

Time to break some data!

So in case you've not heard, Oracle made some changes to the way OpenSolaris was being released that basically for practical sakes killed that branch of open source Solaris and with it our primary way of doing production level ZFS. For lower cost anyway.

So we've been tracking these developments very closely in looking for a solid ZFS solution for what we use it for: dedupe and backups of our cluster filesystems.

FreeBSD version v28 ZFS has been added to 8-STABLE as a patch and so far seems pretty solid. Some outstanding work for those that want a quick install (and even root filesystems on ZFS) is located here .

For those that like to build their own, the patch area listed there should be tracked against the SVN checkout of FreeBSD or it doesn't quite apply properly.

But the cool part is after those patches you do indeed get a v28 capable ZFS in 8-STABLE.

bsd81# zpool upgrade -v
This system is currently running ZFS pool version 28.

The following versions are supported:

VER  DESCRIPTION
---  --------------------------------------------------------
 1   Initial ZFS version
 2   Ditto blocks (replicated metadata)
 3   Hot spares and double parity RAID-Z
 4   zpool history
 5   Compression using the gzip algorithm
 6   bootfs pool property
 7   Separate intent log devices
 8   Delegated administration
 9   refquota and refreservation properties
 10  Cache devices
 11  Improved scrub performance
 12  Snapshot properties
 13  snapused property
 14  passthrough-x aclinherit
 15  user/group space accounting
 16  stmf property support
 17  Triple-parity RAID-Z
 18  Snapshot user holds
 19  Log device removal
 20  Compression using zle (zero-length encoding)
 21  Deduplication
 22  Received properties
 23  Slim ZIL
 24  System attributes
 25  Improved scrub stats
 26  Improved snapshot deletion performance
 27  Improved snapshot creation performance
 28  Multiple vdev replacements

We are doing heavy testing to make sure its reliable and watching for a formal commit.

On the Linux ZFS front we continue to watch closely the efforts of the native kernel module. While we have no issue with the FUSE approach, we like to have as many options as we can. We have used the FUSE modules for ZFS successfully.

That work continues at the primary release site and the ZPL portion was recently added to a development branch. We are testing that as well under Centos based kernels.

Finally, we watch two OpenSolaris based spin-offs just to cover all the bases:

OpenIndiana

And Nexenta's release of the core OS.

Nexenta CP

Despite the fact it appears we are very quiet here its due to being busy. But I thought I'd write up a few battles of late in the hopes some find them interesting! Stay tuned!

So I used to use virtual PXE booted floppies to handle various tools that my cluster nodes required to update firmware and other misc items that often come as DOS based programs. This worked well, but I got lazy and didn't track the fine features of MEMDISK from the SYSLINUX closely enough when most of these tools started showing up on ISO images.

So I missed that they recently added ISO booting support to MEMDISK. And thanks to a very well written article I've moved a pile of my common "utility" ISO images into a PXE bootable menu. (And not all of these have to do with clusters below)

So here is what I did:

• Go get the latest version of Syslinux just to be sure
  

• Place the latest pxelinux.bin into your tftp server which you probably already have and add "memdisk" to the same location

• Also copy the menu.c32 file. that provides a basic menu system for PXE boots

Then, in your pxelinux.cfg area I've made a file called "chooser" which when I want to boot a system off that config, I symlink the needed IP address file to it. It contains right now:


UI menu.c32
DEFAULT bios

LABEL bios
MENU LABEL FreeDOS BIOS Tool ISO
KERNEL memdisk
APPEND iso initrd=isos/bios.iso

LABEL ntchpass
MENU LABEL NT Change Password ISO
KERNEL memdisk
APPEND iso initrd=isos/ntchpass.iso

LABEL bartpe
MENU LABEL Bart PE ISO
KERNEL memdisk
APPEND iso initrd=isos/bartpe.iso

LABEL diskzapr
MENU LABEL Diskzapr
KERNEL memdisk
APPEND iso initrd=isos/diskzapr.iso


LABEL seagate
MENU LABEL Seagate Firmware Disk ES ST31000340NS Series SN06
KERNEL memdisk
APPEND iso initrd=isos/ES2SN06B-GOCHECK.iso



The first one is where I've been collecting various BIOS flashing tools for quite some time. Its a FreeDOS based disk with just a set of directories sorted by vendor and board name for the various tools. It was better than trying to mash all the tools together to have the room of the ISO image.

That last one, DO NOT just blindly deploy Seagate Firmware updates for the recent problems without fully reading the Seagate KB article on which ones are right for your drive. I've changed the ISO name to prevent that.

So I've spent a few weeks catching up on the latest in a slightly different type of "cluster": Highly Available clusters via the ongoing project formerly known as Heartbeat.

Pacemaker

I was able to get some of the basic stuff going quite easily under the new CRM system (compared to the very easy to use Heartbeat config file of the good old days) and even some basic Libvirt redundant KVM based system moving around. Not live migration of active network connections, but rapid relocation of a KVM node based on a shared backend storage.

The goal for this is a paired set of redundant servers that provide "utility" functions to a cluster. Be it an Kickstart install server, a Ganglia server, etc.