NFS Server

Summary

The NFS (Network File System) server contains a large quantity of hard drives configured in a RAID 5 array. This array is then shared over the network using the NFS protocol.

These are the instructions for configuring one that I made at work in March 2007, so you may have to modify them slightly as software changes fast.

Specs

• OS: CentOS 4.4 Server
• Motherboard: ASUS A8N32-SLI Deluxe (Socket 939)
• CPU: AMD Athlon 64 4000+ San Diego 2.4GHz Socket 939 (Model ADA4000DKA5CF)
• Memory: WINTEC AMPO 1GB (2 x 512MB) 184-Pin DDR 400 (PC3200) SDRAM Dual Channel
• NIC: On-board dual gigabit NIC
• Controller card: Promise Technology SATA300 TX2plus/TX4
• CD drive: ASUS Beige E-IDE/ATAPI CD-ROM Drive (CD-S520/A5)
• Hard disks:
• (1) Maxtor DiamondMax 20 STM3802110A 80GB 7200 RPM IDE (ATA100)
• (7) Western Digital Caviar SE16 WD3200KS 320GB 7200 RPM SATA3
• Power supply: 500W

Network

• Connected to internal network as 172.16.2.50
• Connected to cluster network as 192.168.13.50

Maintenance

Recovering from Disk Failure

Replacing failed disk drives is not optional! Replace failed drives immediately!

A failed RAID array disk drive must be replaced immediately. Assuming the array has 1 spare drive configured, the array can only handle 2 hard drive failures before suffering permanent and total data loss.

The array is generally smart enough to automatically activate a spare drive.

Simulating Drive Failure

Simulating drive failure is easy:
mdadm –manage –set-faulty /dev/md0/ /dev/sdd1

The array should immediately start the recovery process, which will take an hour or so.

To “fix” the failed drive, use the following:
mdadm –remove /dev/md0 /dev/sdd1
mdadm –add /dev/md0 /dev/sdd1

Check Health and Determine Source of Failure

mdadm is your friend. This little buddy will tell you how your array is setup, and help you diagnose problems, reconfigure the array, or recover from failures. mdadm‘s commands have many aliases, but I’ll use the more verbose ones to make things easier to remember/understand.

The following command will show you the status of your array, assuming your array is /dev/md0:
mdadm –detail /dev/md0

There are several important pieces of information displayed here. The first is state. In a healthy environment, it will display clean. If the array hasn’t fully initialized yet, is recovering from a failure, or has had a failure, it may show degraded or recovering.

Individual disk health can be viewed further down.

Remove the failed drive from the array:
mdadm –remove /dev/md0 /dev/sdd1

The tough part is figuring out which of the drives it actually is inside of the computer. Good luck.

Once the faulty drive is replaced with the new drive, follow the instructions in the Adding an Additional Drive section for setting up the disk’s file system and adding it to the array. Because the disk will be added as a spare, do not follow the instructions to grow the array.

Adding an Additional Drive

Note: Adding an additional active drive to the array requires a custom kernel. If the server hasn’t been configured with the custom kernel, these instructions will not work.

Plug the drive in, and boot the computer up. Since the drive hasn’t been used yet, you’ll need to partition it. To do this, we use fdisk.

First, we need to know which hard drive needs worked on. To find this out, use fdisk to get a list of the drives connected to the computer:
fdisk -l

The drive will likely be listed as /dev/sd?, where the ? is a letter. It will not have any Linux raid autodetect partitions listed.

To add the Linux raid autodetect partition, use the following (replace /dev/sde with the drive you want to modify):
fdisk /dev/sde

Create the partition:
n
p
1
[Enter]
[Enter]

Change the partition type to Linux raid autodetect:
t
fd

Write the changes:
w

The drive has to be added as a spare, and we accomplish this by using mdadm. We specify which array to add to, and which partition we want added:
mdadm –add /dev/md0 /dev/sde1

To activate spare drives, we again use mdadm. We specify which array to activate the drives on, and how many active drives we want:
mdadm –grow /dev/md0 –raid-devices=6

Reshaping the drive will take a very very very long time (hours, possibly even days) and will likely take up considerable CPU power. It is highly recommended that this activity takes place over the weekend so that there is minimal impact on customers.

After reshaping the drive, we need to enlarge the file system to fill the new space. For safety, we check the existing file system for errors first using e2fsck, and then use resize2f2 to actually resize the file system. Unfortunately, these commands cannot be run on a mounted file system, and so we have to temporarily unmount the array, and then remount it after.
umount /dev/md0
e2fsck -f /dev/md0
resize2fs /dev/md0
mount /dev/md0

From my experience, this process takes around 20 minutes.

Step-by-step Setup Initial Setup Instructions

Following these instructions should get you up and running in just a few hours.

None of these steps are optional! Failure to follow these instructions will likely result in a botched NFS server that only half-works.

Initial OS Setup

Obtain and Install CentOS

Our NFS server is currently using CentOS 4.4 Server. Newer versions should probably work fine and have nearly identical installation procedures. Go to their webpage, download the ISO, burn it to disk, and boot off of it.

When the prompt comes up, hit the enter key to start the graphical installer interface.

You will be prompted as to whether you want to test the CD or not. If you haven’t used this disk before then you should probably test it first. To skip the test, choose Skip.

Once the graphical installer starts, choose Next, Next, and Next again, until you reach the Disk Partitioning Setup page.

Choose Manually partition with Disk Druid, and click Next. Delete any existing partitions.

Warning! This will obviously destroy any data currently on any of the disks, including the RAID array. If your RAID array is fine, then do not delete the RAID partitions or modify them in any way. Only delete/recreate the primary drive partitions. Re-specify /raid as the array’s mount point.

Create the new partitions on the primary hard drive:

• 200MB /boot ext3 partition on hda
• 2048MB swap partition on hda
• Fill to maximum allowable size / ext3 partition on hda

Create a new partition on each RAID disk one at a time:

• Fill to maximum allowable size software RAID on each RAID disk, but only do one at a time (sda, then sdb, then sdc, etc.)

Configure the RAID array:

• Click RAID. Choose Create a RAID device [default=/dev/md0]. and click OK.
• Change Mount Point to /raid.
• Leave File System Type as ext3.
• Leave RAID Device as md0.
• Change RAID Level to RAID5.
• Leave all RAID Members checked. If one of your array drives is missing, then you messed up when creating the partitions.
• Change Number of Spares to 1.
• Click OK.

Click Next, then Next again.

At the Network Configuration page, select both network devices to be active on boot. For each device, click Edit. Uncheck Configure using DHCP. For the first device, use 172.16.2.50 for the IP, and 255.255.255.0 for the netmask. For the second device, use 192.168.13.50 for the IP, and 255.255.255.0 for the netmask. Click Ok.

Choose to manually set the hostname, and enter nfs.

Enter the Gateway, Primary DNS, Seconday DNS, and then click Next:

• Gateway: 172.16.2.1
• Primary DNS: 66.195.216.131
• Secondary DNS: 66.195.216.132

Choose No Firewall, and disable SELinux. Click Next. Click Proceed.

Click Next. Make sure the timezone is correct, check System clock uses UTC, and click Next again.

Enter the root password. Click Next.

Select Customize software packages to be installed, and click Next.

De-select everything but Editors, Server Configuration Tools, and Development Tools. Click Next.

Click Next to start the installation. This will take awhile.

Click Reboot.

Update the System

There are probably a ton of updates for the OS and the tools included with the OS. We want to update these before we go any further, otherwise we might run into known bugs.

At this point you need to be connected to the network and have access to the internet. Log into the system as root and then run the following command:
yum -y update

It will take awhile (~30 minutes) to download and install the updates.

Note: This will automatically apply the 2007 Daylight Saving Time update, if needed.

We want a few extra packages that don’t come pre-installed:
yum -y install gcc ncurses-devel rpm-build

We need to update mdadm using the latest tarball from http://www.cse.unsw.edu.au/~neilb/source/mdadm/:
cd /usr/local/src
wget http://www.cse.unsw.edu.au/~neilb/source/mdadm/mdadm-2.6.1.tgz
tar -zxvf mdadm-2.6.1.tgz
cd mdadm-2.6.1
make
make install

We also want the slocate database to be updated automatically. Edit /etc/updatedb.conf and change DAILY_UPDATE=no to DAILY_UPDATE=yes.

This doesn’t really fit here, but we don’t want logwatch running (because it is annoying). Use the following command to remove the symlink that allows it to run:
rm /etc/cron.daily/00-logwatch

Crontab

Add appropriate crontab entries using crontab -e:
0 */1 * * * /bin/rm /raid/vpopmail/domains/a4isp.com/postmaster/Maildir/new/* -f && /bin/rm /raid/vpopmail/domains/a4isp.com/postmaster/Maildir/maildirsize -f

Explanation:

• First line empties the postmaster@a4isp.com mail account and deletes the maildirsize file. This prevents this account for overfilling from bounced messages and the like, as the default qmail aliases point to it.

Set the Time

First, you need to tell Linux what timezone you are in. Symlink the localtime file to the timezone file that is appropriate for the server:
ln -fs /usr/share/zoneinfo/US/Eastern /etc/localtime

Next, you need to tell Linux what timezone you are using, and that your hardware clock is set to UTC. Edit /etc/sysconfig/clock so it looks like the following:
ZONE=”US/Eastern”
UTC=true
ARC=false

Reboot and enter the BIOS. Visit http://www.time.gov/timezone.cgi?UTC/s/0/java to get the correct UTC time, and update it in the BIOS. Save and finish booting.

Verify that Linux displays the correct local date/time using the following command:
date

OpenNTPD

We need the server to have the correct time, all the time. This handy utility makes sure that we do.

Download, install, and symlink config file:
cd /usr/local/src/
wget ftp://ftp.openbsd.org/pub/OpenBSD/OpenNTPD/openntpd-3.9p1.tar.gz
tar zxvf openntpd-3.9p1.tar.gz
cd openntpd-3.9p1
./configure
make && make install
ln -fs /usr/local/etc/ntpd.conf /etc/ntpd.conf

Create /etc/init.d/ntpd:
#!/bin/bash
#
# ntpd This shell script runs OpenNTPD.
#
# chkconfig: 2345 80 30
# description: Keeps time accurate.

/usr/local/sbin/ntpd

Give permissions:
chmod a+x /etc/init.d/ntpd

Create group/user:
groupadd _ntp
useradd -g _ntp -s /sbin/nologin -d /var/empty/ -c ‘OpenNTP daemon’ _ntp

Custom Kernel

Compiling the Custom Kernel

We want a custom kernel that can handle more advanced RAID5 configuration options (such as growing an array).

Get the latest kernel source from http://www.kernel.org/pub/linux/kernel/v2.6/:
cd /usr/src
wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.20.tar.gz
tar -zxvf linux-2.6.20.tar.gz

Symlink linux to the Linux source directory:
ln -s linux-2.6.20 linux
cd linux

Copy configuration file from current kernel to new kernel:
make clean && make mrproper
cp /boot/config-`uname -r` ./.config

Load configuration utility:
make menuconfig

Scroll down to Load an Alternate Configuration File and type .config. Choose Ok.

Go to General setup, and change the Local version – append to kernel release to raid5-reshape. Select Exit.

Go to Device Drivers, then Multi-device support (RAID and LVM). Ensure RAID-4/RAID-5/RAID-6 mode and Support for adding drives to a raid-5 array are selected with an asterisk (*) by using the space bar. Choose Exit, and then Exit again.

Go to Device Drivers, then Serial ATA (prod) and Parallel ATA (experimental) drivers. Ensure ATA device support is selected with an asterisk (*) by using the space bar. Also select NVIDIA SATA support (NEW) and Promise SATA TX2/TX4 support (NEW). (These options may be different if you are using different hardware). Choose Exit, and then Exit again.

Go to Processor type and features, then Processor family. Scroll down to Opteron/Athlon64/Hammer/K8, and select it. This option will obviously be different if you are using a different CPU. Choose Select. Unselect Toshiba Laptop support and Dell laptop support.

Choose Exit, and then Exit again. Answer yes to any prompts to save the configuration file.

Next we actually build the kernel. This may take a long time (hours).
make rpm

Installing the Custom Kernel

Change to the directory that the kernel RPM is in, and then install the RPM:
cd /usr/src/redhat/RPMS/i386/
rpm -ivh –nodeps kernel-2.6.20raid5reshape-1.i386.rpm

If this is your second (or third or fourth) attempt at making the custom kernel, you may need to uninstall your previous attempts first:
rpm -e kernel-2.6.20raid5reshape-1.i386

Next we need to create a ramdisk for it:
mkinitrd –builtin=sata_nv –builtin=raid5 –builtin=sata_promise /boot/initrd-2.6.20raid5-reshape.img 2.6.20raid5-reshape

Once the kernel is installed, we need to configure the boot loader.
vi /boot/grub/menu.lst

Add the following above the first title CentOS line and after hiddenmenu:
title CentOS (2.6.20-raid5-reshape)
root (hd0,0)
kernel /vmlinuz-2.6.20raid5-reshape ro root=LABEL=/
initrd /initrd-2.6.20raid5-reshape.img

Save and reboot.

On the first reboot, Kudzu may tell you that the nVidia Corporation CK804 Ethernet Controller is no longer there. Choose Keep Configuration, or eth0 won’t work.

NFS Configuration

Assuming you selected the correct options during the CentOS installation, this should be easy.

First, we want to make sure the proper services are set to run automatically and are actually running.

To see if portmap is set to run automatically, use this command:
chkconfig –list portmap

It will return something like this:
portmap 0:off 1:off 2:off 3:on 4:on 5:on 6:off

0 through 6 represent run levels, and whether the item is set to run at that run level or not.

3, 4, and 5 should be on. If they aren’t, run this command to turn them on:
chkconfig portmap on

You also need to check if nfs runs automatically:
chkconfig –list nfs

It will return something like this:
nfs 0:off 1:off 2:on 3:on 4:on 5:on 6:off

3, 4, and 5 should be on. If they aren’t, run this command to turn them on:
chkconfig nfs on

Make sure /var/lib/nfs/v4recovery exists:
mkdir /var/lib/nfs/v4recovery

Now we want to restart portmap and nfs to make sure everything is okay:
service portmap restart
service nfs restart

Always restart portmap before nfs. I don’t know why, but all the websites say it is important.

You should get something like this:
[root@localhost ~]# service portmap restart
Stopping portmap: [ OK ]
Starting portmap: [ OK ]
[root@localhost ~]# service nfs restart
Shutting down NFS mountd: [ OK ]
Shutting down NFS daemon: [ OK ]
Shutting down NFS quotas: [ OK ]
Shutting down NFS services: [ OK ]
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS daemon: [ OK ]
Starting NFS mountd: [ OK ]
[root@localhost ~]#

It is okay if the shutting down items fail, but all of the starting items MUST display OK. If they don’t, then you likely have a problem with your /etc/exports file. But you’ve been a good direction follower and haven’t touched that file yet, right? ;o)

Next we need to edit the /etc/exports file to specify which directory to share. Add the following to your /etc/exports file:
/raid 172.16.2.0/255.255.255.0(rw,no_wdelay,sync,no_root_squash) 192.168.13.0/255.255.255.0(rw,no_wdelay,sync,no_root_squash)

Explanation:

• /raid specifies which local directory we are going to be sharing.
• 172.16.2.0/255.255.255.0/192.168.13.0/255.255.255.0 allows all computers in the 172.16.2.0-172.16.2.254/192.168.13.0-192.168.13.254 range to use the NFS share.
• rw allows read-write access.
• no_wdelay requires data to be written to disk as soon as possible. This decreases performance, but significantly reduces the risk of lost data in the event of a server crash/reboot.
• sync causes replies to the NFS request to be sent only after all data has been written to disk. This decreases performance, but significantly reduces the risk of lost data.
• no_root_squash allows root on the client to have root access to the NFS share. This is a huge security problem if a public network has access to the NFS share.

We also want to add allowed hosts to /etc/hosts.allow:
portmap: 172.16.2.0/255.255.255.0, 192.168.13.0/255.255.255.0
lockd: 172.16.2.0/255.255.255.0, 192.168.13.0/255.255.255.0
rquotad: 172.16.2.0/255.255.255.0, 192.168.13.0/255.255.255.0
mountd: 172.16.2.0/255.255.255.0, 192.168.13.0/255.255.255.0
stad: 172.16.2.0/255.255.255.0, 192.168.13.0/255.255.255.0

The IP range should match the IP range used in /etc/exports.

Restart portmap and nfs:
service portmap restart
service nfs restart

Mounting NFS on the Client

Linux

Mounting the NFS array on a Linux client is ridiculously easy.

First, create the mount point on the client:
mkdir /raid

Add the following to the end of /etc/fstab to cause the share to be mounted at boot:
192.168.13.50:/raid /raid nfs rw,bg,intr,hard 0 0

Explanation:

• 192.168.13.50 is the IP of the NFS server, and the first /raid is the NFS share on the NFS server.
• The second /raid is where to mount the device on the client.
• nfs specifies which file system the device is using.
• rw puts the device in read-write mode.
• bg tells the client to attempt to reconnect the device in the background if the connection is lost.
• intr allows applications to continue running on the NFS device even if the connection is temporarily lost.
• hard keeps requests alive, even if the server goes down. This has the advantage of allowing the client to pick up where it left off.
• The two zeros signify that the drive doesn’t need to be dumped or checked.

The device will automatically mount at boot. To mount the device immediately without rebooting, use the following command:
mount /raid

Windows

Windows Vista Ultimate

Windows Vista Ultimate has native NFS and ext2/ext3 support. Simply enable them in your Windows configuration and then map the drive.

MySQL Server Install

This must be done:
mysql
grant all privileges on *.* to vpopmail@’%’ identified by ‘password’;
grant all privileges on *.* to vpopmail@localhost identified by ‘password’;
create database vpopmail;
quit