====== Debian ======

These instructions document the installation and configuration of Debian 4.0 on our servers.

For installation and configuration of Debian 5.0 on RedHook, see [[build/debian/redhook | Debian/RedHook]]

===== Installation =====

We downloaded and burned the Debian 4.0r3 netinst CD image. We then booted each system with the CD-ROM in the drive. We hit **Enter** at the boot prompt to accept the default installation mode. We then proceeded with the installation as follows. This initial installation process will take about 30 minutes.

  - Select **English** as the language.
  - Select **United States** as our location.
  - Select **American English** keymap.
  - Select **eth0** as Primary network interface.
  - DHCP will fail, so choose **Continue**, then **Configure network manually**.
    * IP address: 206.196.99.162 (for Bud), 206.196.99.163 (for BudLight)
    * Netmask: 255.255.255.240
    * Gateway: 206.196.99.161
    * Nameserver: 205.242.92.2 (for Hostirian/Primary.Net)
    * Hostname: bud (or budlight)
    * Domain name: sluug.org
  - Select **Manual** partitioning method.
    * See below for details.
    * On 7/30/2007 re-install, chose to keep existing partition table.
      * Had to set mount points and FS types for each partition.
      * Kept data on ''/home'' and ''/usr/local''; reformatted the rest.
    * On 3/2/2008 re-install, chose to keep existing partition table.
      * Formatted all file systems.
    * Write changes to disk when completed with partitioning.
  - Select **Central** time zone.
    * In the extended list, this is labeled as **America/Chicago**.
    * Was not asked on re-install, but we decided to set the system clock to GMT.
  - Set a new root password when prompted.
  - Create the first user when prompted.
    * Use a temporary account named **build**, so that account does not get confused with any real user accounts.
  - Set the options for APT packages.
    * Use a network mirror? **Yes**
    * Country: **United States**
    * Site: **ftp.us.debian.org**
    * Accept blank entry for the HTTP Proxy information.
  - If prompted to participate in package popularity contest, say **No**.
  - Select **Standard system** software to install, and unselect everything else.
  - Skip the package installation, if asked.
  - Say **yes** when asked to install GRUB boot loader to the master boot record.
  - The CD ejects at this point.
  - Hit **Continue** to reboot the system.

===== Partitioning =====

We had 36 GB to work with on each system. The systems came with 4 x 18 GB drives. With 3 of the drives in a RAID 5 array, and 1 hot spare drive, this gave us 36 GB of usable storage.

After some debate, we decided upon the following partition setup.
The table lists the partitions in the order they were created.

^Mount Point^Size^Notes^
|/boot     |  100 MB|Primary partition, bootable |
|/         |    1 GB|Primary partition |
|swap      |    2 GB|do not mount |
|/home     |    5 GB|  |
|/var      |   10 GB|Reiser |
|/usr      |    4 GB|  |
|/usr/local|    3 GB|  |
|/tmp      |    1 GB|  |
|/spare    |    10 GB|Left-over space |

All file systems were formatted as ext3, except for ''/var''. 
We formatted ''/var'' with reiserfs, because we expected to use Maildir format.
Maildir format uses a separate file for each email message, 
which ends up creating lots of small files within a single directory.
Reiserfs is much more efficient at storing small files and having a large
number of files within a directory.

Note that the Compaq system and RAID utilities are on primary partition #3 (38 MB), so we can only have 2 other primary partitions.

Here's the result of running ''df'':
  Filesystem           1K-blocks      Used Available Use% Mounted on
  /dev/ida/c0d0p2         914108     77048    788318   9% /
  tmpfs                  1989212         0   1989212   0% /dev/shm
  /dev/ida/c0d0p1          91763     15029     71838  18% /boot
  /dev/ida/c0d0p6        4807072     32960   4529928   1% /home
  /dev/ida/c0d0p11       9879016     32860   9344320   1% /spare
  /dev/ida/c0d0p10        914108      8239    857127   1% /tmp
  /dev/ida/c0d0p8        3843160    211724   3436212   6% /usr
  /dev/ida/c0d0p9        2883376     32896   2704012   2% /usr/local
  /dev/ida/c0d0p7        9767184    178060   9589124   2% /var

====Disk Array====

On budlight, we have recently (May 2009) added a Clariion DAE (http://support.dell.com/support/edocs/stor-sys/dae/fcdae/fchardwr.pdf). The enclosure has an EMC label on it, but the documentation we have is for the same device rebranded and resold through Dell. We have it connected to budlight via a QLogic ql2100 HBA. The Clariion has 10 bays, each with a 36G FCAL drive. 

To make budlight see the drives in the DAE, we needed to install the qlogic-firmware package.

<code root>
# apt-get install firmware-qlogic
</code>

After a reboot, budlight saw /dev/sd[a-j]

We have assembled the drives into a RAID-5 array with one hot spare. This should give us roughly 300G, which is a huge increase in available storage.

The assembly command is:

<code root>
# mdadm --create /dev/md0 --level=5 --raid-devices=9 --spare-devices=1 /dev/sd[a-j]1
</code>

/proc/mdstat will show the progress of building the array. This is what it looks like when it is fully built:

<code root>
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdi1[8] sdj1[9](S) sdh1[7] sdg1[6] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]
      279302144 blocks level 5, 64k chunk, algorithm 2 [9/9] [UUUUUUUUU]

unused devices: <none>

</code>

After /proc/mdstat shows that md0 is fully assembled, we'll update mdadm.conf:

<code root>
# mdadm --detail --scan >> /etc/mdadm/mdadm.conf
</code>

In order to gain flexibility with the EMC DAE, we will need to install some additional packages:

<code root>
# apt-get install lvm2 dmsetup
</code>

This will allow us to create LVMs which can be resized as needed out of the logical RAID-5 device on the Clariion.

Label /dev/md0 as a physical volume:

<code root>
# pvcreate /dev/md0
</code>

Create the volume group:

<code root>
# vgcreate sluug /dev/md0
</code>

Create a 40G logical volume in the sluug VG called 'rsnapshot':

<code root>
# lvcreate -L 100G -n rsnapshot sluug
</code>

Format the rsnapshot volume and mount it:

<code root>
# mke2fs -j /dev/mapper/sluug-rsnapshot
# mkdir /rsnapshot
# mount /dev/mapper/sluug-rsnapshot /rsnapshot
</code>

Revel in the newly available space:

<code root>
df -h /dev/mapper/sluug-rsnapshot
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/sluug-rsnapshot
                       40G  177M   38G   1% /rsnapshot
</code>

And add the following to /etc/fstab:

<code root>
/dev/mapper/sluug-rsnapshot     /rsnapshot      ext3    defaults 0 0
</code>

===== Package Selection =====

We started with a minimal ("netinst") installation, with only a few packages installed. We will install all the required packages manually. This provides some added security, as we've minimized our attack surface to only the applications we actually need.

This system is intended to be a server, and should never run any X programs. Any GUI-type administration should be done over HTTPS. So we did not install any X server or X client programs.

===== Networking =====

Make sure that the network settings are all correct. Review the settings in ''/etc/network/interfaces''. It should look something like this (for Bud at Primary.Net):
<file>
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug eth0
iface eth0 inet static
        address 206.196.99.162
        netmask 255.255.255.240
        gateway 206.196.99.161
        network 206.196.99.160
        broadcast 206.196.99.175
</file>

Note that you'll need to be at the system console if you make changes to the network settings, because any network connections may be dropped. (You might be able to make the change and re-connect after getting dropped, if you don't take the interface down before making the changes, but it's not as clean, and might not work.)

If you make changes, restart the network:
<code rootshell>
/etc/init.d/networking restart
</code>

Ensure that you can connect to some Internet hosts to make sure that your configuration is correct. If you run into problems, try these troubleshooting steps:
  - See if you can ping another system on the same subnet.
    * NOTE: The firewalls on Bud and BudLight don't allow responding to ping, so don't try pinging them.
  - See if you can ping the default gateway.
  - See if you can ping the DNS server.
  - Check the routing table (''route -n'').
  - See if you can look up a name (using ''dig'' or ''nslookup'') and get an IP address back.
  - See if you can connect to an Internet service (using ''ssh'', ''wget'', or some other program).

===== Hostname =====

Ensure that the hostname is correct:
<code bash>
hostname
</code>

If it is not, change it:
<code rootshell>
hostname bud
</code>

Edit ''/etc/hosts'' to add the IP address of the system. We also put the other system in there. Note that the fully-qualified name must come before the short name, so that the system can determine the domain name properly. Also, do not put the hostname on the localhost line. The file should look something like this:
<file>
127.0.0.1       localhost
206.196.99.162  bud.sluug.org           bud
206.196.99.163  budlight.sluug.org      budlight
</file>

Verify that the system can get the domain name and fully qualified hostname:
<code rootshell>
hostname -d
hostname -f
</code>


===== Security Updates =====

Ensure that ''/etc/apt/sources.list'' contains a pointer to servers to fetch security updates. Also remove the lines that reference the installation CD-ROM. You may need to add the ''contrib'' and ''non-free'' items on each line. The ''/etc/apt/sources.list'' file should look like this:
<file>
deb http://ftp.us.debian.org/debian etch main contrib non-free
deb-src http://ftp.us.debian.org/debian etch main contrib non-free
deb http://security.debian.org/ etch/updates main contrib non-free
deb-src http://security.debian.org/ etch/updates main contrib non-free
</file>

Update the package list:
<code rootshell>
apt-get update
</code>

Upgrade any new packages:
<code rootshell>
apt-get upgrade
</code>

The first time through, the kernel will likely be updated. If so, it will tell you that you need to reboot.
<code rootshell>
reboot
</code>


==== Schedule Regular Updates ====

It would be nice to have the updates install automatically, but in order to prevent problems, it's best to have a system administrator apply the updates manually, so he can fix any problems that crop up. So instead, we'll alert the system administrators when there are updates available. 

We've adapted code from [[http://wiki.splitbrain.org/debiansnippets#send_mail_on_new_packages | here]] to check for new Debian updates. Save the following code to ''/etc/cron.daily/check-debian-updates'':
<file>
#!/bin/sh

HOSTNAME=`hostname`
MAILTO="sysadmin@sluug.org"
MAILFROM="Debian update checker <sysadmin@sluug.org>"

apt-get update >/dev/null 2>&1

NEWPACKAGES=`apt-get --print-uris -qq -y upgrade 2>/dev/null |awk '{print $2}'`

if [ ! -z "$NEWPACKAGES" ]
then
 mail -a "From: $MAILFROM" -s "New Packages for $HOSTNAME" $MAILTO <<EOF
There are new Packages available for $HOSTNAME:

$NEWPACKAGES

please run:
 apt-get upgrade
as root on $HOSTNAME.

If a package is listed as "held back", then also run:
 apt-get dist-upgrade
EOF
fi

exit 0;
</file>

Change the permissions on the script to make it executable:
<code rootshell>
chmod 755 /etc/cron.daily/check-debian-updates
</code>

Adding this script to the ''/etc/cron.daily'' directory will cause it to be run every day. By default, the daily cron scripts run at 6:25 AM. One nice thing about running them daily and sending them to a mailing list is that it's easy to see if the updates have or have not been applied by the next day. The more times the message is sent, the more likely someone will be to log in and run the updates.

**NOTE**: We should probably replace this custom script with ''cron-apt''.

==== Send Out Alerts for Low Disk Space ====

This script works much like the previous script, sending an email only if any partition is over 90% full. Save the following code to ''/etc/cron.daily/check-disk-space'':
<file>
#!/bin/sh

HOSTNAME=`hostname`
MAILTO="sysadmin@sluug.org"
MAILFROM="Drive space checker <sysadmin@sluug.org>"

DF_OUTPUT=`df -h | grep '^/' | sort -r -n -k5 | awk '$5 > "90%" {print "  " $6 " is " $5 " full"}'`

if [ ! -z "$DF_OUTPUT" ]
then
 mail -a "From: $MAILFROM" -s "Drive space report for $HOSTNAME" $MAILTO <<EOF
Drive space on $HOSTNAME is critical:

$DF_OUTPUT

Please clear up some space on the listed partitions.

EOF
fi

exit 0;
</file>

Change the permissions on the script to make it executable:
<code rootshell>
chmod 755 /etc/cron.daily/check-disk-space
</code>


==== Send Out Root Password Change Reminders ====

Root passwords should be changed at least every 6 months.
We decided to send out an email reminder to help ensure that we do that.

Save the following code to ''/etc/cron.monthly/root-password-reminder'':
<file>
#!/bin/sh

HOSTNAME=`hostname`
MAILTO="sysadmin@sluug.org"
MAILFROM="Root password reminder <sysadmin@sluug.org>"
MONTH=`date +'%1m'`

# This checks to see if it is July or January. If so, send out the reminder.
# Since this script is in cron.monthly, it only runs on the 1st of the month.
if [ $MONTH = '07' -o $MONTH = '01' ]; then
  mail -a "From: $MAILFROM" -s "Change root password on $HOSTNAME" $MAILTO <<EOF
Please change the root password on $HOSTNAME.

Whoever changes the root password, please reply to this email to
let everyone know that you've changed it. Provide your phone number
so that the other admins can call you to get the new password.

This script is located in /etc/cron.monthly/root-password-reminder,
and send emails out on July 1 and January 1.
EOF
fi

exit 0;
</file>

Change the permissions on the script to make it executable:
<code rootshell>
chmod 755 /etc/cron.monthly/root-password-reminder
</code>

Adding this script to the ''/etc/cron.monthly'' directory will cause it to be run on the 1st day of every month. The script itself checks to see if it's January or July, and only sends an email for those months. By default, the daily cron scripts run at 6:52 AM.

===== TODO =====

The cron job scripts should probably be moved to a different wiki page. Perhaps the security page, or create a new page to document system maintenance tasks.

Check that the update emails are getting sent. Outbound email may not be working (yet), or the list may not accept emails from the source address we used.

Consider replacing ''allow-hotplug eth0'' with ''auto eth0'', per HowtoForge document (linked below). Otherwise restarting the network doesn't work, and we'd have to reboot the whole system.

Consider adding user disk quotas to /home. See the HowtoForge document.

Create a few (temporary?) users for building the system.

Determine how we want to maintain users. If we decide to allow all our users to have accounts on the system, we probably want to keep them in sync with the existing AIX systems. Run periodic reports to ensure that users on systems do not get out of sync.

See the general [[todo | TODO]] page for more work to be done.


===== Credits =====

James Pattie led the installation effort, 2005-02-19.

Lots of other folks helped in the decision-making and installation process.

John House and Craig Buchek were the primary documentors.

Craig Buchek and Jeff Muse led the installation effort on the production servers, 2005-07-30.

Lee Lammert led the effort to move the systems to Primary Networks, 2006-01-16.

Craig Buchek rebuilt Bud with Debian 4.0 on 2007-05-30. Installation took about 2 hours, 1 hour of which was correcting a networking issue due to incorrectly documented network settings.

Craig Buchek rebuilt Bud with Debian 4.0r3 on 2008-03-02. Installation of the OS took about 30 minutes, with good documentation.

See [[http://www.howtoforge.com/perfect_setup_debian_etch | this HowtoForge document]] for an excellent step-by-step guide to installing Debian 4.0, with screenshots.