================================================================================
Steps to implement rsync backups on new client/server pair:
--------------------------------------------------------------------------------
Do all this as root on each indicated system except as noted.

Definitions and substitutions to make when doing this procedure:
  - Server system  = The system receiving backups, the ssh server.
  - Client system _= The system initiating backups, the ssh client.
  - __yourself__   = Your personal, not root, account.
  - __server__     = The short name of the server system.
  - __client__     = The short name of the client system.
  - __client_ip__  = The IP address of the client system.

Implementation systems:
  - The server system is currently bud or budlight, which is why "bud*" is used
    on the "Host" statement in the client's ssh_config file.  Aside from that,
    all references to the server are "__server__"
      + At the time of this writing, the only available server is bud.
  - The client systems are currently michelob and dark.
      + Due to problems with the ancient ssh on michelob, there are special
        steps for michelob only.  They haven't been fully tested at the time of
        this writing.

On the client, create a directory with custom config file and generated heys.
  - This only needs to be done once per client, regardless of the number
    of servers.
  - Create a directory for the custom config file and generated heys.
    mkdir     "$HOME"/xfer/
    chmod go= "$HOME"/xfer/
    ls -ld    "$HOME"/xfer/
  - Edit "$HOME"/xfer/config
           #*******************************************************************************
           # Special ssh client configuration for automated backups to a remote server    *
           #*******************************************************************************

           Host bud*.sluug.org.
           User root
           Compression no
           Protocol 2
           RSAAuthentication yes
           StrictHostKeyChecking yes
           ForwardAgent no
           ForwardX11 no
           IdentityFile ~/xfer/id_dsa
      + Can't use $HOME on the IdentityFile statement, but can use tilde.
  - Generate key pair on the client:
      + Even though not documented, the "-d" option works on all OpenSSH systems
        tested and works the same as "-t dsa".
      + On each client:
        time ssh-keygen \
          -t dsa \
          -N "" \
          -f "$HOME"/xfer/id_dsa \
          -C "Data transfer for client `hostname`"
  - Copy the generated public key for the client to a temporary location and
    edit:
      + cd  "$HOME"/xfer
      + cp  id_dsa.pub  /tmp/"`hostname`".id_dsa.pub
      + vi  /tmp/"`hostname`".id_dsa.pub
      + Add to the front of the line (unfolded, no spaces):
            from="__client_ip__",command="/usr/local/bin/authprogs",
            no-port-forwarding,no-X11-forwarding,no-pty,no-agent-forwarding
          = Include a space after the added stuff.
  - Copy the edited public key from the client to each server it will use:
      + Do this command on the client
        scp -p /tmp/"`hostname`".id_dsa.pub __yourself__@__server__:/tmp
      + When done distributing it to all servers:
        rm     /tmp/"`hostname`".id_dsa.pub

On the client, put server's host key in default known_keys for FQDN:
  - This needs to be repeated on each client once per server that will be used.
  - Execute ssh command to get the host key:
      + For all systems including michelob:
            ssh __server__.sluug.org.
      + When it asks to verify the key fingerprint, compare against
            http://www.sluug.org/members/accounts/ssh_keys.shtml
      + Don't actually need to login, probably can't since root access denied.

Install validation script and configuration file on each server:
  - Could install it in a system-wide location or "$HOME"/.ssh, or elsewhere.
    Might as well just use /usr/local/bin/
  - Copy from where it was modified to /usr/local/bin on the server.
  - Changes for use:
        chown root:root /usr/local/bin/authprogs
        chmod 755       /usr/local/bin/authprogs

Configure ssh for root account on each server:
  - mkdir -p "$HOME"/.ssh # Probably already exists
  - ls -ld   "$HOME"/.ssh # Check its not readable or writable by group or other
      + chmod go= "$HOME"/.ssh
  - Merge the edited public key from the client into (create if needed)
    "$HOME"/.ssh/authorized_keys
      + cd "$HOME"/.ssh
      + touch     authorized_keys
      + chmod 600 authorized_keys
      + cat /tmp/__client__.id_dsa.pub >> authorized_keys
      + rm  /tmp/__client__.id_dsa.pub
  - Create configuration program for authprogs:
      + cd "$HOME"/.ssh
      + touch     authprogs.conf
      + chmod 600 authprogs.conf
      + Edit      authprogs.conf to contain:
        #*******************************************************************************
        #
        # Configuration file for authprogs for remote access to this account.
        #
        # See authprogs source for documentation
        #
        #*******************************************************************************
        # For client: __client__
        [ __client_ip__ ]
            # Allow date and id commands for testing only
                re = date +(\+(%\w)+)*
                EX=id
            # Allow backups and restores via rsync to one specific directory
                RE=rsync --server -\w+ (--\S+(=\S+)?\s+)*\. /var/backups/__client__/\S+
  - Change "PermitRootLogin" from "no" to "forced-commands-only" in the server's
    /etc/ssh/sshd_config if it is tighter already.
      + Will need to signal the sshd server to reread the configuration:
            kill -HUP `cat /var/run/sshd.pid`

Create backup directory on the server for each client:
  - Create base directory only if needed:
      + mkdir     /var/backups
      + chmod 755 /var/backups
  - Create backup directory for this client:
      + mkdir     /var/backups/__client__
  - We want to change the permissions, and possibly ownership of the
    client level backup directory to allow read access to lower level
    directories that hold the backups of individual client directories.
    So make this 755 to allow access to the lower directories being backed up.
      + chmod 755 /var/backups/__client__
  - Depending on how the rsync is configured, the permission and ownership
    of the directory being backed up will be copied from the client.
    For example, doing all of /home by doing a cd to /home and running rsync
    with . for the SRC on the rsync command line copied the ownership and
    permission of /home.  Specifying a lower directory of /home for the SRC
    on the command line set the permission and ownership of that directory,
    but not /home.  Putting a / at the end of the SRC might also make a
    difference.

Test from the client:
  - ssh -F $HOME/xfer/config  __server__.sluug.org. id
  - ssh -F $HOME/xfer/config  __server__.sluug.org. date +%Y%m%d

Special for michelob:
  - The OpenSSH on michelob doesn't support -F option.
      + Can set $HOME to the base where the customized .ssh will be,
        except that didn't seem to actually work.
      + Created a simple Perl script to read the config file and convert
        into -o optons on the ssh command line.
  - Must edit the $HOME/xfer/config to change "IdentityFile" to "IdentityFile2"
    due to the different syntax for this older versin of OpenSSH.
  - ssh -o"IdentityFile2 /root/xfer/id_dsa"  __server__.sluug.org. id
  - ssh -o"IdentityFile2 /root/xfer/id_dsa"  __server__.sluug.org. date +%Y%m%d

Create script on the client to run:
  - Install the scripts:
        cd __temporary_location__
        cp -p             disksync.sh disksync-subs.sh /usr/local/bin
        cd /usr/local/bin
        chown root:system disksync.sh disksync-subs.sh
        chmod 755         disksync.sh disksync-subs.sh
  - Customize disksync.sh for the directories to back up.
  - For michelob only, install run_ssh.pl to compensate for the limitations
    of its ancient OpenSSH version.
        cd __temporary_location__
        cp -p             run_ssh.pl /usr/local/bin
        cd /usr/local/bin
        chown root:system run_ssh.pl
        chmod 755         run_ssh.pl

Schedule using cron on the client:
  - Add an entry to run /usr/local/bin/disksync.sh at desired time.
      + Don't redirect output anyplace, so it will be mailed to root.

Details on the scripts:
  - Baseed on existing cross-system backup scripts.
  - The original basis for the script and testing was setting some variables and
    running rsync:
        opt_std="--archive --hard-links --numeric-ids --one-file-system --stats"
        opt_std="${opt_std} --bwlimit=100"
        opt_del="--delete"
        opt_rsh="--rsh=ssh -F $HOME/xfer/config"
        options="$opt_std $opt_del"
        remote_host="__server__.sluug.org."
        in_base=/home
        out_base=/var/backups/__client__/home
        in_dir=steer-com # No slash to include directory name
        cd $in_base ; time rsync $options "$opt_rsh" \
            $in_dir  $remote_host:$out_base \
            2>&1 | tee -a /tmp/rsync_log.txt
  - Needed a check for "michelob" to set alternate ssh syntax.
  - Ended up with three scripts on clients and one on server:
      + disksync.sh just lists directories to sync and calls disksync-subs.sh.
      + disksync-subs.sh defines more configuration and runs rsync.
      + run_ssh.pl is only on michelob for problems with ssh.
      + For ssh authentication on server, using modified authprogs
        originally from http://www.hackinglinuxexposed.com/tools/authprogs