Table of Contents

MailMan

We originally considered using Majordomo, but then decided to go with Mailman. There are several advantages of MailMan over Majordomo:

  1. Newer
  2. Better docs
  3. Works out of the box with postfix
  4. Web management interface
  5. Tool for member migration from majordomo
  6. htdig patch

The initial installed version was mailman 2.1.9 installed on 11/18/06 by Jeff Muse.

Installation

http://www.gnu.org/software/mailman/index.html

http://www.list.org/mailman-install/

The list.org site was the primary reference used.

We also used a patch from http://www.openinfo.co.uk/mm/patches/444884/index.html to enable searching list archives via htdig.

Preface

We got mailman up and running on bud after the two of you left. Search functionality is working as well. Right now, mailman is only handling mail lists for bud.sluug.org. We'll need to update it to handle lists for all of sluug.org before moving it into production.

Unfortunately, mailman setup required editing a number of config files, so there's some missing detail in my .bash_history. Also, there was a fair amount of backtracking and undoing, so I'm just going to summarize what I did for documentation purposes.

Jefff Muse to Carl Fitch

Obtain Source

Download mailman-2.1.9 from http://sourceforge.net/projects/mailman/, verifying with gnupg. Also download the patch to allow htdig functionality from http://www.openinfo.co.uk/mm/patches/444884/index.html. Put both in /usr/local/src.

Create User and Group

sudo groupadd mailman
sudo useradd -c "Gnu Mailman" -s /no/shell -d /no/home -g mailman mailman

Install Dependencies

sudo apt-get install htdig htdig-doc gawk lockfile-progs gettext
sudo apt-get install python-dev

NOTE: MailMan requires at least Python 2.4. Debian 4.0 meets this requirement. Debian 3.1 did not, so we had to install Python 2.4 and use update-alternatives to make it the default version. Debian 3.1 also required installing libdb2.

Pre-installation Steps

As root, create the Mailman directory, set its ownership and permissions: 

mkdir /usr/local/mailman-2.1.9
chown mailman /usr/local/mailman-2.1.9
chmod 2775 /usr/local/mailman-2.1.9
ln -s mailman-2.1.9 /usr/local/mailman

Extract and Prepare Source

The Debian Mailman package did not include search capability. To enable search Mailman has to be compiled with htdig. Extract the source file into /usr/local/src/mailman-2.1.9 and move the patch into that directory. gunzip the patch.

Disable most language support. This is an ugly hack to get around a problem with Japanese-language support in python on Debian Etch. Comment out the following line in messages/Makefile.in: 

#LANGUAGES=     ar ca cs da de en es et eu fi fr hr hu ia it ja ko lt nl \
#                               no pl pt pt_BR ro ru sl sr sv tr uk vi zh_CN zh_TW

and replace it with: 

LANGUAGES=     de es

In templates/Makefile.in, also comment out the original LANGUAGES line. 

#LANGUAGES=     ar ca cs da de en es et eu fi fr hr hu ia it ja ko lt nl \
 #                               no pl pt pt_BR ro ru sl sr sv tr uk vi zh_CN zh_TW

and replace it with: 

LANGUAGES=     de en es

In misc/paths.py.in, comment out the following lines: 

import japanese
import korean
import korean.aliases

Apply the Patch and Build

patch -p1 < htdig-2.1.9.0.1.patch
./configure --prefix=/usr/local/mailman-2.1.9 \
--with-mailhost=localhost --with-urlhost=bud.sluug.org
make
sudo make install
cd /usr/local/
sudo ln -s /usr/local/mailman-2.1.9/ mailman
cd /usr/local/mailman
sudo chgrp mailman .
sudo chmod a+rx,g+ws .
cd bin
sudo ./check_perms
sudo ./check_perms -f
cd archives
sudo chown www-data private
sudo chmod o-x private
cd ..
sudo chown root mailmain-2.1.9

Apache configuration for all lists

Then edit /etc/apache2/sites-available/000-www.sluug.org and add: 

        # Mailing list archives using Mailman
        ScriptAlias /mailman/ /usr/local/mailman/cgi-bin/
	Alias /pipermail/ /usr/local/mailman/archives/public/

Protect with Password

Do this for lists that have their archives password protected to prevent e-mail address harvesting. If a list's archives are not password protected, this section is not needed for that list. At this time, the discuss and steercom lists share the discuss list's password, while the sysadmin list has a separate password.

Tell Apache to use password protection

Edit /etc/apache2/sites-available/000-www.sluug.org and add a new directory section, filling in the directory where the archive is stored, the name of the list for the AuthName, and the file with the password. 

        # Define password protection for this list's archives
	<Directory /usr/local/mailman/archives/public/CUSTOM>
		Options FollowSymlinks
		AuthType Basic
		AuthName "SLUUG CUSTOM Archive Access"
		AuthUserFile /etc/apache2/CUSTOM-passwords
		Require valid-user
	</Directory>

Security holes

In 2018, on amber, it was found there were two major holes that allowed bots to access all mailing list contents:

  1. Each archived posting has two URLs, /pipermail/ and /mailman/htdig/, however only the first is protected.
  2. The sluug.org and stlwebdev.org archives are defined in separate config files, and only protect the lists defined in each, but all lists are accessible through either domain.

Therefore, the above <Directory> sections were replaced with <LocationMatch> sections that first denied all access to the two URLs, then allowed password protected access to to selected lists available through that domain. Also unlimited access to two lists used for announcements. Any list not overridden by a subsequent section will be blocked by the first global section. Using wildcards allowed protecting both URL paths without duplicating all the password statements. 

    # Heavy use of symbolic links in Mailman configuration
    <Directory /usr/local/mailman/archives/public>
        Options FollowSymlinks
    </Directory>

    # Block all access to other lists' archives
    # Alternate path to same files via htdig search results
    <LocationMatch "^(/mailman/htdig|/pipermail)">
        Order allow,deny
        Deny from all
    </LocationMatch>

    # Define password protection for this list's archives
    # For all these lists: discuss steercom jobs test*
    # Alternate path to same files via htdig search results
    <LocationMatch "^(/mailman/htdig/(discuss|steercom|jobs|test)|/pipermail/(discuss|steercom|jobs|test))">
        Allow from all
        AuthType Basic
        AuthName "SLUUG Discussion Archive Access"
        AuthUserFile /etc/apache2/discuss-passwords
        Require valid-user
    </LocationMatch>

    # Define password protection for this list's archives
    # For sysadmin list only
    # Alternate path to same files via htdig search results
    <LocationMatch "^(/mailman/htdig/sysadmin|/pipermail/sysadmin)">
        Allow from all
        AuthType Basic
        AuthName "SLUUG Sysadmin Archive Access"
        AuthUserFile /etc/apache2/sysadmin-passwords
        Require valid-user
    </LocationMatch>

    # No password protection for this list's archives
    # For all these lists: announce users
    # Alternate path to same files via htdig search results
    <LocationMatch "^(/mailman/htdig/(announce|users)|/pipermail/(announce|users))">
        Allow from all
    </LocationMatch>

All common parts of the port 80 and port 443 <VirtualHost> definitions were moved to a common file to eliminate complete duplication.

Make similar changes to the stlwebdev web site definition.

Create the password file

The name of the file matches the AuthUserFile configuration statement. The username for the htpasswd command is whatever is used for that password file. You will be prompted for the password by the htpasswd command. 

cd /etc/apache2/
sudo htpasswd -c CUSTOM-passwords CUSTOM
sudo chown root:www-data CUSTOM-passwords
sudo chmod 640 CUSTOM-passwords

Have Apache recognize the configuration file changes

sudo /etc/init.d/apache2 reload

Configure Mailman

Then edit /usr/local/mailman/Mailman/mm_cfg.py and add: 

IMAGE_LOGOS='/images/'
MTA='Postfix'
USE_HTDIG='true'
HTDIG_HTSEARCH_PATH = '/usr/lib/cgi-bin/htsearch'
HTDIG_RUNDIG_PATH = '/usr/bin/rundig'

Start Mailman on Boot

cd /usr/local/mailman
sudo cp scripts/mailman /etc/init.d/mailman
sudo /usr/sbin/update-rc.d mailman defaults
cd ../bin
sudo ./genaliases''

Postfix

main.cf

Edit /etc/postfix/main.cf: 

alias_maps = hash:/etc/aliases,hash:/usr/local/mailman/data/aliases
unknown_local_recipient_reject_code = 550''

maps

sudo postfix reload
cd /usr/local/mailman/data
sudo chown mailman:mailman aliases*
sudo chmod g+w aliases*
cd ../bin
sudo ./newlist mailman

– Jeff, What does this do? – 

sudo config_list -i data/sitelist.cfg mailman

Mailman uses a site-wide mailing list called "mailman". This list is where system-generated mails appear to come from. This command applies a generic template defined in sitelist.cfg to the mailman list.

cron

cd ../cron
sudo crontab -u mailman crontab.in

Edit crontab before or after installation to:

Copy Apache Images to Mailman

sudo ../bin/mailmanctl start
su
cd /usr/local/mailman/archives/private
cp *png *jpg /home/web/www.sluug.org/images/

Initiallize htdig

I'm not sure if these next steps were absolutely necessary 

rundig
htfuzzy -v synonyms
htfuzzy -v endings
htmerge

This is absolutely necessary: create a list via the web page (see below) and send some email to it. So is the next step. 

/usr/bin/python -S /usr/local/mailman/cron/nightly_htdig -v

Everything appears to be working now. The search engine updates once a day at 2:19 via mailman's cron.

Importation of SLUUG Archives

Expand list to all of SLUUG

First, edit /usr/local/mailman/Mailman/Defaults.py:

Change 

DEFAULT_EMAIL_HOST = 'bud. sluug.org' to
DEFAULT_EMAIL_HOST = 'sluug.org'

This will let us handle lists for all of sluug.org instead of just bud. The bud_test list remains functional.

Restart mailman: 

$sudo /etc/init.d/mailman restart

Even though the following sections might refer to only the discuss and steercom lists, the announce, sysadmin, test, testing, and users lists also were created. This probably needs to be reworked as a generic procedure for any new lists added in the future.

Create Lists

Create the discuss and steercom lists: 

$cd /usr/local/mailman/bin
$./newlist discuss
$./newlist steercom

Import lists

Import the lists: 

$gunzip /home/jmuseinst/discuss*gz
$gunzip /home/jmuseinst/steercom*gz
$cd /usr/local/mailman/bin
$for file in /home/jmuseinst/discuss*
> do sudo ./arch dicsuss $file
> done
$for file in /home/jmuseinst/steercom*
> do sudo ./arch steercom $file
> done

I also imported the subscriber lists from majordomo on michelob into mailman on bud. This, too, was trivial: 

scp /opt/majordomo/lists/discuss  and /opt/majordomo/lists/steercom over to bud.

Then on bud: 

$cd /usr/local/mailman/bin
$ sudo ./add_members -r /home/jmuseinst/discuss -w n discuss
$ sudo ./add_members -r /home/jmuseinst/steercom -w n steercom

Mailman can only base the archive on the timestamp in the mail, so we've got a few mails archived from 2021 and 2013. Now we know who had problems with keeping their clocks set accurately :-(

Generate Search index

Regenerate the search index: 

$ sudo /usr/bin/python -S /usr/local/mailman-2.1.9//cron/nightly_htdig -v

Also, comment out all the majordomo entries in /etc/aliases and run "newaliases". 

newaliases

There's a cron job that gzip's the monthly archives for all the lists.

On dark only, change the first alias in /etc/mail/aliases for the list from @mail.sluug.org to @sluug.org (because @mail.sluug.org currently goes to michelob and @sluug.org goes to bud). On michelob only, uncomment the alias that redirects to @sluug.org and comment out the block of Majordomo aliases for the list. Run newaliases on both systems.

Create List Descriptions

In the administrative interface for discuss (http://bud.sluug.org/mailman/admin/discuss), add the following under "terse description": 

SLUUG general discussion

Under "introductory description", add: 

This is where we discuss Unix and Unix-like operating systems. We also discuss applications that run on these operating systems, programming in these environments, and even a little bit of hardware.

IIn the administrative interface for steercom (http://bud.sluug.org/mailman/admin/steercom), add the following under "terse description": 

SLUUG Steering Committee discussion
This is the place to discuss SLUUG business matters, planning and related items.

In both administrative interfaces, where it says "Where are replies to list messages directed? Poster is strongly recommended for most mailing lists.", check "This list".

Archives

We changed the template for archive index entries, to show the date and time of each post. This was done by editing /usr/local/mailman/templates/en/archidxentry.html. We also had to modify /usr/local/mailman/Mailman/Archiver/HyperArch.py to send the date string to the template: 

--- /usr/local/mailman-2.1.9/templates/en/archidxentry.html.ORIGINAL    2007-06-13 14:20:00.000000000 -0500
+++ /usr/local/mailman-2.1.9/templates/en/archidxentry.html     2007-07-23 00:05:04.000000000 -0500
@@ -1,4 +1,4 @@
 <LI><A HREF="%(filename)s">%(subject)s
 </A><A NAME="%(sequence)i">&nbsp;</A>
-<I>%(author)s
+<I>%(author)s <span style="font-size:50%%;">(%(datestr)s)</span>
 </I>
--- /usr/local/mailman-2.1.9/Mailman/Archiver/HyperArch.py.ORIGINAL     2007-06-13 14:19:59.000000000 -0500
+++ /usr/local/mailman-2.1.9/Mailman/Archiver/HyperArch.py      2007-06-14 14:22:30.000000000 -0500
@@ -1213,7 +1213,8 @@
             'filename': urllib.quote(article.filename),
             'subject':  subject,
             'sequence': article.sequence,
-            'author':   author
+            'author':   author,
+            'datestr':  article.datestr
         }
         print quick_maketext(
             'archidxentry.html', d,

We ran into some problems using a % character in the templates (to specify a 50% font size). The % character is used to delimit variable substitution, so to make it work (instead of printing just the variable names) we had to double the % sign to have it print a literal % character.

Discuss archives are at http://bud.sluug.org/pipermail/discuss/. List info is at http://bud.sluug.org/mailman/listinfo/discuss.

Steercom list info is at http://bud.sluug.org/mailman/listinfo/steercom. Archives are at http://bud.sluug.org/pipermail/steercom/

I haven't moved sysadmin over, because it looks like it has a different password protection scheme than the other lists.

List Configuration

Each of our lists will be a little different. Configuration of most settings is done via the admin web interface.

ANNOUNCE

Under Privacy Options / Sender Filters, we changed the following, in order to keep the list admins from getting all the spams that are sent to the list by non-members:

DISCUSS

Not sure what we changed.

Create new mailing lists at http://bud.sluug.org/mailman/create

Get information about lists on bud at http://bud.sluug.org/mailman/listinfo

Get on the test list at http://bud.sluug.org/mailman/listinfo/bud_test

Look at the test list archives at http://bud.sluug.org/pipermail/bud_test/

Configure the bud_test list at http://bud.sluug.org/mailman/admin/bud_test/

Creating lists for virtual domains

* Make sure your domain's MX records point to bud.sluug.org

* Make sure your domain is included in virtual_alias_maps in /etc/postfix/main.cf

* In /etc/postfix/virtual, add a new entry for your list. The LHS should be the address to which users send their posts, and the RHS should be the name of the list you are creating (no domain part, just the list name).

* Run 

# postmap /etc/postfix/virtual

* In /usr/local/mailman/Mailman/mm_cfg.py, add your domain to POSTFIX_STYLE_VIRTUAL_DOMAINS. If there needs to be more than one domain set in POSTFIX_STYLE_VIRTUAL_DOMAINS, this variable probably needs to be set in python list syntax.

* Restart mailman 

# /etc/init.d/mailman restart

* Create a new vhost in /etc/apache2/sites-available for your virtual domain if it doesn't already exist. Using /etc/apache2/sites-available/www.sluug.org as an example, copy the mailman <directory> containers into your vhost, making sure to update passwords, descriptions, and archive paths as necessary.

* Restart apache 

# apache2ctl graceful

* Create your list 

#/usr/local/mailman/bin/newlist list@domain

You will be prompted for the email address of a list admin and for an administrative password for the list. Check that email account for a welcome mail. Go to the admin URL in that mail and select "Privacy Options"→"Recipient Filters". Add your list's email address to the box labeled "Alias names (regexps) which qualify as explicit to or cc destination names for this list." If you don't, all posts to your new list will be held for admin approval with a message about implicit destinations.

* Send some test mails to verify that the list is working as expected.

Currently we are hosting announce@stlwebdev.org and discuss@stlwebdev.org.

Mailman TODO:

Spam Subscriptions

In 2018, it was discovered that one or more criminals were using a bot network to make repeated subscription requests to multiple lists, with the intent of SLUUG sending thousands of subscription confirmation e-mails to one address, that would eventually change to the next target. Looking in the logs, this had been happening for years, with SLUUG sending over 80k spam confirmations. This also taxed the SLUUG mail server when hundreds of e-mails were rejected and sitting in the mail queue for retry.

In response, a local mod was made to subscribe.py 

sluug_sub_mod1_value = cgidata.getvalue('sluug_sub_mod1', '')
    if not sluug_sub_mod1_value:
        syslog('mischief', 'Subscribe w/o local mod form field as: %s: %s', email, remote)
        results.append(_('Subscription failed due to internal error!'))
    elif sluug_sub_mod1_value != 'sluug_sub_mod1-20180517':
        syslog('mischief', 'Subscribe w/ wrong local mod form field value %s as: %s: %s', sluug_sub_mod1_value, email, remote)
        results.append(_('Subscription failed due to internal error!'))

Also modified the generic listinfo.html template and the customized version for the announce list (no other customized versions needed changes) to add: 

<input type="hidden" name="sluug_sub_mod1" value="sluug_sub_mod1-20180517">

The permanent fix to stop all spam subscriptions is to upgrade to a current release of mailman that includes SUBSCRIBE_FORM_SECRET and probably other new features to combat the bots.

GENERAL MAIL TODO:

That should do it for now - enjoy the rest of the weekend.

Jeff Muse

Notes

Logs

Logs are in /usr/local/mailman/logs/. The log files are used as follows:

TODO: We should move these to /var/log and put them under log rotation. See contrib/*redhat_fhs.patch for a source modification to change log and data file locations.

Important information about problems might also be in the Apache server logs. Currently in /var/log/apache2/.

Problems seen

Following a system reload and restore of mailman directories from backups, searching failed with search failed -12-. This turned out to be a missing symbolic link from /usr/local/bin/htsearch to /usr/lib/cgi-bin/htsearch. It is not clear if this symbolic link is created automatically htdig, or if it was originally created manually, and left out of the procedure.

TODO

this, and two options are postfix maps and a mysql database. We didn't make any decisions.

Two particular challenges will be copying existing passwords for POP3/IMAP access and mail filtering (procmail/maildrop/whatever). Once we get users set up, we'll need to migrate their mail spools.

look at the performance impact of scanning list mail. This should probably be done incoming list mail only.

Horde becauses it has a powerful interface and a ton of cool modules. I haven't used the password module, but it might be particularly useful for us. See http://www.horde.org/accounts/screenshots/accounts.png. If we use horde, we'll be using mysql, so that might be the way to go for virtual users.

Majordomo

NOTE: This was the original attempt, but is no longer being used.

Majordomo was installed to use the same config & archives as current system

Installed Majordomo from source: 

Source distribution (gzip'd) of current version (1.94.5) of Majordomo
http://www.greatcircle.com/majordomo/1.94.5/majordomo-1.94.5.tar.gz
Source distribution (compressed) of current version (1.94.5) of Majordomo
http://www.greatcircle.com/majordomo/1.94.5/majordomo-1.94.5.tar.Z
Source distribution (uuencoded) of current version (1.94.5) of Majordomo
http://www.greatcircle.com/majordomo/1.94.5/majordomo-1.94.5.tar.Z.uu

http://www.greatcircle.com/majordomo/1.94.5/majordomo-1.94.5.tar.gz
#tar xzf majordomo-1.94.5.tar.gz
#cd majordomo-1.94.5

Create majordomo user and group using the following commands
#groupadd majordomo
#useradd majordomo -g majordomo
#cat /etc/passwd | grep majordomo

Create installation directory for majordomo
#mkdir /usr/local/majordomo

Edit the makefile
#vi Makefile

Change the line with PERL variable definition to
PERL = /usr/bin/perl
Change the line with W_HOME variable definition to
W_HOME = /usr/local/majordomo
Change the lines with W_USER and W_GROUP variables definition to

W_USER = 1008
W_GROUP = 1008
set TMPDIR = /tmp

Edit the configuration file
#cp sample.cf majordomo.cf
#vi majordomo.cf

Set $whereami variable to your host name and that is enough
$whereami = "your-host.com";

Install the majordomo
#make wrapper
#make install
#make install-wrapper

Check the installation
#cd /usr/local/majordomo; ./wrapper config-test

Installed MHonArc from package

Tarr'd & gzip'd /opt/majordomo directory on Michelob

scp'd to bud & extracted to /usr/local/majordomo

Instlled MHonArc on Bud (to use same tool as Michelob)

Credits

- Jeff Muse - Initial install of Mailman and providing notes from the install.

Comments