This document will be used to coordinate the effort associated with replacing Mailman 2.

https://www.sluug.org/pipermail/sysadmin/2023-August/010383.html

Date: Thu, 17 Aug 2023 11:31:08 -0500
From: Ken Johnson
Subject:  Mailman 3

mailman and mailman3 can co-exist under Debian 11. The cli commands are completely different, for example. I did not convert until on Debian 11. There can be python issues, of course – the longer you wait to dump python2, the more that can be messy, though I did not have very many issues personally. In some ways it may be easier to convert after the update, because the version of mailman3 you are moving to is closer to current, so it is easier to find help and information online. Also, for SLUUG the mailing list is pretty exposed to the internet, so best to avoid delay in that conversion, since no updates for mailman will be available.

As you look into this, you will probably see that Xapian is the recommended search option, but it is not the default. Once I got it working, Xapian was fine for me. If I recall correctly,it took me a while to figure out the configuration, because I had a hard time finding examples.

–

First of all, my message of 17 August 2023 contains a critical error. _Xapian_ is the recommended search option, but _Whoosh_ is the default. Replace 'Whoosh' with 'Xapian' in my message of 17 August.

===

This message on the Debian Users mailing list was very helpful to me when I was struggling to configure mailman3.

https://lists.debian.org/debian-user/2022/07/msg00528.html

===

This message thread on the mailman3 mailing list may be helpful:

https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/ZSSVDKKVKZYNQFKLTLWC4V6IIWL7Y3RP/

=== This fragment from /etc/mailman3/mailman-web.py may be helpful as an example:

## 12 Jul 2023 - KLJ - Configure Xapian instead of default Whoosh # # Full-text search engine # HAYSTACK_CONNECTIONS = {

  'default': {
      # You can also use the Xapian engine, it's faster and more accurate,
      # but requires another library.
      # http://django-haystack.readthedocs.io/en/v2.4.1/installing_search_engines.html#xapian
      # Example configuration for Xapian:
      'ENGINE': 'xapian_backend.XapianEngine',
      'PATH': '/var/lib/mailman3/web/fulltext_index',
  },

}

–

Password management: Mailman3 and Postgresql (don't use sqlite for mailman3!) will require the installer to specify some passwords during the setup process. Be prepared, and have a plan in advance for where and how to store and communicate those passwords.

Will use Debian packages.

No need to create mailman database and user before installing packages.

Package mailman3 depends on any of 4 dbconfig-* packages. Currently installed is "dbconfig-common", which is not acceptable. Testing shows installation of "mailman3" package without specifying any "dbconfig-*" will also install "dbconfig-sqlite3" by default. Specifying "dbconfig-mysql" will install that and not "dbconfig-sqlite3".

What packages to install?

• mailman3 mailman3-web
• python3-mailman-hyperkitty
• python3-django-postorius
• python3-django-mailman3
• listadmin
• dbconfig-mysql
• Skipping xapian (not Whoosh, but mailman3-web depends on python3-whoosh)
• And -doc for any package that has it, except prereq libraries: mailman3-doc xapian-doc

Since the Debian package of "mailman3-web" depends on "python3-whoosh", will not install "xapian" initially, as it just complicates the procedure. If desired, can install and use "xapian" later. After installation and testing, search works without any extra configuration.

export DEBIAN_PRIORITY=low
export DEBIAN_FRONTEND=readline
export PROD_LIST="mailman3 mailman3-web python3-mailman-hyperkitty \
        python3-django-postorius python3-django-mailman3 listadmin \
        mailman3-doc dbconfig-mysql  python3-pymysql python3-mysqldb"
script -a /var/tmp/mailman3-install-log.txt
TERM=dumb apt-get install $PROD_LIST
Answer the installation dialog questions, see below.
# End the script before continuing

The Hyperkitty key in "/etc/mailman3/mailman-web.py" ("MAILMAN_ARCHIVER_KEY") and "/etc/mailman3/mailman-hyperkitty.cfg" ("api_key") need to match. In testing, "MAILMAN_ARCHIVER_KEY" was set to "SecretArchiverAPIKey".

export DEBIAN_PRIORITY=low
export DEBIAN_FRONTEND=readline
dpkg-reconfigure python3-mailman-hyperkitty

This didn't ask any questions, exiting with message: "Replacing config file /etc/mailman3/mailman-hyperkitty.cfg with new version". Now both files have the same key.

As documented in "/usr/share/doc/mailman3/README.Debian", edit /etc/postfix/main.cf to "add the following settings". Also verify there were none of those settings already in it.

Verify the hash files are in "/var/lib/mailman3/data/" However, not readable by user/group, except "list". As of installation, both are empty, except for comments.

Restart postfix with "postfix reload".

• The installation configuration scripts created a sym link from "/etc/apache2/conf-available/mailman3.conf" to "/etc/mailman3/apache.conf".
• And a sym link from "/etc/apache2/conf-enabled/mailman3.conf" to "../conf-available/mailman3.conf".
• The installation also created sym links in "/etc/apache2/mods-enabled/" to existing files in "../mods-available/" for 3 "proxy*".
• So this is already done.

• Change "TIME_ZONE" from "UTC" to "America/Chicago"
• In the "INSTALLED_APPS" values, comment 'django_mailman3.lib.auth.fedora'. Apparently you can't comment 'allauth.socialaccount' or it will fail. Commenting the fedora entry will remove the "Please sign in with one of your existing third party accounts. Or …".
• The builtin Gravatar support can be disabled by adding "HYPERKITTY_ENABLE_GRAVATAR = False". However, this didn't seem to work in testing. Neither did "GRAVATAR_SECURE_URL = ''".
• Also review "/usr/share/mailman3-web/settings.py" for other settings.

• Change "site_owner" from "changeme@example.com" to the same used above for "Postorius superuser".

• mailman-hyperkitty.cfg: "base_url" - no change, since "localhost".
• mailman-web.py: "POSTORIUS_TEMPLATE_BASE_URL" - no change, since "localhost".
• mailman.cfg" "use_https" - leave at "no", this is for internal use?

Update "/etc/logrotate.conf" to add before the "include" line:

tabooext + .bak .off .out .save
taboopat + *[-.][0-9][0-9][0-9][0-9][0-9][0-9]
taboopat + *[-.][0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]

Update "/etc/logrotate.d" files "mailman3" and "mailman3-web" to use monthly rotation, longer retention, file name format, permissions, etc, Also using a wild card for multiple log files.

Update "/etc/mailman3/mailman.cfg" to add some separate log files, to match some in Mailman 2. After the "[logging.debian]" section, add new subsections:

[logging.bounce]
path: bounce.log
propagate: yes
[logging.vette]
path: vette.log
[logging.mischief]
path: mischief.log
[logging.subscribe]
path: subscribe.log
[logging.error]
path: error.log
propagate: yes
[logging.archiver]
path: archiver.log

Change existing logs to make readable.

Restart both "mailman3" and "mailman3-web".

date;systemctl restart mailman3;date
date;systemctl restart mailman3-web;date

Create and edit "/var/lib/mailman3/templates/hyperkitty/top.html" Copy the SLUUG logo section of "/srv/www/www.sluug.org/public/header.html" into it. Make sure it is owned by user:group "list:list".

Create and edit "/var/lib/mailman3/templates/postorius/"

cp -ip /usr/lib/python3/dist-packages/postorius/templates/postorius/base.html /var/lib/mailman3/templates/postorius/
vi /var/lib/mailman3/templates/postorius/base.html

Copy the SLUUG logo section of "/srv/www/www.sluug.org/public/header.html" into it. Make sure it is world readable.

Restart "mailman3-web".

"mailman-wrapper create --language no --owner postmaster@___ -no-notify ____@sluug.org

Set the list description via "Django site admin".

Verify Postfix hashes are accessible and automatically populated.

Test subscriptions.

Test posting.

Test archives.

Do archives hide the address of the poster? No.

The acceptable operation is that ASCII postings are delivered and archived as plain text ASCII. Not converted to "quoted-printable" or "base64". As discovered with Mailman 2, when the default footers are added, the charset is "utf-8", and the Content-Transfer-Encoding: is "base64". There might also be a concern with added content breaking DKIM signatures.

mkdir -p /var/lib/mailman3/templates/site/en/
touch    /var/lib/mailman3/templates/site/en/list:member:generic:footer.txt
chown -R list:list /var/lib/mailman3/templates/site/

This worked, and the postings are now delivered without being converted.

Or do it via the web interface at URL …/mailman3/postorius/lists/testmm3.bock2.sluug.org/templates That saves it in the database instead of as a file. Always use one or the other, don't mix them. This is only for templates of things added to mail, not changing web pages. Since this change was done for Mailman 2 via the "config_list" command to update each list in their database, the same should be done for Mailman 3. It should be done this way for existing lists when they are imported.

What about "list:member:digest:footer" for a digest message? In the directory of template files, there is no separate footer for regular and digest, only "generic:footer". In the web pages for creating templates, there is no "generic:footer", but there are "regular:footer" and "digest:footer".

Many places are still referring to "example.com".

• "Subject: [example.com]"
• "… account on example.com."
• "Thank you for using example.com!"

This is in database "mm3webdb" table "django_site". Change by logging in to the web site as admin and going to URL "…/mailman3/admin/sites/site/1/change/". Change the "Domain name" value to "sluug.org" and the "Display name" value to "SLUUG mailing lists".

In /var/log/syslog, every 15 minutes:

mariadbd[____]: ... [Warning] Aborted connection ____ to db: 'mm3db' user: 'mm3dbuser' host: 'localhost' (Got an error reading communication packets)

Does creating a list stop the errors about "gatenews"? No. Still seen in mariadb logs. Commented the "gatenews" entry in "/etc/cron.d/mailman3". This stopped the warning every 15 minutes, but there are still two each day, at the time of the other cron entries. Searching online found this is a common problem and is just an annoyance with nothing wrong. It might be a problem with "SQLAlchemy" not issuing "quit" to properly close the connection.

Will "mailman_queue_check.sh" work? Mailman 3 might be compatible with minor changes. Mailman 2 uses "/var/lib/mailman/qfiles/". Mailman 3 uses "/var/lib/mailman3/queue/".

Everything below has not been tested yet and is for planning

Check archive accuracy.

Configure to delete attachments from the archive during import? Initial testing shows about half the attachments from the MM2 archives are imported to MM3, others aren't. The difference hasn't been determined.

Test list deletion:

• Delete the list - ????
• Delete the archive - ????

Protect archives from address scraping and secure private lists. The protection method used for Mailman 2 doesn't apply to Mailman 3. Instead, make the list archive private, forcing subscribers (Or owners and moderators) to log in with their personal credentials to see the archive. This means there is no configuration at this time, unless we decide to make all archives public, and use the old protection method.

Test new configuration options and enhancements to make now. Especially any global options that could impact all production lists.

MM3 restricts the list "info" field to under 255 characters. Change any lists that are longer, to avoid truncation during import. Most MM2 production lists have much longer text. They also contain information that will be obsolete, such as how to access the archives. The text could be changed before or after import. Even if changed after import, planning what the text should be can speed the process.

Messages posted with certain non-ASCII characters may fail to be imported to the MM3 archives. Most non-ASCII characters cause no problems.

TBD settings:

• "header_checks" are now global?
• "[dmarc]" and/or "[ARC]".
• "charset". Leaving as "us-ascii".

After a reboot or restarting multiple processes, Mailman will often try to connect to the SQL database server before that server has completed startup to the point it can accept connections and perform operations. This makes the Mailman service fail and terminate. This is a well known problem. There is no systemd configuration present to state Mailman depends on the SQL service. This has been corrected in later Debian Mailman releases.

Correction is by a systemd override to establish the dependency.

Detailed actions are to be documented here. —-

To check the archives for potential errors before attempting import:

• Use cleanarch or cleanarch3 to check for bad From and Date headers.
• Use check_hk_archive (replaced by check_hk_import) to check for messages that would fail import.

While cleanarch is a standard part of MM2, cleanarch3 is not in the currently installed Debian python3-django-hyperkitty package. It is in the more recent 1.3.12-1 (1.3.8 and later) release. The same for check_hk_archive (replaced by check_hk_import). Download the latest package, extact the two scripts to a temporary location.

For 3 lists, that existed before the 2007 move from Majordomo, the "/var/lib/mailman/archives/private/LISTNAME.mbox/LISTNAME.mbox" file doesn't go back as far as the original Majordomo files imported into mailman. The monthly text files created during the import to generate the HTML do have all the mail. In "var/lib/mailman/archives/private/LISTNAME/" are the "YYYY-MMMMM.txt" and "YYYY-MMMMM.txt.gz" files for each month since 2000. Should just need to concatenate them in the proper order. Since these are not being updated, this preparation work can be done anytime before the archive import.

months='January
February
March
April
May
June
July
August
September
October
November
December'
years='2000 2001 2002 2003 2004 2005 2006 2007'

archive_path=/var/lib/mailman/archives/private
export listname=____

for year in $years
  do
    for month in $months
      do
        file="${archive_path}/${listname}/$year-$month.txt"
        #echo "$file"
        if [ -f "$file" ]
          then
            cat "$file"
          fi
      done
  done > /tmp/${listname}-rebuilt.txt

Edit /tmp/${listname}-rebuilt.txt and delete everything starting with the
first message in the mbox file until the end.
Except the original monthly text file for the initial import from Majordomo
has sections that go back in time, and some are repeated.
This makes it more complicated to determine what to delete or keep.

Also have to change the "From " lines that were massaged to obscure the sender.
Before that, search for any that are incorrect "From " lines within a message.
These might have been archived that way.  In that case, leave them alone.
<code>
1G/^\CFrom .*@/ Any within a message to fix
:%g/^\CFrom .* at .*200\d$/s/ at /_at_/ " Make normal mail standard line

Run the two checking tools, the same as below. The "From " will probably change for a small number of messages.

/tmp/cleanarch3 \
    /tmp/${listname}-rebuilt.txt \
    > /tmp/${listname}-rebuilt.mbox
diff \
    /tmp/${listname}-rebuilt.txt \
    /tmp/${listname}-rebuilt.mbox

/tmp/check_hk_import /tmp/${listname}-rebuilt.mbox

Stop postfix, to prevent any incoming mail going to the mailman 2 queue while in the process of transitioning. Not needed for low volume lists where no postings can be expected during this period.

Since the import and indexing of a large archive can take three hours, use a technique to continue running if an interactive session is lost. Don't attempt during a storm.

Stop mailman 2 …

Must create an empty list before import. For each list:

export config_path=/var/lib/mailman/lists
export archive_path=/var/lib/mailman/archives/private
export listname=____
export listfull=${listname}@sluug.org

/tmp/cleanarch3 \
    ${archive_path}/${listname}.mbox/${listname}.mbox \
    > /tmp/${listname}-cleaned.mbox
diff \
    ${archive_path}/${listname}.mbox/${listname}.mbox \
    /tmp/${listname}-cleaned.mbox

/tmp/check_hk_import  /tmp/${listname}-cleaned.mbox

Check the number of archive messages is reasonable.
The MM2 list archives imported from Majordomo have many duplicate HTML files.
ls ${archive_path}/${listname}/[12]???-* | grep -c 0......html

mailman-wrapper create   ${listfull}
((The only output is that the list was created))

mailman-wrapper import21 ${listfull} ${config_path}/${listname}/config.pck
((If there are no problems, there might be no output))
((The only output might be errors and "Importing members" progress))

chmod a+r /tmp/${listname}-*.mbox # Since run as "www-data" user

((Include "rebuilt.mbox" only if it was created.))
date ; time mailman-web hyperkitty_import --no-sync-mailman \
    --verbosity 3 --no-color \
    --list-address ${listfull} \
    /tmp/${listname}-rebuilt.mbox \
    /tmp/${listname}-cleaned.mbox \
    >> /tmp/import-${listname}-log.txt 2>&1 ; date
((It lists each message as it is processed.))

date ; time mailman-web update_index_one_list --verbosity 2 --no-color \
    ${listfull} \
    >> /tmp/index-${listname}-log.txt 2>&1 ; date
((There is no output during the first phase, a quick summary of
the number indexed, then nothing during last phase))

Test by posting and web access?

Delete the list from the Mailman 2 configuration by moving the config out of the production directory.

mv -iv ${config_path}/${listname} ${config_path}-disabled/

Delete Mailman 2 list aliases and create Mailman 3 aliases? No. Mailman 2 uses "/var/lib/mailman/data/aliases" for "alias_maps". Mailman 3 uses "/var/lib/mailman3/data/postfix_lmtp" for "local_recipient_maps". No action is needed. The new list aliases are created when the lists are imported. The "local_recipient_maps" takes precedence over "alias_maps".

Start mailman 2 …

Start postfix if it was stopped.

There is a postfix configuration conflict that results in warning "do not list domain in BOTH virtual_mailbox_domains and relay_domains". This is a well known problem. See https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.html#unusual-postfix-configuration

Check all configuration web pages on the old and new systems, to make sure no options, settings, or values were lost.

Change the list "info" field for each list.

Check the list archive HyperKitty Description and Archive policy. Might need to set Owners and Moderators.

Test …

Delete or deflate HTML postings and delete attachments on all lists. The "Collapse alternatives" and "Convert html to plaintext" options.

Change the "Action to take on messages which have no content" setting from "Discard" to "Reject".

Create "/etc/cron.d/mailman3-local" that runs some check that all expected processes are running.

• 15 "^list .*/usr/lib/mailman3/bin/" for "mailman3",
• 2 "www-data .*uwsgi" for "mailman3.web",
• 6-7 "manage.py qcluster" for "mailman3.web",
• Perhaps also check the output of the "mailman-wrapper status" command for "running".

Is there a way to disable "Download" of HyperKitty archive as a ".mbox" file?

Stop and disable the Mailman 2 service.

Edit "/etc/postfix/main.cf" and remove "hash:/var/lib/mailman/data/aliases" from "alias_maps".

Save the monthly text files of the Mailman 2 archives.

Delete "/etc/cron.d/mailman-local" that runs "mailman_queue_check.sh". Delete "/usr/local/bin/mailman_queue_check.sh".

Delete Mailman 2 from the Apache site configuration.

Since the services are started by systemd, it is probably better to do start/stop/restart using the systemctl command. Using the direct commands might not update systemd on the service status.

Never use the "mailman" command, only "mailman-wrapper" that runs as user "list".

Useful "mailman-wrapper" commands:

• "mailman-wrapper –help" - display the available commands.
• "mailman-wrapper COMMAND –help" - display help for one command.
• "mailman-wrapper lists" - list the available lists.
• "mailman-wrapper members FULL_LIST_NAME_WITH_HOSTNAME | wc -l" - count of members to one list. In a future release, this might be handled by the "–count-only" option.
• "mailman-wrapper findmember 'REGEXP'" - Display all memberships for matching users.
• "mailman-wrapper restart" - Stop and restart the Mailman runner subprocesses. See above about using the systemctl command.

"mailman-web" - Django's command-line utility for administrative tasks. Runs the "mailman3-web" "manage.py" command as user "www-data".

"django-admin" - Similar to "mailman-web", but without the proper settings.

• https://WEBSERVER/mailman3 / - General list access and user login. What should be announced to humans.
• https://WEBSERVER/mailman3/postorius/lists/ - What just specifying "mailman3" redirects to. What should be used in links from the main web site.
• https://docs.mailman3.org/en/latest/userguide.html - List Member Manual

• https://WEBSERVER/mailman3/hyperkitty/ - long list of "Available lists", with "Sign In" for regular list members.
• https://WEBSERVER/mailman3/postorius/ - short list of "Mailing Lists".
• https://WEBSERVER/mailman3/postorius/lists/ - the same as just "postorius/".

• https://WEBSERVER/mailman3/admin/ - Administration login, only "for a staff account".
• https://WEBSERVER/mailman3/LISTNAME.HOSTNAME/ - Access private lists not published on the index of lists. You will still have to log in or directly subscribe to do anything more than see the list description. This uses a "." instead of "@".
• https://WEBSERVER/mailman3/hyperkitty/list/LISTNAME@HOSTNAME/ - Access private archives. You will still have to log in to see the private archive. This uses a "@".

You can use the general URL for lists, then login and you will see private "lists that you are owner, moderator or subscriber for".

To approve or reject a held message:

• Log into the admin URL
• Go to "View site".
• Select the list name from the table of lists.
• Go to "Held messages".
• Select the check box for the messages to be handled.
• Select the action to perform.

Trying to log into the general URL with the admin account might fail with a 500 error.

• Logging into the general URL with your personal account/password used for list subscription doesn't give access to list administration, only changing your account and relationships to lists.
• This is even if your account is the list owner.
• Use this with your personal credentials to login and control "lists that you are owner, moderator or subscriber for".

Use the admin URL, then log in, then either select

• For the mailing list, select "View site" in the heading,
  • "Mailman settings" to change subscription settings for this address.
  • A list to change "List Settings", handle pending actions, etc.
• For list archives, select "Mailing lists" under the "HyperKitty", then a list to change description, etc.
  • There is no command line interface for most HyperKitty actions.
  • To remove the archive for a list, you must use the admin web interface and the "HyperKitty > Mailing lists" (admin/hyperkitty/mailinglist/) web page. This only affects the list archive, not the base list.

When logging out of the web interface, watch for the logout confirmation web page.

There are separate services and processes for the mailing list and web.

• The "mailman3-web" service controls the "uwsgi" processes. and includes "python3 manage.py" processes.
• The "mailman3" service is Mailman 3, for processing mail. For the "/usr/lib/mailman3/bin/" processes.
• The "mailman" service is Mailman 2, not 3.

If you update a template file, you may need to restart the matching service in order for the changed file to be used.

The logs are in "/var/log/mailman3/", with web actions in the "web/" subdirectory.

The structure of log files was changed to be closer to the separate log files of Mailman 2.