This document will be used to coordinate the effort associated with replacing Mailman 2.

https://www.sluug.org/pipermail/sysadmin/2023-August/010383.html

Date: Thu, 17 Aug 2023 11:31:08 -0500
From: Ken Johnson
Subject:  Mailman 3

mailman and mailman3 can co-exist under Debian 11. The cli commands are completely different, for example. I did not convert until on Debian 11. There can be python issues, of course – the longer you wait to dump python2, the more that can be messy, though I did not have very many issues personally. In some ways it may be easier to convert after the update, because the version of mailman3 you are moving to is closer to current, so it is easier to find help and information online. Also, for SLUUG the mailing list is pretty exposed to the internet, so best to avoid delay in that conversion, since no updates for mailman will be available.

As you look into this, you will probably see that Xapian is the recommended search option, but it is not the default. Once I got it working, Xapian was fine for me. If I recall correctly,it took me a while to figure out the configuration, because I had a hard time finding examples.

–

First of all, my message of 17 August 2023 contains a critical error. _Xapian_ is the recommended search option, but _Whoosh_ is the default. Replace 'Whoosh' with 'Xapian' in my message of 17 August.

===

This message on the Debian Users mailing list was very helpful to me when I was struggling to configure mailman3.

https://lists.debian.org/debian-user/2022/07/msg00528.html

===

This message thread on the mailman3 mailing list may be helpful:

https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/ZSSVDKKVKZYNQFKLTLWC4V6IIWL7Y3RP/

=== This fragment from /etc/mailman3/mailman-web.py may be helpful as an example:

## 12 Jul 2023 - KLJ - Configure Xapian instead of default Whoosh # # Full-text search engine # HAYSTACK_CONNECTIONS = {

  'default': {
      # You can also use the Xapian engine, it's faster and more accurate,
      # but requires another library.
      # http://django-haystack.readthedocs.io/en/v2.4.1/installing_search_engines.html#xapian
      # Example configuration for Xapian:
      'ENGINE': 'xapian_backend.XapianEngine',
      'PATH': '/var/lib/mailman3/web/fulltext_index',
  },

}

–

Password management: Mailman3 and Postgresql (don't use sqlite for mailman3!) will require the installer to specify some passwords during the setup process. Be prepared, and have a plan in advance for where and how to store and communicate those passwords.

Will use Debian packages.

No need to create mailman database and user before installing packages.

Package mailman3 depends on any of 4 dbconfig-* packages. Currently installed is "dbconfig-common", which is not acceptable. Testing shows installation of "mailman3" package without specifying any "dbconfig-*" will also install "dbconfig-sqlite3" by default. Specifying "dbconfig-mysql" will install that and not "dbconfig-sqlite3".

What packages to install?

• mailman3 mailman3-web
• python3-mailman-hyperkitty
• python3-django-postorius
• python3-django-mailman3
• listadmin
• dbconfig-mysql
• Skipping xapian (not Whoosh, but mailman3-web depends on python3-whoosh)
• And -doc for any package that has it, except prereq libraries: mailman3-doc xapian-doc

Since the Debian package of "mailman3-web" depends on "python3-whoosh", will not install "xapian" initially, as it just complicates the procedure. If desired, can install and use "xapian" later.

export DEBIAN_PRIORITY=low
export DEBIAN_FRONTEND=readline
export PROD_LIST="mailman3 mailman3-web python3-mailman-hyperkitty \
        python3-django-postorius python3-django-mailman3 listadmin \
        mailman3-doc dbconfig-mysql  python3-pymysql python3-mysqldb"
script -a /var/tmp/mailman3-install-log.txt
TERM=dumb apt-get install $PROD_LIST
Answer the installation dialog questions, see below.
# End the script before continuing

> Email address of the Postorius superuser: | mailing-lists@sluug.org |

Password for the Postorius superuser: | (Hidden) |
Web server(s) to configure automatically: | 2 |
Should the webserver(s) be restarted now? | yes |

^ Configuring libpaper1 ^^

System's default paper size: | 1 |

^ It now starts installing and setting up all the other packages ^^

The Hyperkitty key in "/etc/mailman3/mailman-web.py" ("MAILMAN_ARCHIVER_KEY") and "/etc/mailman3/mailman-hyperkitty.cfg" ("api_key") need to match. In testing, "MAILMAN_ARCHIVER_KEY" was set to "SecretArchiverAPIKey".

export DEBIAN_PRIORITY=low
export DEBIAN_FRONTEND=readline
dpkg-reconfigure python3-mailman-hyperkitty

This didn't ask any questions, exiting with message: "Replacing config file /etc/mailman3/mailman-hyperkitty.cfg with new version". Now both files have the same key.

As documented in "/usr/share/doc/mailman3/README.Debian", edit /etc/postfix/main.cf to "add the following settings". Also verify there were none of those settings already in it.

Verify the hash files are in "/var/lib/mailman3/data/" However, not readable by user/group, except "list". As of installation, both are empty, except for comments.

Restart postfix with "postfix reload".

• The installation configuration scripts created a sym link from "/etc/apache2/conf-available/mailman3.conf" to "/etc/mailman3/apache.conf".
• And a sym link from "/etc/apache2/conf-enabled/mailman3.conf" to "../conf-available/mailman3.conf".
• The installation also created sym links in "/etc/apache2/mods-enabled/" to existing files in "../mods-available/" for 3 "proxy*".
• So this is already done.

• Change "TIME_ZONE" from "UTC" to "America/Chicago"
• Also review "/usr/share/mailman3-web/settings.py" for other settings.

• Change "site_owner" from "changeme@example.com" to the same used above for "Postorius superuser".

• mailman-hyperkitty.cfg: "base_url" - no change, since "localhost".
• mailman-web.py: "POSTORIUS_TEMPLATE_BASE_URL" - no change, since "localhost".
• mailman.cfg" "use_https" - leave at "no", this is for internal use?

Update "/etc/logrotate.conf" to add before the "include" line:

tabooext + .bak .off .out .save
taboopat + *[-.][0-9][0-9][0-9][0-9][0-9][0-9]
taboopat + *[-.][0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]

Update "/etc/logrotate.d" files "mailman3" and "mailman3-web" to use monthly rotation, longer retention, file name format, permissions, etc, Also using a wild card for multiple log files.

Update "/etc/mailman3/mailman.cfg" to add some separate log files, to match some in Mailman 2. After the "[logging.debian]" section, add new subsections:

[logging.bounce]
path: bounce.log
propagate: yes
[logging.vette]
path: vette.log
[logging.mischief]
path: mischief.log
[logging.subscribe]
path: subscribe.log
[logging.error]
path: error.log
propagate: yes
[logging.archiver]
path: archiver.log

Restart both "mailman" and "mailman-web". Change existing logs to make readable.

"mailman-wrapper create --language no --owner postmaster@___ -no-notify ____@lists.__

Set the list description via "Django site admin".

Verify Postfix hashes are accessible and automatically populated.

Test subscriptions.

Test posting.

Test archives.

Do archives hide the address of the poster? No.

The acceptable operation is that ASCII postings are delivered and archived as plain text ASCII. Not converted to "quoted-printable" or "base64". As discovered with Mailman 2, when the default footers are added, the charset is "utf-8", and the Content-Transfer-Encoding: is "base64". There might also be a concern with added content breaking DKIM signatures.

mkdir -p /var/lib/mailman3/templates/site/en/
touch    /var/lib/mailman3/templates/site/en/list:member:generic:footer.txt
chown -R list:list /var/lib/mailman3/templates/site/

This worked, and the postings are now delivered without being converted.

Or do it via the web interface at URL …/mailman3/postorius/lists/testmm3.bock2.sluug.org/templates That saves it in the database instead of as a file. Always use one or the other, don't mix them. This is only for templates of things added to mail, not changing web pages. Since this change was done for Mailman 2 via the "config_list" command to update each list in their database, the same should be done for Mailman 3. It should be done this way for existing lists when they are imported.

What about "list:member:digest:footer" for a digest message? In the directory of template files, there is no separate footer for regular and digest, only "generic:footer". In the web pages for creating templates, there is no "generic:footer", but there are "regular:footer" and "digest:footer".

Many places are still referring to "example.com".

• "Subject: [example.com]"
• "… account on example.com."
• "Thank you for using example.com!"

This is in database "mm3webdb" table "django_site". Change by logging in to the web site as admin and going to URL "…/mailman3/admin/sites/site/1/change/". Change the "Domain name" value to "sluug.org" and the "Display name" value to "SLUUG mailing lists".

Everything below has not been tested yet and is for planning

Check archive accuracy.

Configure to delete attachments from the archive during import?

Test list deletion.

Does creating a list stop the errors about "gatenews"?

Will "/usr/local/bin/mailman_queue_check.sh" work?

Protect archives from address scraping and secure private lists. The protection method used for Mailman 2 doesn't apply to Mailman 3. Instead, make the list archive private, forcing subscribers (Or owners and moderators) to log in with their personal credentials to see the archive. This means there is no configuration at this time, unless we decide to make all archives public, and use the old protection method.

Test new configuration options and enhancements to make now. Especially any global options that could impact all production lists.

TBD:

• "header_checks" are now global?
• "[dmarc]" and/or "[ARC]".

For each list:

mailman-wrapper import21 ...
sudo -u www-data /usr/share/mailman3-web/manage.py hyperkitty_import ...
sudo -u www-data /usr/share/mailman3-web/manage.py update_index_one_list ...

Check all configuration web pages on the old and new systems, to make sure no options, settings, or values were lost.

Test …

Index mailing list archives.

Delete or deflate HTML postings and delete attachments on all lists. The "Collapse alternatives" and "Convert html to plaintext" otpions.

Never use the "mailman" command, only "mailman-wrapper" that runs as user "list".

"mailman-web" - ?

"django-admin" - ?

• https://HOSTNAME/mailman3 / - General list access and user login. What should be announced to humans.
• https://HOSTNAME/mailman3/postorius/lists/ - What just specifying "mailman3" redirects to. What should be used in links from the main web site.

• https://HOSTNAME/mailman3/hyperkitty/ - long list of "Available lists", with "Sign In" for regular list members.
• https://HOSTNAME/mailman3/postorius/ - short list of "Mailing Lists".
• https://HOSTNAME/mailman3/postorius/lists/ - the same as just "postorius/".

• https://HOSTNAME/mailman3/admin/ - Administration login, only "for a staff account".
• https://HOSTNAME/mailman3/LISTNAME.HOSTNAME/ - Access private lists not published on the index of lists. You will still have to log in or directly subscribe to do anything more than see the list description.
• https://HOSTNAME/mailman3/hyperkitty/list/LISTNAME@HOSTNAME/ - Access private archives. You will still have to log in to see the private archive.

You can use the general URL for lists, then login and you will see private "lists that you are owner, moderator or subscriber for".

Trying to log into the general URL with the admin account might fail with a 500 error.

• Logging into the general URL with your personal accout/password used for list subscription doesn't give access to list administration, only changing your account and relationships to lists.
• If you are the bbbThis is even if your account is the list owner.
• Use this with your personal credentials to login and control "lists that you are owner, moderator or subscriber for".

Use the admin URL, then log in, then either select

• "Mailing lists" under the "HyperKitty", then a list to change description, etc.
• "View site" in the heading,
  • "Mailman settings" to change subscription settings for this address.
  • A list to change "List Settings", handle pending actions, etc.

Watch for the logout confirmation web page.