This document will be used to coordinate the effort associated with replacing Mailman 2.

https://www.sluug.org/pipermail/sysadmin/2023-August/010383.html

Date: Thu, 17 Aug 2023 11:31:08 -0500
From: Ken Johnson
Subject:  Mailman 3

mailman and mailman3 can co-exist under Debian 11. The cli commands are completely different, for example. I did not convert until on Debian 11. There can be python issues, of course – the longer you wait to dump python2, the more that can be messy, though I did not have very many issues personally. In some ways it may be easier to convert after the update, because the version of mailman3 you are moving to is closer to current, so it is easier to find help and information online. Also, for SLUUG the mailing list is pretty exposed to the internet, so best to avoid delay in that conversion, since no updates for mailman will be available.

As you look into this, you will probably see that Xapian is the recommended search option, but it is not the default. Once I got it working, Xapian was fine for me. If I recall correctly,it took me a while to figure out the configuration, because I had a hard time finding examples.

–

First of all, my message of 17 August 2023 contains a critical error. _Xapian_ is the recommended search option, but _Whoosh_ is the default. Replace 'Whoosh' with 'Xapian' in my message of 17 August.

===

This message on the Debian Users mailing list was very helpful to me when I was struggling to configure mailman3.

https://lists.debian.org/debian-user/2022/07/msg00528.html

===

This message thread on the mailman3 mailing list may be helpful:

https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/ZSSVDKKVKZYNQFKLTLWC4V6IIWL7Y3RP/

=== This fragment from /etc/mailman3/mailman-web.py may be helpful as an example:

## 12 Jul 2023 - KLJ - Configure Xapian instead of default Whoosh # # Full-text search engine # HAYSTACK_CONNECTIONS = {

  'default': {
      # You can also use the Xapian engine, it's faster and more accurate,
      # but requires another library.
      # http://django-haystack.readthedocs.io/en/v2.4.1/installing_search_engines.html#xapian
      # Example configuration for Xapian:
      'ENGINE': 'xapian_backend.XapianEngine',
      'PATH': '/var/lib/mailman3/web/fulltext_index',
  },

}

–

Password management: Mailman3 and Postgresql (don't use sqlite for mailman3!) will require the installer to specify some passwords during the setup process. Be prepared, and have a plan in advance for where and how to store and communicate those passwords.

Will use Debian packages.

No need to create mailman database and user before installing packages.

Package mailman3 depends on any of 4 dbconfig-* packages. Currently installed is "dbconfig-common", which is not acceptable. Testing shows installation of "mailman3" package without specifying any "dbconfig-*" will also install "dbconfig-sqlite3" by default. Specifying "dbconfig-mysql" will install that and not "dbconfig-sqlite3".

What packages to install?

• mailman3 mailman3-web
• python3-mailman-hyperkitty
• python3-django-postorius
• python3-django-mailman3
• listadmin - no, this only for MM2
• dbconfig-mysql
• Skipping xapian (not Whoosh, but mailman3-web depends on python3-whoosh)
• And -doc for any package that has it, except prereq libraries: mailman3-doc xapian-doc

Since the Debian package of "mailman3-web" depends on "python3-whoosh", will not install "xapian" initially, as it just complicates the procedure. If desired, can install and use "xapian" later. After installation and testing, search works without any extra configuration.

The following list of packages does not include around 200 additional packages that will be installed automatically.

export DEBIAN_PRIORITY=low
export DEBIAN_FRONTEND=readline
export PROD_LIST="mailman3 mailman3-web python3-mailman-hyperkitty \
        python3-django-postorius python3-django-mailman3 \
        mailman3-doc dbconfig-mysql  python3-pymysql python3-mysqldb"
script -a /var/tmp/mailman3-install-log.txt
TERM=dumb apt-get install $PROD_LIST
Answer the installation dialog questions, see below.
# End the script before continuing

The Hyperkitty key in "/etc/mailman3/mailman-web.py" ("MAILMAN_ARCHIVER_KEY") and "/etc/mailman3/mailman-hyperkitty.cfg" ("api_key") need to match. In testing installation, "MAILMAN_ARCHIVER_KEY" was set to "SecretArchiverAPIKey". In production installation, it was the opposite, "api_key" was set to "SecretArchiverAPIKey".

export DEBIAN_PRIORITY=low
export DEBIAN_FRONTEND=readline
dpkg-reconfigure python3-mailman-hyperkitty

This didn't ask any questions, exiting with message: "Replacing config file /etc/mailman3/mailman-hyperkitty.cfg with new version". Now both files have the same key.

• Change "TIME_ZONE" from "UTC" to "America/Chicago"
• In the "INSTALLED_APPS" values, comment 'django_mailman3.lib.auth.fedora'. Apparently you can't comment 'allauth.socialaccount' or it will fail. Commenting the fedora entry will remove the "Please sign in with one of your existing third party accounts. Or …".
• The builtin Gravatar support can be disabled by adding "HYPERKITTY_ENABLE_GRAVATAR = False". However, this didn't seem to work in testing. Neither did "GRAVATAR_SECURE_URL = ''".
• Also review "/usr/share/mailman3-web/settings.py" for other settings.

• Change "site_owner" from "changeme@example.com" to the same used above for "Postorius superuser".

• mailman-hyperkitty.cfg: "base_url" - no change, since "localhost".
• mailman-web.py: "POSTORIUS_TEMPLATE_BASE_URL" - no change, since "localhost".
• mailman.cfg" "use_https" - leave at "no", this is for internal use?

Update "/etc/logrotate.conf" to add before the "include" line:

tabooext + .bak .off .out .save
taboopat + *[-.][0-9][0-9][0-9][0-9][0-9][0-9]
taboopat + *[-.][0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]

Update "/etc/logrotate.d" files "mailman3" and "mailman3-web" to use monthly rotation, longer retention, file name format, permissions, etc, Also using a wild card for multiple log files.

Update "/etc/mailman3/mailman.cfg" to add some separate log files, to match some in Mailman 2. After the "[logging.debian]" section, add new subsections:

[logging.bounce]
path: bounce.log
propagate: yes
[logging.vette]
path: vette.log
[logging.mischief]
path: mischief.log
[logging.subscribe]
path: subscribe.log
[logging.error]
path: error.log
propagate: yes
[logging.archiver]
path: archiver.log

Update "/etc/mailman3/uwsgi.ini" to add some separate log files, by uncommenting the three existing commented sections for "req", "cron", and "qcluster". And reverse the commenting of the two lines in the "Last log" section. Also, on the "plugins" line, add ",logfile" to the end.

Change existing logs to make them world readable.

Verify "/etc/mailman3/mailman.cfg" and "/etc/mailman3/mailman-web.py" are not world readable.

Restart both "mailman3" and "mailman3-web".

date;systemctl restart mailman3;date
date;systemctl restart mailman3-web;date

This is done now to allow changes and testing that can't be done without a list. Configuration and testing of this list will be later.

mailman-wrapper create --language no --owner postmaster@___ -no-notify ____@sluug.org

• The installation configuration scripts created a sym link from "/etc/apache2/conf-available/mailman3.conf" to "/etc/mailman3/apache.conf".
• And a sym link from "/etc/apache2/conf-enabled/mailman3.conf" to "../conf-available/mailman3.conf".
• The installation also created sym links in "/etc/apache2/mods-enabled/" to existing files in "../mods-available/" for 3 "proxy*".
• So this is already done. Mailman 3 will work on any site.

Except "mailman3.conf" being enabled as a global configuration option means mailman3 is available on all sites, not just the one site it is intended to be used on. So we need to disable the "mailman3.conf" and add similar configuration to the site(s) where mailman3 is to be available.

• Disable the global configuration with "a2disconf mailman3".
• Edit the "/etc/apache2/sites-available/" file for any sites to host mailman3 to add "Include conf-available/mailman3.conf" just after the MM2 section.
• Reload apache with "systemctl reload apache2".
• Test.

However, the hyperkitty configuration is set to use "http://localhost" in several configuration files, so would need to change them, or expose "localhost" as another apache site name. Do this by editing the site definition for the main site to add this and reload apache:

    <VirtualHost localhost:80 localhost:443>
      ServerName localhost
        # Statements that are identical between the ports for this site
        Include sites-available/common-www.sluug.org.conf
    </VirtualHost>

For a problem where the "3" of "mailman3" is lost when it is at the end of the URL when used in a virtualhost, add this before the "Include" to add a "/":

<Location "/mailman3">
    RedirectMatch "(.*/mailman3)$" "$1/"
</Location>

As documented in "/usr/share/doc/mailman3/README.Debian", edit /etc/postfix/main.cf to "add the following settings". Also verify there were none of those settings already in it.

However, have to later change this again, see the "Virtual mailbox incompatibility" section below. Making both changes before creating the "postfix_vmap" below will result in errors and incoming mail being temporarily rejected.

Verify the hash files are in "/var/lib/mailman3/data/" However, not readable by user/group, except "list". As of installation, both are empty, except for comments. They are populated when a list is created.

Restart postfix with "postfix reload".

Customize the template used for list archives access by users.

• Must create the directory first: "mkdir /var/lib/mailman3/templates/hyperkitty/"
• Create and edit "/var/lib/mailman3/templates/hyperkitty/top.html".
• Include the comment in "/usr/lib/python3/dist-packages/hyperkitty/templates/hyperkitty/top.html",
• Copy the SLUUG logo section of "/srv/www/www.sluug.org/public/header.html" into it.
• Do the same for "/var/lib/mailman3/templates/hyperkitty/bottom.html".
• Include the comment in "/usr/lib/python3/dist-packages/hyperkitty/templates/hyperkitty/bottom.html",
• Copy the webmaster section of "/srv/www/www.sluug.org/public/footer.html" into it. Also the copyright notice if desired.
• Make sure both are owned by user:group "root:root" and world readable.

Customize the template used for list web access by users.

• Must create the directory first: "mkdir /var/lib/mailman3/templates/postorius/"
• Create with: "cp -ip /usr/lib/python3/dist-packages/postorius/templates/postorius/base.html /var/lib/mailman3/templates/postorius/"
• Edit "/var/lib/mailman3/templates/postorius/base.html".
• Copy the SLUUG logo section of "/srv/www/www.sluug.org/public/header.html" into it, just after the "<body>" tag.
• Copy the webmaster section of "/srv/www/www.sluug.org/public/footer.html" into it, just after the "</body>" tag. Also the copyright notice if desired.
• Make sure it is owned by user:group "root:root" and world readable.

These templates will not be used unless "mailman-web.py" is updated to point to them. Edit "mailman-web.py" and copy the entire "TEMPLATES" section of "/usr/share/mailman3-web/settings.py" to the end of it. The add two entries to the 'DIRS' array.

        'DIRS': ['/var/lib/mailman3/templates/postorius',
                '/var/lib/mailman3/templates'],

Restart "mailman3-web".

The choice of "/var/lib/mailman3/templates/" is based on it being listed in "mailman.cfg". That is for mail text templates, not HTML, but no better location was found for the HTML templates. There is no "/var/lib/mailman3-web/". The configuration files for the web packages is the same as the base package. Another option might be "/etc/mailman3/templates/" for the HTML templates.

Create aliases for "mailing-lists" and "postoriusroot" to forward to "root". However, with virtual mailboxes, have to also create a virtual alias.

Log in as regular user "postoriusroot" and set the names (First="Mailman", Last="Administrator") and time zone in the "Account" for the user. Or can do the same as the administrator, but with different web pages.

This is the list created above that wasn't configured.

See the below steps to do now, such as for creating an mbox archive.

Set the list description via "Django site admin".

Verify Postfix hashes are accessible and automatically populated.

Test subscriptions.

Test posting.

Test archives.

Do archives hide the address of the poster? No.

The acceptable operation is that ASCII postings are delivered and archived as plain text ASCII. Not converted to "quoted-printable" or "base64". As discovered with Mailman 2, when the default footers are added, the charset is "utf-8", and the Content-Transfer-Encoding: is "base64". There might also be a concern with added content breaking DKIM signatures.

mkdir -p /var/lib/mailman3/templates/site/en/
touch    /var/lib/mailman3/templates/site/en/list:member:generic:footer.txt
chown -R list:list /var/lib/mailman3/templates/site/

This worked, and the postings are now delivered without being converted.

Or do it via the web interface at URL …/mailman3/postorius/lists/testmm3.sluug.org/templates That saves it in the database instead of as a file. Always use one or the other, don't mix them. This is only for templates of things added to mail, not changing web pages. Since this change was done for Mailman 2 via the "config_list" command to update each list in their database, the same should be done for Mailman 3. It should be done this way for existing lists when they are imported.

What about "list:member:digest:footer" for a digest message? In the directory of template files, there is no separate footer for regular and digest, only "generic:footer". In the web pages for creating templates, there is no "generic:footer", but there are "regular:footer" and "digest:footer".

Many places are still referring to "example.com".

• "Subject: [example.com]"
• "… account on example.com."
• "Thank you for using example.com!"

This is in database "mm3webdb" table "django_site". Change by logging in to the web site as admin and going to URL "…/mailman3/admin/sites/site/1/change/" via "View site" ⇒ "Domains" ⇒ "Edit". Change the "Domain name" value to "sluug.org" and the "Display name" value to "SLUUG mailing lists".

In /var/log/syslog, every 15 minutes:

mariadbd[____]: ... [Warning] Aborted connection ____ to db: 'mm3db' user: 'mm3dbuser' host: 'localhost' (Got an error reading communication packets)

Does creating a list stop the errors about "gatenews"? No. Still seen in mariadb logs. Commented the "gatenews" entry in "/etc/cron.d/mailman3". This stopped the warning every 15 minutes, but there are still two each day, at the time of the other cron entries. Searching online found this is a common problem and is just an annoyance with nothing wrong. It might be a problem with "SQLAlchemy" not issuing "quit" to properly close the connection.

There is a postfix configuration conflict that results in warning "do not list domain in BOTH virtual_mailbox_domains and relay_domains". This is a well known problem. See https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.html#unusual-postfix-configuration

While logged into the admin account, go through links "View site" ⇒ "Domains" ⇒ "Edit"(see below)

On the "Domains" page, there are two "Edit" links. The link near the "Web Host" descriptive text and (domain.name) changes those for the "site". The link between the "Templates" and "Delete" links for "Action" changes the "Description" and "Alias Domain" for the "domain. Use this.

For each domain, add a unique and unpublished name not in DNS. We will prepend "mm3-alias-domain." to the front of the normal name.

This might not update the hash files and require restarting mailman3 to regenerate them.

And have to change the MM entries in "/etc/postfix/main.cf":

• Comment the "local_recipient_maps" entry added for MM3 to return to default.
• Since there already is a "virtual_alias_maps" setting, update it to add the new "postfix_vmap" hash to that.

Messages posted with certain non-ASCII characters may fail to be imported to the MM3 archives. Most non-ASCII characters cause no problems. This was encountered during import of MM2 list archives, but ignored since only a tiny percentage of posts were rejected during import. Once in production, a single posting with 32-bit characters resulted in the archive of that posting failing, but staying in the archive queue to be retried and fail again for each new posting to any list. Each failure sent a 500 line e-mail to the root user.

Both import failures and the one new posting only gave clues like: "Incorrect string value: '\\xF0\\x9F\\x94\\x97". The string contents would vary slightly, but all were 4 octets long.

The solution was found in mailing list posts, not MM3 documentation. See https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/B6MEKV57XHO336MSIQXQLV4NUPRS3PQJ/ The default appears to only handle 24-bit characters. Anything with 32-bit characters results in this failure. This requires changes in two places.

ALTER TABLE hyperkitty_email
 DEFAULT CHARACTER SET utf8mb4,
 MODIFY subject varchar(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci,
 MODIFY content longtext     CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

In "/etc/mailman3/mailman-web.py" DATABASES ⇒ default ⇒ OPTIONS Add "'charset': 'utf8mb4'" to the list of key:value in the "OPTIONS" section.

When done, a posting to a test list archived both it and the one that was stuck. If this had been known months ago, the problem with postings being rejected during MM2 archive import would probably been avoided.

Will "mailman_queue_check.sh" work? Mailman 3 might be compatible with minor changes. Mailman 2 uses "/var/lib/mailman/qfiles/". Mailman 3 uses "/var/lib/mailman3/queue/".

Check archive accuracy.

No way was found to scrub attachments when importing past postings to the archive. Settings for filtering attachments and MIME types seems to be in the mail processing, not archiving. Initial testing shows about half the attachments from the MM2 archives are imported to MM3, others aren't. Part of this is because many attachments were imported to MM2 twice.

Test list deletion:

• Delete the list
• Delete the archive

Protect archives from address scraping and secure private lists. The protection method used for Mailman 2 doesn't apply to Mailman 3. Instead, make the list archive private, forcing subscribers (Or owners and moderators) to log in with their personal credentials to see the archive. This means there is no configuration at this time, unless we decide to make all archives public, and use the old protection method.

Test new configuration options and enhancements to make now. Especially any global options that could impact all production lists.

Create a replacement for the MM2 ".mbox" archive backup, in case a similar import of archived mail files is needed in the future.

mkdir             /var/lib/mailman3/mm3-archive-mbox/
chown nobody:list /var/lib/mailman3/mm3-archive-mbox/
chmod 750         /var/lib/mailman3/mm3-archive-mbox/
((For each list to be migrated, create an empty .mbox file.))
listname=____
touch             /var/lib/mailman3/mm3-archive-mbox/$listname.mbox
chown nobody:list /var/lib/mailman3/mm3-archive-mbox/$listname.mbox
chmod 640         /var/lib/mailman3/mm3-archive-mbox/$listname.mbox

Edit "/etc/aliases" to add one line for each list, in the form:

LISTNAME-mm3-archive-mbox:  /var/lib/mailman3/mm3-archive-mbox/LISTNAME.mbox

After a reboot or restarting multiple processes, Mailman will often try to connect to the SQL database server before that server has completed startup to the point it can accept connections and perform operations. This makes the Mailman service fail and terminate. This is a well known problem. There is no systemd configuration present to state Mailman depends on the SQL service. This has been corrected in later Debian Mailman releases.

Correction is by a systemd override to establish the dependency. This was done with this command: "systemctl edit mailman3". Then entered the desired settings into the editor:

    [Unit]
    Requires=network.target network-online.target mariadb.service
    After=network.target network-online.target mariadb.service
    BindsTo=mariadb.service
    [Service]
    ExecStartPre=/bin/sleep 20
    TimeoutStartSec=180
    Restart=on-failure
    RestartSec=15

Confirm with command:

    systemctl show mailman3 --property=ExecStartPre --property=TimeoutStartSec
    systemctl show mailman3 --property=Requires --property=After

To allow viewing the "override.conf" file as a regular user:

    chmod a+rx /etc/systemd/system/mailman3.service.d
    chmod a+r /etc/systemd/system/mailman3.service.d/override.conf

To check the archives for potential errors before attempting import:

• Use cleanarch or cleanarch3 to check for bad From and Date headers.
• Use check_hk_archive (replaced by check_hk_import) to check for messages that would fail import.

While cleanarch is a standard part of MM2, cleanarch3 is not in the currently installed Debian python3-django-hyperkitty package. It is in the more recent 1.3.12-1 (1.3.8 and later) release. The same for check_hk_archive (replaced by check_hk_import). Download the latest package, extract the two scripts to a temporary location.

For 3 lists, that existed before the 2007 move from Majordomo, the "/var/lib/mailman/archives/private/LISTNAME.mbox/LISTNAME.mbox" file doesn't go back as far as the original Majordomo files imported into mailman. The monthly text files created during the import to generate the HTML do have all the mail. In "var/lib/mailman/archives/private/LISTNAME/" are the "YYYY-MMMMM.txt" and "YYYY-MMMMM.txt.gz" files for each month since 2000. Should just need to concatenate them in the proper order. Since these are not being updated, this preparation work can be done anytime before the archive import.

months='January
February
March
April
May
June
July
August
September
October
November
December'
years='2000 2001 2002 2003 2004 2005 2006 2007'

archive_path=/var/lib/mailman/archives/private
export listname=____

for year in $years
  do
    for month in $months
      do
        file="${archive_path}/${listname}/$year-$month.txt"
        #echo "$file"
        if [ -f "$file" ]
          then
            cat "$file"
          fi
      done
  done > /tmp/${listname}-rebuilt.txt

Edit /tmp/${listname}-rebuilt.txt and delete everything starting with the
first message in the mbox file until the end.
Except the original monthly text file for the initial import from Majordomo
has sections that go back in time, and some are repeated.
This makes it more complicated to determine what to delete or keep.

Also have to change the "From " lines that were massaged to obscure the sender.
This is needed for the postings to be recognized as valid mail for import.
Before that, search for any that are incorrect "From " lines within a message.
These might have been archived that way.  In that case, leave them alone.
<code>
1G/^\CFrom .*@/ Any within a message to fix
:%s/^\(\CFrom \w\w*\)\( \+\w\w\w \w\w\w .\d \d\d:.*200\d\)$/\1@localhost\2/ " Lines with no domain
:%s/^\(\CFrom .*\) at \(.*200\d\)$/\1@\2/ " Make normal mail standard line

Run the two checking tools, the same as below. The "From " will probably change for a small number of messages.

/tmp/cleanarch3 \
    /tmp/${listname}-rebuilt.txt \
    > /tmp/${listname}-rebuilt.mbox
diff \
    /tmp/${listname}-rebuilt.txt \
    /tmp/${listname}-rebuilt.mbox

/tmp/check_hk_import /tmp/${listname}-rebuilt.mbox

MM3 restricts the list "info" field to under 255 characters. Change any lists that are longer, to avoid truncation during import. Most MM2 production lists have much longer text. They also contain information that will be obsolete, such as how to access the archives. The text could be changed before or after import. Even if changed after import, planning what the text should be can speed the process.

Notify other admins that mail is being stopped, in case it might be mistake for a problem.

Stop postfix, to prevent any incoming mail going to the mailman 2 queue while in the process of transitioning. Not needed for low volume lists where no postings can be expected during this period.

Since the import and indexing of a large archive can take three hours, use a technique to continue running if an interactive session is lost. Don't attempt during a storm.

Stop mailman 2 …

Must create an empty list before import. For each list:

export config_path=/var/lib/mailman/lists
export archive_path=/var/lib/mailman/archives/private
export listname=____
export listfull=${listname}@sluug.org

/tmp/cleanarch3 \
    ${archive_path}/${listname}.mbox/${listname}.mbox \
    > /tmp/${listname}-cleaned.mbox
diff \
    ${archive_path}/${listname}.mbox/${listname}.mbox \
    /tmp/${listname}-cleaned.mbox

/tmp/check_hk_import  /tmp/${listname}-cleaned.mbox

Check the number of archive messages is reasonable.
The MM2 list archives imported from Majordomo have many duplicate HTML files.
ls ${archive_path}/${listname}/[12]???-* | grep -c 0......html

Create the new empty list:
mailman-wrapper create   ${listfull}
((The only output is that the list was created))

mailman-wrapper import21 ${listfull} ${config_path}/${listname}/config.pck
((If there are no problems, there might be no output))
((The only output might be errors and "Importing members" progress))

chmod a+r /tmp/${listname}-*.mbox # Since run as "www-data" user

((Include "rebuilt.mbox" only if it was created.))
date ; time mailman-web hyperkitty_import --no-sync-mailman \
    --verbosity 3 --no-color \
    --list-address ${listfull} \
    /tmp/${listname}-rebuilt.mbox \
    /tmp/${listname}-cleaned.mbox \
    >> /tmp/import-${listname}-log.txt 2>&1 ; date
((It lists each message as it is processed.))

date ; time mailman-web update_index_one_list --verbosity 2 --no-color \
    ${listfull} \
    >> /tmp/index-${listname}-log.txt 2>&1 ; date
((There is no output during the first phase, a quick summary of
the number indexed, then nothing during last phase))

((Move the temporary mbox files to a permanent location for future reference.))
mv -iv /tmp/${listname}-*.mbox /var/lib/mailman3/mm2-reference/
gzip -9v /var/lib/mailman3/mm2-reference/*.mbox

Delete the list from the Mailman 2 configuration by moving the config out of the production directory.

mv -iv ${config_path}/${listname} ${config_path}-disabled/

Create a list member pointing to the mbox alias created above. Since "/etc/aliases" is used, the domain must one accepted by the mail system, but not one handled by the virtual alias function.

echo $listname-mm3-archive-mbox@lists.sluug.org \
    | mailman-wrapper addmembers - $listfull
mailman-wrapper findmember $listname-mm3-archive-mbox@lists.sluug.org

Do a quick check of the import and archives with URL: http://localhost/mailman3/postorius/lists/$listfull

Must set archives to private in either of the two places. The "View site" option and the "HyperKitty" option both control both the listing in the index of archives, and access without being a list subscriber. Changing either controls both aspects of archive access. Changing the value on the "View site" web page is reflected on the "HyperKitty" page, but the reverse is not true. So there is some unknown difference. Use a separate browser to confirm the private status is in effect by trying to access the archive index and directly to the archive as an anonymous user.

TBD settings:

• "header_checks" are now global? In global settings, but imported per list.
• "[dmarc]" and/or "[ARC]". No changes at the time of transition.
• "charset". Leaving as "us-ascii".

Test by posting and web access?

Automated postings from the test system are sent using an alternate domain outside the virtual mailbox domain. Since MM2 works with any incoming mail domain and MM3 lists are for specific mail domains, the postings are still processed by MM2. The solution is to add an alias to "/etc/aliases" for any affected list, to redirect to the specific mail domain.

Delete Mailman 2 list aliases and create Mailman 3 aliases? No. Mailman 2 uses "/var/lib/mailman/data/aliases" for "alias_maps". Mailman 3 uses "/var/lib/mailman3/data/postfix_lmtp" for "local_recipient_maps". No action is needed. The new list aliases are created when the lists are imported. The "local_recipient_maps" takes precedence over "alias_maps".

Start mailman 2 …

Start postfix if it was stopped.

Check all configuration web pages on the old and new systems, to make sure no options, settings, or values were lost.

Change the list "info" field for each list.

Check the list archive HyperKitty Description and Archive policy. Might need to set Owners and Moderators.

One list had MM2 "bounce_matching_headers" longer than the 255 characters limit. This was truncated, but at a point that results in "bad bounce_matching_header line" for every posting. There is no MM3 web page to edit this, so direct SQL table editing is needed.

Test …

Customize the MM2 "listinfo" to include a notice of the transition, and add links to the MM3 and MM2 archives for the three advertised lists. This will apply to the overview and to specific MM2 list pages. Do this by editing "/var/lib/mailman/Mailman/Cgi/listinfo.py" and:

• Change the "legend" setting to state this is for MM2. This is used for the overview.
• For the overview, add to the "welcome" setting using "extend" to add notices of the migration and links to MM3 for the advertised lists, and the MM2 archives for those lists.
• For the list details, add text to the "replacements" setting for "<mm-list-description>" with a notice that the list will be migrated soon. After a list is migrated, this will no longer be seen.

Delete or deflate HTML postings and delete attachments on all lists. The "Collapse alternatives" and "Convert html to plaintext" options.

Change the "Action to take on messages which have no content" setting from "Discard" to "Reject".

Create "/etc/cron.d/mailman3-local" that runs some check that all expected processes are running.

• 15 "^list .*/usr/lib/mailman3/bin/" for "mailman3",
• 2 "www-data .*uwsgi" for "mailman3.web",
• 6-7 "manage.py qcluster" for "mailman3.web",
• Perhaps also check the output of the "mailman-wrapper status" command for "running".
• Check for things in the postfix queue for "mm3-alias-domain.*", since things are stuck there when MM is unable to receive them.
• Might have a check for pending list moderator actions that are old. Such as using "mailman-wrapper notify –dry-run –verbose". However, this is not needed. There is already a cron job running at 0800 to "send out notices of pending requests to list moderator".

Allow all members of the "adm" group to query for pending moderator actions. Use "visudo" to create "/etc/sudoers.d/mailman3" and add:

    Cmnd_Alias MM_NOTIFY = /usr/bin/mailman notify --dry-run --verbose
    %adm  ALL=(list) NOPASSWD: MM_NOTIFY

Is there a way to disable "Download" of HyperKitty archives as a ".mbox" file? HyperKitty 1.3.6 adds "HYPERKITTY_MBOX_EXPORT".

In the "[mta]" section of "mailman.cfg", consider setting "verp_probes: yes". Also consider "verp_confirmations: yes", "verp_personalized_deliveries: yes", and "verp_delivery_interval: 25". A small interval (in number of messages, to do a VERP recipient address) increases mail delivery overhead, while a large interval will be ineffective in generating enough returned postings to exceed bounce thresholds.

Stop and disable the Mailman 2 service.

Delete "/etc/cron.d/mailman-local" that runs "mailman2_process_check.sh" and "mailman_queue_check.sh". Delete "/etc/cron.d/mailman" that runs periodic support functions.

Delete "/usr/local/bin/mailman2_process_check.sh". Delete "/usr/local/bin/mailman_queue_check.sh".

Test Mailman 3 after this and other significant steps.

Edit "/etc/postfix/main.cf" and remove "hash:/var/lib/mailman/data/aliases" from "alias_maps".

Delete all MM2 list aliases in the SQL virtual alias table.

Save the monthly text files of the Mailman 2 archives.

Delete Mailman 2 from the Apache site configuration. Replace the Mailman 2 "listinfo" page with redirection to MM3 or a static web page that emulates "listinfo".

Remove the "mailman" package.

Remove the "python-dnspython" and "python-is-python2" packages.

Leave all the other Python 2 packages.

Since the services are started by systemd, it is probably better to do start/stop/restart using the systemctl command. Using the direct commands might not update systemd on the service status.

Never use the "mailman" command, only "mailman-wrapper" that runs as user "list".

Useful "mailman-wrapper" commands:

• "mailman-wrapper –help" - display the available commands.
• "mailman-wrapper COMMAND –help" - display help for one command.
• "mailman-wrapper lists" - list the available lists.
• "mailman-wrapper members FULL_LIST_NAME_WITH_HOSTNAME | wc -l" - count of members to one list. In a future release, this might be handled by the "–count-only" option.
• "mailman-wrapper findmember 'REGEXP'" - Display all memberships for matching users.
• "mailman-wrapper restart" - Stop and restart the Mailman runner subprocesses. See above about using the systemctl command.

"mailman-web" - Django's command-line utility for administrative tasks. Runs the "mailman3-web" "manage.py" command as user "www-data". The usefulness of this is mostly limited to scripting. Includes the "mmclient" shell and the mailmanclient library. See:

• https://docs.djangoproject.com/en/5.1/ref/django-admin/
• https://mailman-cli.readthedocs.io/en/latest/src/mailmanclient/cli docs/using_cli_shell.html
• https://mailman-cli.readthedocs.io/en/latest/src/mailmanclient/cli/docs/using_cli_tools.html
• https://docs.mailman3.org/projects/mailmanclient/en/latest/src/mailmanclient/docs/using.html

"django-admin" - Similar to "mailman-web", but without the proper settings.

• https://WEBSERVER/mailman3 / - General list access and user login. What should be announced to humans.
• https://WEBSERVER/mailman3/postorius/lists/ - What just specifying "mailman3" redirects to. What should be used in links from the main web site.
• https://docs.mailman3.org/en/latest/userguide.html - List Member Manual

• https://WEBSERVER/mailman3/hyperkitty/ - long list of "Available lists", with "Sign In" for regular list members.
• https://WEBSERVER/mailman3/postorius/ - short list of "Mailing Lists".
• https://WEBSERVER/mailman3/postorius/lists/ - the same as just "postorius/".

• https://WEBSERVER/mailman3/admin/ - Administration login, only "for a staff account".
• https://WEBSERVER/mailman3/LISTNAME.HOSTNAME/ - Access private lists not published on the index of lists. You will still have to log in or directly subscribe to do anything more than see the list description. This uses a "." instead of "@".
• https://WEBSERVER/mailman3/hyperkitty/list/LISTNAME@HOSTNAME/ - Access private archives. You will still have to log in to see the private archive. This uses a "@".

You can use the general URL for lists, then login and you will see private "lists that you are owner, moderator or subscriber for".

To approve or reject a held message:

• Log into the admin URL
• Go to "View site".
• Select the list name from the table of lists.
• Go to "Held messages".
• Select the check box for the messages to be handled.
• Select the action to perform.

A local copy of some documentation is available in HTML and plain text(rst) at:

• /usr/share/doc/mailman3/
• /usr/share/doc/python3-django-hyperkitty/

Trying to log into the general URL with the admin account might fail with a 500 error.

• Logging into the general URL with your personal account/password used for list subscription doesn't give access to list administration, only changing your account and relationships to lists.
• This is even if your account is the list owner.
• Use this with your personal credentials to login and control "lists that you are owner, moderator or subscriber for".

Use the admin URL, then log in, then either select

• For the mailing list, select "View site" in the heading,
  • "Mailman settings" to change subscription settings for this address.
  • A list to change "List Settings", handle pending actions, etc.
• For list archives, select "Mailing lists" under the "HyperKitty", then a list to change description, etc.
  • There is no command line interface for most HyperKitty actions.
  • To remove the archive for a list, you must use the admin web interface and the "HyperKitty > Mailing lists" (admin/hyperkitty/mailinglist/) web page. This only affects the list archive, not the base list.

When logging out of the web interface, watch for the logout confirmation web page.

There are separate services and processes for the mailing list and web.

• The "mailman3-web" service controls the "uwsgi" processes. and includes "python3 manage.py" processes.
• The "mailman3" service is Mailman 3, for processing mail. For the "/usr/lib/mailman3/bin/" processes.
• The "mailman" service is Mailman 2, not 3.

If you update a template file, you may need to restart the matching service in order for the changed file to be used.

The logs are in "/var/log/mailman3/", with web actions in the "web/" subdirectory.

The structure of log files was changed to be closer to the separate log files of Mailman 2.

In the "mailman.log", expect to see about 1000 "404" errors every day, shortly after midnight for "GNU Mailman REST client" looking for "/users/". The cause is unknown, but appears to be for maintenance activities of archived postings by former list subscribers.