Differences

This shows you the differences between two versions of the page.

--- replace_mailman_2 [2025/05/05 04:57]
SLUUG Administration Correct indention
+++ replace_mailman_2 [2025/11/18 02:13] (current)
SLUUG Administration Progress update
@@ Line 64: / Line 64: @@
 Password management:  Mailman3 and Postgresql (don't use sqlite for mailman3!) will require the installer to specify some passwords during the setup process.  Be prepared, and have a plan in advance for where and how to store and communicate those passwords.
+----
 ====== Planning ======
@@ Line 91: / Line 93: @@
 will not install "xapian" initially, as it just complicates the procedure.
 If desired, can install and use "xapian" later.
+After installation and testing, search works without any extra configuration.
 ====== Installation ======
@@ Line 132: / Line 135: @@
 | Domain name for sender email addresses:                   | sluug.org       |
 | Username of the Postorius superuser:                      | postoriusroot   |
-> Email address of the Postorius superuser:                 | mailing-lists@sluug.org |
+| Email address of the Postorius superuser:                 | mailing-lists@sluug.org |
-> Password for the Postorius superuser:                     | (Hidden)        |
+| Password for the Postorius superuser:                     | (Hidden)        |
-> Web server(s) to configure automatically:                 | 2               |
+| Web server(s) to configure automatically:                 | 2               |
-> Should the webserver(s) be restarted now?                 | yes             |
+| Should the webserver(s) be restarted now?                 | yes             |
 ^  Configuring libpaper1  ^^
-> System's default paper size:                              | 1               |
+| System's default paper size:                              | 1               |
 ^  It now starts installing and setting up all the other packages  ^^
@@ Line 236: / Line 239: @@
 ====== Additional configuration and testing ======
-==== Customize ====
+===== Customize =====
 Create and edit "/var/lib/mailman3/templates/hyperkitty/top.html"
@@ Line 254: / Line 257: @@
 Restart "mailman3-web".
-==== Create a new test list ====
+===== Create a new test list =====
 <code>
@@ Line 272: / Line 275: @@
 Do archives hide the address of the poster?  No.
-=== Encoding of postings ===
+==== Encoding of postings ====
 The acceptable operation is that
@@ Line 304: / Line 307: @@
 there are "regular:footer" and "digest:footer".
-==== Change example.com again ====
+===== Change example.com again =====
 Many places are still referring to "example.com".
@@ Line 332: / Line 335: @@
 Will "mailman_queue_check.sh" work?
-No, mailman 3 has a completely different structure.
+Mailman 3 might be compatible with minor changes.
+Mailman 2 uses "/var/lib/mailman/qfiles/".
+Mailman 3 uses "/var/lib/mailman3/queue/".
 __**Everything below has not been tested yet and is for planning**__
-==== Test import of an existing test list ====
+===== Test import of an existing test list =====
 Check archive accuracy.
 Configure to delete attachments from the archive during import?
+Initial testing shows about half the attachments from the MM2 archives are
+imported to MM3, others aren't.  The difference hasn't been determined.
 Test list deletion:
@@ Line 358: / Line 365: @@
 Especially any global options that could impact all production lists.
-TBD:
+MM3 restricts the list "info" field to under 255 characters.
+Change any lists that are longer, to avoid truncation during import.
+Most MM2 production lists have much longer text.
+They also contain information that will be obsolete, such as how to
+access the archives.  The text could be changed before or after import.
+Even if changed after import, planning what the text should be can speed the process.
+Messages posted with certain non-ASCII characters may fail to be imported
+to the MM3 archives.  Most non-ASCII characters cause no problems.
+TBD settings:
   * "header_checks" are now global?
   * "[dmarc]" and/or "[ARC]".
+  * "charset".  Leaving as "us-ascii".
+==== Mailman 3 fails if SQL not ready ====
+After a reboot or restarting multiple processes, Mailman will often try
+to connect to the SQL database server before that server has completed
+startup to the point it can accept connections and perform operations.
+This makes the Mailman service fail and terminate.
+This is a well known problem.
+There is no systemd configuration present to state Mailman depends on the
+SQL service.  This has been corrected in later Debian Mailman releases.
+Correction is by a systemd override to establish the dependency.
+Detailed actions are to be documented here. ----
 ====== Import Mailman 2 lists ======
-For 4 lists, that existed before the 2007 move from Majordomo, the
+To check the archives for potential errors before attempting import:
+  * Use cleanarch or cleanarch3 to check for bad From and Date headers.
+  * Use check_hk_archive (replaced by check_hk_import) to check for messages that would fail import.
+While cleanarch is a standard part of MM2, cleanarch3 is not in the
+currently installed Debian python3-django-hyperkitty package.
+It is in the more recent 1.3.12-1 (1.3.8 and later) release.
+The same for check_hk_archive (replaced by check_hk_import).
+Download the latest package, extact the two scripts to a temporary location.
+===== Extra steps for 3 lists =====
+For 3 lists, that existed before the 2007 move from Majordomo, the
 "/var/lib/mailman/archives/private/LISTNAME.mbox/LISTNAME.mbox" file
 doesn't go back as far as the original Majordomo files imported into mailman.
@@ Line 372: / Line 416: @@
 are the "YYYY-MMMMM.txt" and "YYYY-MMMMM.txt.gz" files for each month
 since 2000.  Should just need to concatenate them in the proper order.
+Since these are not being updated, this preparation work can be done
+anytime before the archive import.
-... need the procedure for concatenating and validating them ...
+<code>
-To "/tmp/combined_months/${listname}.mbox"
+months='January
+February
+March
+April
+May
+June
+July
+August
+September
+October
+November
+December'
+years='2000 2001 2002 2003 2004 2005 2006 2007'
+archive_path=/var/lib/mailman/archives/private
+export listname=____
+for year in $years
+  do
+    for month in $months
+      do
+        file="${archive_path}/${listname}/$year-$month.txt"
+        #echo "$file"
+        if [ -f "$file" ]
+          then
+            cat "$file"
+          fi
+      done
+  done > /tmp/${listname}-rebuilt.txt
+Edit /tmp/${listname}-rebuilt.txt and delete everything starting with the
+first message in the mbox file until the end.
+Except the original monthly text file for the initial import from Majordomo
+has sections that go back in time, and some are repeated.
+This makes it more complicated to determine what to delete or keep.
+Also have to change the "From " lines that were massaged to obscure the sender.
+Before that, search for any that are incorrect "From " lines within a message.
+These might have been archived that way.  In that case, leave them alone.
+<code>
+G/^\CFrom .*@/ Any within a message to fix
+:%g/^\CFrom .* at .*200\d$/s/ at /_at_/ " Make normal mail standard line
+</code>
+Run the two checking tools, the same as below.
+The "From " will probably change for a small number of messages.
+<code>
+/tmp/cleanarch3 \
+    /tmp/${listname}-rebuilt.txt \
+    > /tmp/${listname}-rebuilt.mbox
+diff \
+    /tmp/${listname}-rebuilt.txt \
+    /tmp/${listname}-rebuilt.mbox
+/tmp/check_hk_import /tmp/${listname}-rebuilt.mbox
+</code>
+===== Standard steps for all lists =====
+Stop postfix, to prevent any incoming mail going to the mailman 2 queue
+while in the process of transitioning.  Not needed for low volume lists
+where no postings can be expected during this period.
+Since the import and indexing of a large archive can take three hours,
+use a technique to continue running if an interactive session is lost.
+Don't attempt during a storm.
 Stop mailman 2 ...
 Must create an empty list before import.
 For each list:
 <code>
-config_path=/var/lib/mailman/lists
+export config_path=/var/lib/mailman/lists
-archive_path=/var/lib/mailman/archives/private
+export archive_path=/var/lib/mailman/archives/private
-((Or archive_path=/tmp/combined_months))
+export listname=____
-listname=____
+export listfull=${listname}@sluug.org
-listfull=${listname}@sluug.org
+/tmp/cleanarch3 \
+    ${archive_path}/${listname}.mbox/${listname}.mbox \
+    > /tmp/${listname}-cleaned.mbox
+diff \
+    ${archive_path}/${listname}.mbox/${listname}.mbox \
+    /tmp/${listname}-cleaned.mbox
+/tmp/check_hk_import  /tmp/${listname}-cleaned.mbox
+Check the number of archive messages is reasonable.
+The MM2 list archives imported from Majordomo have many duplicate HTML files.
+ls ${archive_path}/${listname}/[12]???-* | grep -c 0......html
 mailman-wrapper create   ${listfull}
 ((The only output is that the list was created))
 mailman-wrapper import21 ${listfull} ${config_path}/${listname}/config.pck
-((There is no output))
+((If there are no problems, there might be no output))
-date ; mailman-web hyperkitty_import --no-sync-mailman --verbosity 2 --no-color --list-address ${listfull} ${archive_path}/${listname}.mbox/${listname}.mbox ; date
+((The only output might be errors and "Importing members" progress))
-((There is a summary of actions))
-date ; mailman-web update_index_one_list --verbosity 2 --no-color ${listfull} ; date
+chmod a+r /tmp/${listname}-*.mbox # Since run as "www-data" user
-((There is a summary of the number indexed))
+((Include "rebuilt.mbox" only if it was created.))
+date ; time mailman-web hyperkitty_import --no-sync-mailman \
+    --verbosity 3 --no-color \
+    --list-address ${listfull} \
+    /tmp/${listname}-rebuilt.mbox \
+    /tmp/${listname}-cleaned.mbox \
+    >> /tmp/import-${listname}-log.txt 2>&1 ; date
+((It lists each message as it is processed.))
+date ; time mailman-web update_index_one_list --verbosity 2 --no-color \
+    ${listfull} \
+    >> /tmp/index-${listname}-log.txt 2>&1 ; date
+((There is no output during the first phase, a quick summary of
+the number indexed, then nothing during last phase))
 </code>
+^  Mailing list and list archive settings to review or set  ^^
+^ Setting                   ^ Action          ^
+^ List settings  ^^
+| Show list on index page   | Probably OK               |
+| Description               | Probably OK               |
+| Information               | Update for new situation  |
+| Display name              | Probably OK               |
+| Subject prefix            | Probably OK               |
+| Other "Settings"          | Probably OK               |
+^ Other list setting pages  ^^
+| Header Filters            | Verify they were imported |
+| Alter Messages            | Verify they were imported |
+| DMARC Mitigations         | Verify they were imported |
+| Digest                    | Verify they were imported |
+| Message Acceptance        | Verify they were imported |
+| Archiving                 | Probably need to set this |
+| Member Policy             | Verify they were imported |
+| Bounce Processing         | Verify they were imported |
+^ Archive settings  ^^
+| Display name              | Probably OK               |
+| Description               | Probably blank, must set  |
+| Subject prefix            | Probably OK               |
+| Archive policy            | Defaults to "public", must set this to "private" for most lists |
+| Owners                    | Review, need at least one selected  |
+| Moderators                | Review, need at least one selected  |
+Test by posting and web access?
+Delete the list from the Mailman 2 configuration by moving the config out of
+the production directory.
+<code>
+mv -iv ${config_path}/${listname} ${config_path}-disabled/
+</code>
+Delete Mailman 2 list aliases and create Mailman 3 aliases?  No.
+Mailman 2 uses "/var/lib/mailman/data/aliases" for "alias_maps".
+Mailman 3 uses "/var/lib/mailman3/data/postfix_lmtp" for "local_recipient_maps".
+No action is needed.
+The new list aliases are created when the lists are imported.
+The "local_recipient_maps" takes precedence over "alias_maps".
 Start mailman 2 ...
-Delete the list from the Mailman 2 configuration by moving the .... to ....
+Start postfix if it was stopped.
-Delete Mailman 2 aliases, create Mailman 3 aliases.
+There is a postfix configuration conflict that results in warning
+"do not list domain ____ in BOTH virtual_mailbox_domains and relay_domains".
+This is a well known problem.
+See https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.html#unusual-postfix-configuration
 Check all configuration web pages on the old and new systems,
 to make sure no options, settings, or values were lost.
+Change the list "info" field for each list.
+Check the list archive HyperKitty Description and Archive policy.
+Might need to set Owners and Moderators.
 Test ...
 ====== Other changes to make later ======
-Index mailing list archives.
 Delete or deflate HTML postings and delete attachments on all lists.
 The "Collapse alternatives" and "Convert html to plaintext" options.
+Change the "Action to take on messages which have no content" setting
+from "Discard" to "Reject".
+Create "/etc/cron.d/mailman3-local" that runs some check that all
+expected processes are running.
+  * 15 "^list .*/usr/lib/mailman3/bin/" for "mailman3",
+  * 2 "www-data .*uwsgi" for "mailman3.web",
+  * 6-7 "manage.py qcluster" for "mailman3.web",
+  * Perhaps also check the output of the "mailman-wrapper status" command for "running".
+Is there a way to disable "Download" of HyperKitty archive as a ".mbox" file?
 ====== Cleanup for deletion of Mailman 2 ======
@@ Line 429: / Line 630: @@
 ===== Command line =====
+Since the services are started by systemd, it is probably better to
+do start/stop/restart using the systemctl command.  Using the direct
+commands might not update systemd on the service status.
 Never use the "mailman" command,
@@ Line 434: / Line 639: @@
 Useful "mailman-wrapper" commands:
-  * "mailman-wrapper --help" - display the available commands
+  * "mailman-wrapper --help" - display the available commands.
-  * "mailman-wrapper COMMAND --help" - display help for one command
+  * "mailman-wrapper COMMAND --help" - display help for one command.
-  * "mailman-wrapper lists" - list the available lists
+  * "mailman-wrapper lists" - list the available lists.
-  * "mailman-wrapper members FULL_LIST_NAME_WITH_HOSTNAME | wc -l" - count of members to one list
+  * "mailman-wrapper members FULL_LIST_NAME_WITH_HOSTNAME | wc -l" - count of members to one list.  In a future release, this might be handled by the "--count-only" option.
-  * "mailman-wrapper findmember 'REGEXP'" - Display all memberships for matching users
+  * "mailman-wrapper findmember 'REGEXP'" - Display all memberships for matching users.
-  * "mailman-wrapper restart" - Stop and restart the Mailman runner subprocesses
+  * "mailman-wrapper restart" - Stop and restart the Mailman runner subprocesses.  See above about using the systemctl command.
 "mailman-web" - Django's command-line utility for administrative tasks.
@@ Line 451: / Line 656: @@
 ==== URLs for general users ====
-  * https://HOSTNAME/mailman3     / - General list access and user login.  What should be announced to humans.
+  * https://WEBSERVER/mailman3     / - General list access and user login.  What should be announced to humans.
-  * https://HOSTNAME/mailman3/postorius/lists/ - What just specifying "mailman3" redirects to.  What should be used in links from the main web site.
+  * https://WEBSERVER/mailman3/postorius/lists/ - What just specifying "mailman3" redirects to.  What should be used in links from the main web site.
+  * https://docs.mailman3.org/en/latest/userguide.html - List Member Manual
 ==== Alternate URLs to ignore ====
-  * https://HOSTNAME/mailman3/hyperkitty/ - long list of "Available lists", with "Sign In" for regular list members.
+  * https://WEBSERVER/mailman3/hyperkitty/ - long list of "Available lists", with "Sign In" for regular list members.
-  * https://HOSTNAME/mailman3/postorius/ - short list of "Mailing Lists".
+  * https://WEBSERVER/mailman3/postorius/ - short list of "Mailing Lists".
-  * https://HOSTNAME/mailman3/postorius/lists/ - the same as just "postorius/".
+  * https://WEBSERVER/mailman3/postorius/lists/ - the same as just "postorius/".
 ==== URLs for Administration ====
-  * https://HOSTNAME/mailman3/admin/ - Administration login, only "for a staff account".
+  * https://WEBSERVER/mailman3/admin/ - Administration login, only "for a staff account".
-  * https://HOSTNAME/mailman3/LISTNAME.HOSTNAME/ - Access private lists not published on the index of lists.  You will still have to log in or directly subscribe to do anything more than see the list description.
+  * https://WEBSERVER/mailman3/LISTNAME.HOSTNAME/ - Access private lists not published on the index of lists.  You will still have to log in or directly subscribe to do anything more than see the list description.  This uses a "." instead of "@".
-  * https://HOSTNAME/mailman3/hyperkitty/list/LISTNAME@HOSTNAME/ - Access private archives.  You will still have to log in to see the private archive.
+  * https://WEBSERVER/mailman3/hyperkitty/list/LISTNAME@HOSTNAME/ - Access private archives.  You will still have to log in to see the private archive.  This uses a "@".
 You can use the general URL for lists, then login and you will see private
@@ Line 486: / Line 692: @@
 Use the admin URL, then log in, then either select
-  * "Mailing lists" under the "HyperKitty", then a list to change description, etc.
+  * For the mailing list, select "View site" in the heading,
-  * "View site" in the heading,
     * "Mailman settings" to change subscription settings for this address.
     * A list to change "List Settings", handle pending actions, etc.
+  * For list archives, select "Mailing lists" under the "HyperKitty", then a list to change description, etc.
+    * There is no command line interface for most HyperKitty actions.
+    * To remove the archive for a list, you must use the admin web interface and the "HyperKitty > Mailing lists" (admin/hyperkitty/mailinglist/) web page.  This only affects the list archive, not the base list.
-Watch for the logout confirmation web page.
+When logging out of the web interface,
+watch for the logout confirmation web page.
 ===== Processes =====
 There are separate services and processes for the mailing list and web.
-  * The "mailman3-web" service controls the "uwsgi" processes.
+  * The "mailman3-web" service controls the "uwsgi" processes. and includes "python3 manage.py" processes.
-  * The "mailman3" service is Mailman 3, for processing mail.
+  * The "mailman3" service is Mailman 3, for processing mail.  For the "/usr/lib/mailman3/bin/" processes.
   * The "mailman" service is Mailman 2, not 3.

SLUUG Wiki

User Tools

Site Tools

Differences

Page Tools