Differences

This shows you the differences between two versions of the page.

--- replace_mailman_2 [2025/09/25 01:25]
SLUUG Administration Progress update
+++ replace_mailman_2 [2025/11/18 02:13] (current)
SLUUG Administration Progress update
@@ Line 64: / Line 64: @@
 Password management:  Mailman3 and Postgresql (don't use sqlite for mailman3!) will require the installer to specify some passwords during the setup process.  Be prepared, and have a plan in advance for where and how to store and communicate those passwords.
+----
 ====== Planning ======
@@ Line 333: / Line 335: @@
 Will "mailman_queue_check.sh" work?
-No, mailman 3 has a completely different structure.
+Mailman 3 might be compatible with minor changes.
+Mailman 2 uses "/var/lib/mailman/qfiles/".
+Mailman 3 uses "/var/lib/mailman3/queue/".
 __**Everything below has not been tested yet and is for planning**__
@@ Line 342: / Line 346: @@
 Configure to delete attachments from the archive during import?
+Initial testing shows about half the attachments from the MM2 archives are
+imported to MM3, others aren't.  The difference hasn't been determined.
 Test list deletion:
@@ Line 366: / Line 372: @@
 Even if changed after import, planning what the text should be can speed the process.
-TBD:
+Messages posted with certain non-ASCII characters may fail to be imported
+to the MM3 archives.  Most non-ASCII characters cause no problems.
+TBD settings:
   * "header_checks" are now global?
   * "[dmarc]" and/or "[ARC]".
+  * "charset".  Leaving as "us-ascii".
+==== Mailman 3 fails if SQL not ready ====
+After a reboot or restarting multiple processes, Mailman will often try
+to connect to the SQL database server before that server has completed
+startup to the point it can accept connections and perform operations.
+This makes the Mailman service fail and terminate.
+This is a well known problem.
+There is no systemd configuration present to state Mailman depends on the
+SQL service.  This has been corrected in later Debian Mailman releases.
+Correction is by a systemd override to establish the dependency.
+Detailed actions are to be documented here. ----
 ====== Import Mailman 2 lists ======
+To check the archives for potential errors before attempting import:
+  * Use cleanarch or cleanarch3 to check for bad From and Date headers.
+  * Use check_hk_archive (replaced by check_hk_import) to check for messages that would fail import.
+While cleanarch is a standard part of MM2, cleanarch3 is not in the
+currently installed Debian python3-django-hyperkitty package.
+It is in the more recent 1.3.12-1 (1.3.8 and later) release.
+The same for check_hk_archive (replaced by check_hk_import).
+Download the latest package, extact the two scripts to a temporary location.
+===== Extra steps for 3 lists =====
 For 3 lists, that existed before the 2007 move from Majordomo, the
@@ Line 380: / Line 416: @@
 are the "YYYY-MMMMM.txt" and "YYYY-MMMMM.txt.gz" files for each month
 since 2000.  Should just need to concatenate them in the proper order.
+Since these are not being updated, this preparation work can be done
+anytime before the archive import.
-... need the procedure for concatenating and validating them ...
+<code>
-To "/tmp/combined_months/${listname}.mbox"
+months='January
+February
+March
+April
+May
+June
+July
+August
+September
+October
+November
+December'
+years='2000 2001 2002 2003 2004 2005 2006 2007'
+archive_path=/var/lib/mailman/archives/private
+export listname=____
+for year in $years
+  do
+    for month in $months
+      do
+        file="${archive_path}/${listname}/$year-$month.txt"
+        #echo "$file"
+        if [ -f "$file" ]
+          then
+            cat "$file"
+          fi
+      done
+  done > /tmp/${listname}-rebuilt.txt
+Edit /tmp/${listname}-rebuilt.txt and delete everything starting with the
+first message in the mbox file until the end.
+Except the original monthly text file for the initial import from Majordomo
+has sections that go back in time, and some are repeated.
+This makes it more complicated to determine what to delete or keep.
+Also have to change the "From " lines that were massaged to obscure the sender.
+Before that, search for any that are incorrect "From " lines within a message.
+These might have been archived that way.  In that case, leave them alone.
+<code>
+G/^\CFrom .*@/ Any within a message to fix
+:%g/^\CFrom .* at .*200\d$/s/ at /_at_/ " Make normal mail standard line
+</code>
+Run the two checking tools, the same as below.
+The "From " will probably change for a small number of messages.
+<code>
+/tmp/cleanarch3 \
+    /tmp/${listname}-rebuilt.txt \
+    > /tmp/${listname}-rebuilt.mbox
+diff \
+    /tmp/${listname}-rebuilt.txt \
+    /tmp/${listname}-rebuilt.mbox
+/tmp/check_hk_import /tmp/${listname}-rebuilt.mbox
+</code>
+===== Standard steps for all lists =====
+Stop postfix, to prevent any incoming mail going to the mailman 2 queue
+while in the process of transitioning.  Not needed for low volume lists
+where no postings can be expected during this period.
+Since the import and indexing of a large archive can take three hours,
+use a technique to continue running if an interactive session is lost.
+Don't attempt during a storm.
 Stop mailman 2 ...
 Must create an empty list before import.
 For each list:
 <code>
-config_path=/var/lib/mailman/lists
+export config_path=/var/lib/mailman/lists
-archive_path=/var/lib/mailman/archives/private
+export archive_path=/var/lib/mailman/archives/private
-((Or archive_path=/tmp/combined_months))
+export listname=____
-listname=____
+export listfull=${listname}@sluug.org
-listfull=${listname}@sluug.org
+/tmp/cleanarch3 \
+    ${archive_path}/${listname}.mbox/${listname}.mbox \
+    > /tmp/${listname}-cleaned.mbox
+diff \
+    ${archive_path}/${listname}.mbox/${listname}.mbox \
+    /tmp/${listname}-cleaned.mbox
+/tmp/check_hk_import  /tmp/${listname}-cleaned.mbox
+Check the number of archive messages is reasonable.
+The MM2 list archives imported from Majordomo have many duplicate HTML files.
+ls ${archive_path}/${listname}/[12]???-* | grep -c 0......html
 mailman-wrapper create   ${listfull}
 ((The only output is that the list was created))
 mailman-wrapper import21 ${listfull} ${config_path}/${listname}/config.pck
-((If there are no problems, there is no output))
+((If there are no problems, there might be no output))
 ((The only output might be errors and "Importing members" progress))
-date ; time mailman-web hyperkitty_import --no-sync-mailman --verbosity 1 --no-color --list-address ${listfull} ${archive_path}/${listname}.mbox/${listname}.mbox | tee -a /tmp/import-${listname}.txt ; date
-((There is a summary of actions))
+chmod a+r /tmp/${listname}-*.mbox # Since run as "www-data" user
-date ; time mailman-web update_index_one_list --verbosity 2 --no-color ${listfull} | tee -a /tmp/index-${listname}.txt ; date
-((There is a summary of the number indexed))
+((Include "rebuilt.mbox" only if it was created.))
+date ; time mailman-web hyperkitty_import --no-sync-mailman \
+    --verbosity 3 --no-color \
+    --list-address ${listfull} \
+    /tmp/${listname}-rebuilt.mbox \
+    /tmp/${listname}-cleaned.mbox \
+    >> /tmp/import-${listname}-log.txt 2>&1 ; date
+((It lists each message as it is processed.))
+date ; time mailman-web update_index_one_list --verbosity 2 --no-color \
+    ${listfull} \
+    >> /tmp/index-${listname}-log.txt 2>&1 ; date
+((There is no output during the first phase, a quick summary of
+the number indexed, then nothing during last phase))
 </code>
 ^  Mailing list and list archive settings to review or set  ^^
-^ Setting                                                   ^ Action          ^
+^ Setting                   ^ Action          ^
 ^ List settings  ^^
-| Show list on index page                                   | Probably OK               |
+| Show list on index page   | Probably OK               |
-| Description                                               | Probably OK               |
+| Description               | Probably OK               |
-| Information                                               | Update for new situation  |
+| Information               | Update for new situation  |
-| Display name                                              | Probably OK               |
+| Display name              | Probably OK               |
-| Subject prefix                                            | Probably OK               |
+| Subject prefix            | Probably OK               |
-| Other "Settings"                                          | Probably OK               |
+| Other "Settings"          | Probably OK               |
 ^ Other list setting pages  ^^
-| Header Filters                                            | Verify they were imported |
+| Header Filters            | Verify they were imported |
-| Alter Messages                                            | Verify they were imported |
+| Alter Messages            | Verify they were imported |
-| DMARC Mitigations                                         | Verify they were imported |
+| DMARC Mitigations         | Verify they were imported |
-| Digest                                                    | Verify they were imported |
+| Digest                    | Verify they were imported |
-| Message Acceptance                                        | Verify they were imported |
+| Message Acceptance        | Verify they were imported |
-| Archiving                                                 | Probably need to set this |
+| Archiving                 | Probably need to set this |
-| Member Policy                                             | Verify they were imported |
+| Member Policy             | Verify they were imported |
-| Bounce Processing                                         | Verify they were imported |
+| Bounce Processing         | Verify they were imported |
 ^ Archive settings  ^^
-| Display name                                              | Probably OK     |
+| Display name              | Probably OK               |
-| Description                                               | Probably blank, must set               |
+| Description               | Probably blank, must set  |
-| Subject prefix                                            | Probably OK               |
+| Subject prefix            | Probably OK               |
-| Archive policy                                            | Defaults to "public", must set this to "private" for most lists |
+| Archive policy            | Defaults to "public", must set this to "private" for most lists |
-| Owners                                                    | Review, need at least one selected          |
+| Owners                    | Review, need at least one selected  |
-| Moderators                                                | Review, need at least one selected          |
+| Moderators                | Review, need at least one selected  |
 Test by posting and web access?
-Delete the list from the Mailman 2 configuration by moving the .... to ....
+Delete the list from the Mailman 2 configuration by moving the config out of
+the production directory.
+<code>
 mv -iv ${config_path}/${listname} ${config_path}-disabled/
-config_path
+</code>
 Delete Mailman 2 list aliases and create Mailman 3 aliases?  No.
@@ Line 444: / Line 578: @@
 Start mailman 2 ...
+Start postfix if it was stopped.
 There is a postfix configuration conflict that results in warning
@@ Line 464: / Line 600: @@
 Delete or deflate HTML postings and delete attachments on all lists.
 The "Collapse alternatives" and "Convert html to plaintext" options.
+Change the "Action to take on messages which have no content" setting
+from "Discard" to "Reject".
 Create "/etc/cron.d/mailman3-local" that runs some check that all
@@ Line 530: / Line 669: @@
   * https://WEBSERVER/mailman3/admin/ - Administration login, only "for a staff account".
-  * https://WEBSERVER/mailman3/LISTNAME.HOSTNAME/ - Access private lists not published on the index of lists.  You will still have to log in or directly subscribe to do anything more than see the list description.
+  * https://WEBSERVER/mailman3/LISTNAME.HOSTNAME/ - Access private lists not published on the index of lists.  You will still have to log in or directly subscribe to do anything more than see the list description.  This uses a "." instead of "@".
-  * https://WEBSERVER/mailman3/hyperkitty/list/LISTNAME@HOSTNAME/ - Access private archives.  You will still have to log in to see the private archive.
+  * https://WEBSERVER/mailman3/hyperkitty/list/LISTNAME@HOSTNAME/ - Access private archives.  You will still have to log in to see the private archive.  This uses a "@".
 You can use the general URL for lists, then login and you will see private
@@ Line 558: / Line 697: @@
   * For list archives, select "Mailing lists" under the "HyperKitty", then a list to change description, etc.
     * There is no command line interface for most HyperKitty actions.
-    * To remove the archive for a list, you must use the admin web interface and the "HyperKitty > Mailing lists" (admin/hyperkitty/mailinglist/) web page.
+    * To remove the archive for a list, you must use the admin web interface and the "HyperKitty > Mailing lists" (admin/hyperkitty/mailinglist/) web page.  This only affects the list archive, not the base list.
-Watch for the logout confirmation web page.
+When logging out of the web interface,
+watch for the logout confirmation web page.
 ===== Processes =====

SLUUG Wiki

User Tools

Site Tools

Differences

Page Tools