Personal tools

CommunityMigration

From HaskellWiki

(Difference between revisions)
Jump to: navigation, search
 
(Converted wiki formatting from Trac to MediaWiki.)
 
(4 intermediate revisions by one user not shown)
Line 2: Line 2:
   
 
=== Deliverables ===
 
=== Deliverables ===
* c.h.o migrated to new server, all services functioning as before for all users
+
* c.h.o migrated to new server, all services functioning as before for all users
* Regularly scheduled automated off-site backups of new server, so that we can redeploy if we suddenly lose the server (hardware failure, VPS provider belly up, etc.)
+
* Regularly scheduled automated off-site backups of new server, so that we can redeploy if we suddenly lose the server (hardware failure, VPS provider belly up, etc.)
* A set of migration scripts that can be re-used in case we need to migrate again in the future
+
* A set of migration scripts that can be re-used in case we need to migrate again in the future
* A set of policies/procedures that will keep the migration scripts up-to-date
+
* A set of policies/procedures that will keep the migration scripts up-to-date
   
 
=== Assets to be moved ===
 
=== Assets to be moved ===
 
==== Services to migrate ====
 
==== Services to migrate ====
* RT on PostgreSQL
+
* Trac
+
The following are in separately doable chunks:
* http
+
* Exim
+
* Increase lun's memory allocation and move the disk image out of ~igloo
* ClamAV
+
* mrtg
+
* nagios
* darcs
+
* ghc
+
* Admin home directories
* mailman
+
* nagios
+
* Configure exim, mailman, apache, clamav
* planet
+
* c.h.o admin scripts for users
+
* planet.haskell.org
* account_request
+
* planet (might need to migrate one or two users early for this)
* project_request
+
* createlist
+
* projects.haskell.org MX (hmm, this'll divorce the lists from the web interface)
* createtrac
+
* mailman (need to disable list creation script on nun first)
* addtoproject
+
* crontabfor
+
* projects.haskell.org
* c.h.o admin scripts for admins
+
* code.haskell.org
* create_user.sh
+
* community.haskell.org
* create_project.sh
+
* trac.haskell.org
  +
* User accounts and home directories
  +
* RT on PostgreSQL
  +
* Trac
  +
* http
  +
* Exim
  +
* c.h.o admin scripts for users
  +
** account_request
  +
** project_request
  +
** createlist
  +
** createtrac
  +
** addtoproject
  +
** crontabfor
  +
* c.h.o admin scripts for admins
  +
** create_user.sh
  +
** create_project.sh
   
 
==== User data to be transfered ====
 
==== User data to be transfered ====
* User accounts and home directories
+
* User accounts and home directories
* Project groups in /etc/group
+
* Project groups in /etc/group
* Darcs repos from /srv/projects
+
* Darcs repos from /srv/projects
* Project data from /srv/code
+
* Project data from /srv/code
* Trac projects
+
* Trac projects
* Mailman lists
+
* Mailman lists
* Planet Haskell
+
* Planet Haskell
* Physical mailboxes (igloo and malcolm)
+
* Physical mailboxes (igloo and malcolm)
   
 
==== Domains to be transferred ====
 
==== Domains to be transferred ====
* community
+
* community
* code
+
* code
* rt
+
* rt
* planet
+
* planet
* trac
+
* trac
* projects
+
* projects
   
 
=== Migration plan ===
 
=== Migration plan ===
  +
  +
Coordination will happen on the #haskell-infrastructure channel on FreeNode.
  +
 
==== Preparatory stage ====
 
==== Preparatory stage ====
1. Formulate plan on how to communicate with users during the migration process; begin gathering required contact info if necessary
+
# Formulate plan on how to communicate with users during the migration process; begin gathering required contact info if necessary
1. Calculate the approximate size and rate of change of each type of data on the server
+
# Calculate the approximate size and rate of change of each type of data on the server
1. Give preliminary notice to users and ask for feedback
+
# Give preliminary notice to users and ask for feedback
1. Configure domain name etch.haskell.org (will later become read-only copy and left active for a while as a backup)
+
# Configure domain name etch.haskell.org (will later become read-only copy and left active for a while as a backup)
1. Set up an IRC channel for admins to coordinate during the migration process
+
# Change TTL on all domains to be short
1. Change TTL on all domains to be short
+
# Provision the new server
1. Provision the new server
+
# Install and perform basic configuration of all services
1. Install and perform basic configuration of all services
+
# Test all installed services
1. Test all installed services
+
# Set up backup mirror(s)
1. Set up backup mirror(s)
+
# Set up backup mirror services/scripts. Install on new server and test.
1. Set up backup mirror services/scripts. Install on new server and test.
+
# Write scripts to copy user accounts and rsync home directories, and to verify
1. Write scripts to copy user accounts and rsync home directories, and to verify
+
# Write scripts to copy project groups in /etc/group, and to verify
1. Write scripts to copy project groups in /etc/group, and to verify
+
# Write scripts to rsync darcs repos from /srv/projects, and to verify
1. Write scripts to rsync darcs repos from /srv/projects, and to verify
+
# Write scripts to rsync project data from /srv/code, and to verify
1. Write scripts to rsync project data from /srv/code, and to verify
+
# Write scripts to rsync trac projects, and to verify
1. Write scripts to rsync trac projects, and to verify
+
# Write scripts to copy mailman lists and rsync archives, and to verify
1. Write scripts to copy mailman lists and rsync archives, and to verify
+
# Write scripts to rsync Planet Haskell, and to verify
1. Write scripts to rsync Planet Haskell, and to verify
+
# Write a script to make all user data and projects read-only (except mailman archives), and to verify.
1. Write a script to make all user data and projects read-only (except mailman archives), and to verify.
+
# Write a script to *undo* making things read-only, in case of emergency.
1. Write a script to *undo* making things read-only, in case of emergency.
+
# Test all scripts thoroughly
1. Test all scripts thoroughly
+
# Fix dates for beginning initial copy and final migration, and give advance notice and instructions to users.
1. Fix dates for beginning initial copy and final migration, and give advance notice and instructions to users.
+
# Transfer RT database and verify (as a dry run, go through the entire migration process just for RT)
1. Transfer RT database and verify (as a dry run, go through the entire migration process just for RT)
+
# Move the rt domain
1. Move the rt domain
+
# After TTL, verify that RT is working on the new server
1. After TTL, verify that RT is working on the new server
 
   
 
==== Initial copy ====
 
==== Initial copy ====
1. Run script to copy user accounts. Verify.
+
# Run script to copy user accounts. Verify.
1. Run script to copy /etc/group entries for projects. Verify.
+
# Run script to copy /etc/group entries for projects. Verify.
1. Run scripts to rsync home directories, darcs repos, project data, trac projects, mailman archives, and Planet data
+
# Run scripts to rsync home directories, darcs repos, project data, trac projects, mailman archives, and Planet data
1. Monitor progress; update schedule and notify users as needed
+
# Monitor progress; update schedule and notify users as needed
1. When completed, verify.
+
# When completed, verify.
   
 
==== D-Day ====
 
==== D-Day ====
1. Notify users.
+
# Notify users.
1. Stop the Planet Haskell hourly cron job.
+
# Stop the Planet Haskell hourly cron job.
1. Run script to make all user data and projects read-only. Verify.
+
# Run script to make all user data and projects read-only. Verify.
1. If any user accounts or projects were added since initial copy began, add them.
+
# If any user accounts or projects were added since initial copy began, add them.
1. Run scripts to rsync home directories, darcs repos, project data, and Trac projects. Verify.
+
# Run scripts to rsync home directories, darcs repos, project data, and Trac projects. Verify.
1. Stop mailman service on etch.
+
# Stop mailman service on etch.
1. Run the scripts to rsync mailman archives. Verify.
+
# Run the scripts to rsync mailman archives. Verify.
1. Move the domain CNAME for community, trac, projects, and code.
+
# Move the domain CNAME for community, trac, projects, and code.
1. After TTL, verify remotely that the domains moved and that services are working.
+
# After TTL, verify remotely that the domains moved and that services are working.
1. Notify users of current status.
+
# Notify users of current status.
1. Tell Ian and Malcolm to check their mail.
+
# Tell Ian and Malcolm to check their mail.
1. Move the domain MX records.
+
# Move the domain MX records.
1. Run the scripts to rsync Planet Haskell. Verify.
+
# Run the scripts to rsync Planet Haskell. Verify.
1. Move the planet domain.
+
# Move the planet domain.
1. Start the Planet Haskell hourly cron job.
+
# Start the Planet Haskell hourly cron job.
1. After TTL, verify remotely that '''all''' domains are moved and that '''all''' services are working.
+
# After TTL, verify remotely that '''all''' domains are moved and that '''all''' services are working.
1. Notify users and community.
+
# Notify users and community.
   
 
==== Post-migration tasks ====
 
==== Post-migration tasks ====
1. Activate backup mirroring. Verify.
+
# Activate backup mirroring. Verify.
1. Monitor net bandwidth and responsiveness of the new server.
+
# Monitor net bandwidth and responsiveness of the new server.
1. Monitor memory and cpu usage on the new server.
+
# Monitor memory and cpu usage on the new server.
1. Monitor and tune settings of PostgreSQL for resource usage on the new server
+
# Monitor and tune settings of PostgreSQL for resource usage on the new server
1. Monitor and tune settings of Apache for resource usage on the new server
+
# Monitor and tune settings of Apache for resource usage on the new server
1. After a day or two, raise TTL back to normal levels on all domains
+
# After a day or two, raise TTL back to normal levels on all domains
1. After a month or two, delete the Etch server
+
# After a month or two, delete the Etch server

Latest revision as of 06:00, 30 January 2011

Contents

[edit] 1 Migration From Etch

[edit] 1.1 Deliverables

  • c.h.o migrated to new server, all services functioning as before for all users
  • Regularly scheduled automated off-site backups of new server, so that we can redeploy if we suddenly lose the server (hardware failure, VPS provider belly up, etc.)
  • A set of migration scripts that can be re-used in case we need to migrate again in the future
  • A set of policies/procedures that will keep the migration scripts up-to-date

[edit] 1.2 Assets to be moved

[edit] 1.2.1 Services to migrate

The following are in separately doable chunks:

  • Increase lun's memory allocation and move the disk image out of ~igloo
  • nagios
  • Admin home directories
  • Configure exim, mailman, apache, clamav
  • planet.haskell.org
  • planet (might need to migrate one or two users early for this)
  • projects.haskell.org MX (hmm, this'll divorce the lists from the web interface)
  • mailman (need to disable list creation script on nun first)
  • projects.haskell.org
  • code.haskell.org
  • community.haskell.org
  • trac.haskell.org
  • User accounts and home directories
  • RT on PostgreSQL
  • Trac
  • http
  • Exim
  • c.h.o admin scripts for users
    • account_request
    • project_request
    • createlist
    • createtrac
    • addtoproject
    • crontabfor
  • c.h.o admin scripts for admins
    • create_user.sh
    • create_project.sh

[edit] 1.2.2 User data to be transfered

  • User accounts and home directories
  • Project groups in /etc/group
  • Darcs repos from /srv/projects
  • Project data from /srv/code
  • Trac projects
  • Mailman lists
  • Planet Haskell
  • Physical mailboxes (igloo and malcolm)

[edit] 1.2.3 Domains to be transferred

  • community
  • code
  • rt
  • planet
  • trac
  • projects

[edit] 1.3 Migration plan

Coordination will happen on the #haskell-infrastructure channel on FreeNode.

[edit] 1.3.1 Preparatory stage

  1. Formulate plan on how to communicate with users during the migration process; begin gathering required contact info if necessary
  2. Calculate the approximate size and rate of change of each type of data on the server
  3. Give preliminary notice to users and ask for feedback
  4. Configure domain name etch.haskell.org (will later become read-only copy and left active for a while as a backup)
  5. Change TTL on all domains to be short
  6. Provision the new server
  7. Install and perform basic configuration of all services
  8. Test all installed services
  9. Set up backup mirror(s)
  10. Set up backup mirror services/scripts. Install on new server and test.
  11. Write scripts to copy user accounts and rsync home directories, and to verify
  12. Write scripts to copy project groups in /etc/group, and to verify
  13. Write scripts to rsync darcs repos from /srv/projects, and to verify
  14. Write scripts to rsync project data from /srv/code, and to verify
  15. Write scripts to rsync trac projects, and to verify
  16. Write scripts to copy mailman lists and rsync archives, and to verify
  17. Write scripts to rsync Planet Haskell, and to verify
  18. Write a script to make all user data and projects read-only (except mailman archives), and to verify.
  19. Write a script to *undo* making things read-only, in case of emergency.
  20. Test all scripts thoroughly
  21. Fix dates for beginning initial copy and final migration, and give advance notice and instructions to users.
  22. Transfer RT database and verify (as a dry run, go through the entire migration process just for RT)
  23. Move the rt domain
  24. After TTL, verify that RT is working on the new server

[edit] 1.3.2 Initial copy

  1. Run script to copy user accounts. Verify.
  2. Run script to copy /etc/group entries for projects. Verify.
  3. Run scripts to rsync home directories, darcs repos, project data, trac projects, mailman archives, and Planet data
  4. Monitor progress; update schedule and notify users as needed
  5. When completed, verify.

[edit] 1.3.3 D-Day

  1. Notify users.
  2. Stop the Planet Haskell hourly cron job.
  3. Run script to make all user data and projects read-only. Verify.
  4. If any user accounts or projects were added since initial copy began, add them.
  5. Run scripts to rsync home directories, darcs repos, project data, and Trac projects. Verify.
  6. Stop mailman service on etch.
  7. Run the scripts to rsync mailman archives. Verify.
  8. Move the domain CNAME for community, trac, projects, and code.
  9. After TTL, verify remotely that the domains moved and that services are working.
  10. Notify users of current status.
  11. Tell Ian and Malcolm to check their mail.
  12. Move the domain MX records.
  13. Run the scripts to rsync Planet Haskell. Verify.
  14. Move the planet domain.
  15. Start the Planet Haskell hourly cron job.
  16. After TTL, verify remotely that all domains are moved and that all services are working.
  17. Notify users and community.

[edit] 1.3.4 Post-migration tasks

  1. Activate backup mirroring. Verify.
  2. Monitor net bandwidth and responsiveness of the new server.
  3. Monitor memory and cpu usage on the new server.
  4. Monitor and tune settings of PostgreSQL for resource usage on the new server
  5. Monitor and tune settings of Apache for resource usage on the new server
  6. After a day or two, raise TTL back to normal levels on all domains
  7. After a month or two, delete the Etch server