CommunityMigration

Migration From Etch

Deliverables

• c.h.o migrated to new server, all services functioning as before for all users
• Regularly scheduled automated off-site backups of new server, so that we can redeploy if we suddenly lose the server (hardware failure, VPS provider belly up, etc.)
• A set of migration scripts that can be re-used in case we need to migrate again in the future
• A set of policies/procedures that will keep the migration scripts up-to-date

Assets to be moved

Services to migrate

The following are in separately doable chunks:

• Increase lun's memory allocation and move the disk image out of ~igloo

• nagios

• Admin home directories

• Configure exim, mailman, apache, clamav

• planet.haskell.org
• planet (might need to migrate one or two users early for this)

• projects.haskell.org MX (hmm, this'll divorce the lists from the web interface)
• mailman (need to disable list creation script on nun first)

• projects.haskell.org
• code.haskell.org
• community.haskell.org
• trac.haskell.org
• User accounts and home directories
• RT on PostgreSQL
• Trac
• http
• Exim
• c.h.o admin scripts for users
  • account_request
  • project_request
  • createlist
  • createtrac
  • addtoproject
  • crontabfor
• c.h.o admin scripts for admins
  • create_user.sh
  • create_project.sh

User data to be transfered

• User accounts and home directories
• Project groups in /etc/group
• Darcs repos from /srv/projects
• Project data from /srv/code
• Trac projects
• Mailman lists
• Planet Haskell
• Physical mailboxes (igloo and malcolm)

Domains to be transferred

• community
• code
• rt
• planet
• trac
• projects

Migration plan

Coordination will happen on the #haskell-infrastructure channel on FreeNode.

Preparatory stage

1. Formulate plan on how to communicate with users during the migration process; begin gathering required contact info if necessary
2. Calculate the approximate size and rate of change of each type of data on the server
3. Give preliminary notice to users and ask for feedback
4. Configure domain name etch.haskell.org (will later become read-only copy and left active for a while as a backup)
5. Change TTL on all domains to be short
6. Provision the new server
7. Install and perform basic configuration of all services
8. Test all installed services
9. Set up backup mirror(s)
10. Set up backup mirror services/scripts. Install on new server and test.
11. Write scripts to copy user accounts and rsync home directories, and to verify
12. Write scripts to copy project groups in /etc/group, and to verify
13. Write scripts to rsync darcs repos from /srv/projects, and to verify
14. Write scripts to rsync project data from /srv/code, and to verify
15. Write scripts to rsync trac projects, and to verify
16. Write scripts to copy mailman lists and rsync archives, and to verify
17. Write scripts to rsync Planet Haskell, and to verify
18. Write a script to make all user data and projects read-only (except mailman archives), and to verify.
19. Write a script to *undo* making things read-only, in case of emergency.
20. Test all scripts thoroughly
21. Fix dates for beginning initial copy and final migration, and give advance notice and instructions to users.
22. Transfer RT database and verify (as a dry run, go through the entire migration process just for RT)
23. Move the rt domain
24. After TTL, verify that RT is working on the new server

Initial copy

1. Run script to copy user accounts. Verify.
2. Run script to copy /etc/group entries for projects. Verify.
3. Run scripts to rsync home directories, darcs repos, project data, trac projects, mailman archives, and Planet data
4. Monitor progress; update schedule and notify users as needed
5. When completed, verify.

D-Day

1. Notify users.
2. Stop the Planet Haskell hourly cron job.
3. Run script to make all user data and projects read-only. Verify.
4. If any user accounts or projects were added since initial copy began, add them.
5. Run scripts to rsync home directories, darcs repos, project data, and Trac projects. Verify.
6. Stop mailman service on etch.
7. Run the scripts to rsync mailman archives. Verify.
8. Move the domain CNAME for community, trac, projects, and code.
9. After TTL, verify remotely that the domains moved and that services are working.
10. Notify users of current status.
11. Tell Ian and Malcolm to check their mail.
12. Move the domain MX records.
13. Run the scripts to rsync Planet Haskell. Verify.
14. Move the planet domain.
15. Start the Planet Haskell hourly cron job.
16. After TTL, verify remotely that all domains are moved and that all services are working.
17. Notify users and community.

Post-migration tasks

1. Activate backup mirroring. Verify.
2. Monitor net bandwidth and responsiveness of the new server.
3. Monitor memory and cpu usage on the new server.
4. Monitor and tune settings of PostgreSQL for resource usage on the new server
5. Monitor and tune settings of Apache for resource usage on the new server
6. After a day or two, raise TTL back to normal levels on all domains
7. After a month or two, delete the Etch server