John Hallett
 ·   ·  66 posts
  • 7 followers
  • 4 friends

Migration Weekend: Moving This Network to a New Server

What went to plan, what didn't, and what I'd do differently when moving a self-hosted community to new hardware.

Back in February I wrote that we were close to outgrowing our single server. Last weekend we finally moved. If the site felt slow or went read-only for a few hours on Saturday night, that was me. Thank you for your patience.

The plan

  • Build the new server and copy everything across a week early.
  • Run both side by side and have a few volunteers test the copy.
  • On the night: switch the site to read-only, take a final database dump, import it and point the domain at the new server.

What went well

The test run paid off. Our volunteer testers found two broken image paths and a scheduled job that hadn't been copied. Fixing those on Wednesday instead of at midnight on Saturday saved me a lot of stress.

What didn't

I underestimated the final database import. The timeline and search tables took far longer to rebuild than in the test run, so read-only mode lasted about two and a half hours instead of the forty minutes I'd promised.

DNS changes also don't reach everyone at the same speed. A few members kept seeing the old server, still read-only, until Sunday morning. If you couldn't post on Sunday, that's why, and I'm sorry.

The result

Pages now load roughly twice as fast at busy times, image uploads no longer stall, and we finally have a staging copy, so I can stop testing things in production. If you spot anything broken, please post it in Help & Support.

😆 1
  • View
    366
  • Up
    1
    Down

Info

Category:
Architecture
Created:
Updated:

html

asdasdasd

Comments (4)

    • A few follow-up notes for anyone planning a similar move.

      Lower your DNS TTL a few days in advance. I did it the night before, which was too late to help much.

      • Keep the old server running for at least a week. I needed it twice to copy a config file I'd forgotten.

        • Write the switch-over steps down and tick them off as you go. At 1am your memory is not your friend.

          • And tell members in advance, in more than one place. The pinned post got far more attention than the email did.

            Login or Join to comment.