We had about 20 minutes of downtime because someone's bot crashed mw1 with too many requests -- not one of ours -- and we weren't keeping track of when PHP had crashed. So now we're doing that. We also updated to MediaWiki 1.26. There are still issues with Flow after the upgrade, but since we're still using Liquid Threads, nothing affecting ATT.
But the good news is that we've now tripled the memory on the database server, and it seems much more stable. This should mean less of the database connection errors for you all. At 1 GiB RAM: run out of memory every 2-3 hours and restart mariadb. At 2 GiB: out of memory every 30-40 hours. At 3 GiB: only use about 2.6 GiB and no restarts yet.
We've selected a service with a good deal of storage for a backup server, rather than go with something like Amazon S3 or Backblaze B2, a real server will let us do more stuff with it. And then we just need to set up the service to do regular backups. I think we might end up selecting the New Jersey site for this, which I guess means that if something goes wrong, Bob's on call
-- ∇×V
But the good news is that we've now tripled the memory on the database server, and it seems much more stable. This should mean less of the database connection errors for you all. At 1 GiB RAM: run out of memory every 2-3 hours and restart mariadb. At 2 GiB: out of memory every 30-40 hours. At 3 GiB: only use about 2.6 GiB and no restarts yet.
We've selected a service with a good deal of storage for a backup server, rather than go with something like Amazon S3 or Backblaze B2, a real server will let us do more stuff with it. And then we just need to set up the service to do regular backups. I think we might end up selecting the New Jersey site for this, which I guess means that if something goes wrong, Bob's on call
-- ∇×V