Yeah down again, and back up again. If anyone knows a DBA, that might be helpful. Previously mysqld went down from the OOM killer, but that didn't happen this time. I'm sure more details will come out later today. (Also if anyone knows PHP and would be willing to fix T154076 I could actually notify everyone about what's going on during downtime).
Without knowing the causes here I can't say what the fix is. But the main problem is that we're getting processes killed, so the DB performs a whole warmup/checksum when starting up. Because we have so many wikis, this takes ~100 minutes. So a small incident results in a lot of downtime. Theoretically we could speed this up by deleting empty wikis or dormant wikis, which might be the easiest at this point. And then solutions start moving towards sharding or replication or increasing memory -- I can't say for sure until we get a post-mortem. I do have money budgeted for increased resources if necessary.
-- ∇×V
Without knowing the causes here I can't say what the fix is. But the main problem is that we're getting processes killed, so the DB performs a whole warmup/checksum when starting up. Because we have so many wikis, this takes ~100 minutes. So a small incident results in a lot of downtime. Theoretically we could speed this up by deleting empty wikis or dormant wikis, which might be the easiest at this point. And then solutions start moving towards sharding or replication or increasing memory -- I can't say for sure until we get a post-mortem. I do have money budgeted for increased resources if necessary.
-- ∇×V