Infrastructure nerd, gamer, and Lemmy.ca maintainer
No we dropped the ball on this one, we should have posted right away.
Lemmy.ml has been defederated for a long time.
Some instances have their own status pages, we have https://status.lemmy.ca/
The outage several instances faced this morning was from https://github.com/LemmyNet/lemmy/issues/3165
The original issue is just a symptom of all database threads being tied up. People just don’t know how to follow an error message to the root cause.
The real source of the issue is db locking from triggers and cascading deletes on a major user change.
My report in https://github.com/LemmyNet/lemmy/issues/3649 has the offending query.
You made me curious. Since midnight lemmy.ca has averaged 110 requests per second, with multiple periods of doing ~3000 request/sec.
1 request every minute is nothing :)
Please go ahead and add lemmy.ca!
Also I don’t think this is correct, shitjustworks just had an extended outage while they were doing config updates.
FYI all mod actions should show up, but if a post has been “purged” (completely deleted from the lemmy db) then it also disappears from the modlog.