When vlemmy.net disappeared (and it was the only one I had registered to), I registered at lemmy.world, sh.itjust.works, and while I was trying to register at lemmy.ML, I registered at feddit.NL by mistake. (then I requested a login at lemmy.ml, but never got a confirmation).
At the moment feddit.NL is the only instance I have a login at that I can use.
That was a happy mistake :-)
Maybe the problem is me. If feddit.nl goes down, I’ll know for sure.
All services, except Lemmy, is up and running again via the new domain: https://fmhy.net
Lemmy itself will require a significant database cleanup to get users transferred, posts and communities may not be possible
Cleanup will have to be done either manually or with a custom script
And now to answer some common questions I’ve seen floating around:
Why is lemmy.ml not affected?
WE DON’T KNOW. My assumption is that it’s popular (or lucky enough) to not be affected by this change. The only difference between our two domains is that lemmy.ml doesn’t use CF, and AFAIK, multiple other domains (with and without cf) are still unreachable.
Why can’t you just change the domain?
That’s not how federation works. Most services (including Mastodon and Lemmy) do not support changing URL properly, as doing so could potentially break the whole network.
Why is it taking so long to get back up?
Not all staff members have access to the server, CF, or domain registrar. Not to mention that this is a complicated task that requires a lot of effort, and one that we didn’t even know we had to do until the following morning. (We were under the assumption that the .ml TLD didn’t just explode.)
There is a GitHub issue on it and I experienced the exact same thing with my instance. A timeout occurs and the only way to fix it is to restart it seems. Like everyone else, it’s strange that it all happened at the same time.
This probably makes more sense although the issue I was experiencing earlier had similar logs as the issue I linked and others have commented on it too around the same time. I’m guessing they’re related.
It’s not that strange. A timeout occurs on several servers overnight, and maybe a bunch of Lemmy instances are all run in the same timezone, so all their admins wake up around the same time and fix it.
Well it’s a timeout, so by fixing it at the same time the admins have “synchronized” when timeouts across their servers are likely to occur again since it’s tangentially related to time. They’re likely to all fail again around the same moment.
It’s kind of similar to the thundering herd where a bunch of things getting errors will synchronize their retries in a giant herd and strain the server. It’s why good clients will add exponential backoff AND jitter (a little bit of randomness to when the retry is done, not just every x^2 seconds). That way if you have a million clients, it’s less likely that all 1,000,000 of them will attempt a retry at the extract same time, because they all got an error from your server at the same time when it failed.
Edit: looked at the ticket and it’s not exactly the kind of timeout I was thinking of.
This timeout might be caused by something that’s loosely a function of time or resources usage. If it’s resource usage, because the servers are federated, those spikes might happen across servers as everything is pushing events to subscribers. So, failure gets synchronized.
Or it could just be a coincidence. We as humans like to look for patterns in random events.
I get that message pretty much every time I visit a Community that I haven’t visited before (or maybe one that no-else from my instance has visited before).
It get fixed on a refresh (like the message suggests), so I’m guessing it’s a problem with time-out settings when my instance has to pull in a lot of new data.
Everything is back up now. I used connect for Lemmy and it actually tells you why an instance is not working. Lemmy.world was under maintenance, and the other one was just down.
For those affected by these outages in the larger servers and who’d be interested in helping spread around: my instance will be free for the first 250 registrations. There are ~220 spots still left.
The catch is that my registration process is (purposefully) difficult to avoid squatters/spammers/bots. So you need to do one of the following:
sign up to my main portal site first. If you do that, please give your email address so that I can send a confirmation with the first credentials.
signup to communick.news directly, and send me a DM on reddit (/u/rglullis), or here, or on mastodon) with the username you used to signup. If you don’t send the DM, I will assume it’s a bot and will deny the application.
Edit: downvoters, please don’t be so cynical. I’ve been offering this even before the reddit blackout. What is so bad about it?
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: [email protected]
A community to talk about the Fediverse and all it’s related services using ActivityPub (Mastodon, Lemmy, KBin, etc).
If you wanted to get help with moderating your own community then head over to [email protected]!
Rules
Posts must be on topic.
Be respectful of others.
Cite the sources used for graphs and other statistics.
I had that too. Tried multiple instances myself. Nice username btw.
When vlemmy.net disappeared (and it was the only one I had registered to), I registered at lemmy.world, sh.itjust.works, and while I was trying to register at lemmy.ML, I registered at feddit.NL by mistake. (then I requested a login at lemmy.ml, but never got a confirmation).
At the moment feddit.NL is the only instance I have a login at that I can use.
That was a happy mistake :-)
Maybe the problem is me. If feddit.nl goes down, I’ll know for sure.
Ha ha ha
Another DDOS? There’s been a few of them lately
Probably a sign that the platform is getting more popular, nobody wants to DDoS a place without any users
Yeah, for me too. Seems like whole fediverse has issues, problems with loading posts etc. at least for me
Same here, sh.itjust.works is currently down:
https://www.isitdownrightnow.com/sh.itjust.works.html
Which is not explained by
.mlfiasco, it is worth pointing out.sh.itdont.work
sh.itjustdoesn’tfucking.work
It does most of the time, to be honest
I know, but I couldn’t pass up this opportunity to make that joke.
Nice Domain Blaze 😀
Thank you! 😄
sh.itjust.works admin via matrix:
Not an attack.
See https://github.com/LemmyNet/lemmy/issues/3649 and https://github.com/LemmyNet/lemmy/issues/3165
That’s the problem with the fediverse. They tend to go down a lot. Same thing happens with mastodon servers.
So does reddit.
That’s fine by me. Opportunity to do something else.
But infinite contenttttt engagementtttt shareholderssssd
Here is the reason for lemmy.fmhy.ml
https://very.bignutty.xyz/notes/9hf13it1ced3b2za
And it seem to getting worse for lemmy fmhy
https://very.bignutty.xyz/notes/9hg4dquksvbha67h
What is Freenom and why are we supposed to know what that is?
Freedom in a domains registrar that gives out ‘free’ domains for up to a year. They were the registrar for .ml domains until this happened.
There is a GitHub issue on it and I experienced the exact same thing with my instance. A timeout occurs and the only way to fix it is to restart it seems. Like everyone else, it’s strange that it all happened at the same time.
wrong issue lol
This probably makes more sense although the issue I was experiencing earlier had similar logs as the issue I linked and others have commented on it too around the same time. I’m guessing they’re related.
The original issue is just a symptom of all database threads being tied up. People just don’t know how to follow an error message to the root cause.
The real source of the issue is db locking from triggers and cascading deletes on a major user change.
My report in https://github.com/LemmyNet/lemmy/issues/3649 has the offending query.
Thanks for clarifying.
It’s not that strange. A timeout occurs on several servers overnight, and maybe a bunch of Lemmy instances are all run in the same timezone, so all their admins wake up around the same time and fix it.
Well it’s a timeout, so by fixing it at the same time the admins have “synchronized” when timeouts across their servers are likely to occur again since it’s tangentially related to time. They’re likely to all fail again around the same moment.
It’s kind of similar to the thundering herd where a bunch of things getting errors will synchronize their retries in a giant herd and strain the server. It’s why good clients will add exponential backoff AND jitter (a little bit of randomness to when the retry is done, not just every x^2 seconds). That way if you have a million clients, it’s less likely that all 1,000,000 of them will attempt a retry at the extract same time, because they all got an error from your server at the same time when it failed.
Edit: looked at the ticket and it’s not exactly the kind of timeout I was thinking of.
This timeout might be caused by something that’s loosely a function of time or resources usage. If it’s resource usage, because the servers are federated, those spikes might happen across servers as everything is pushing events to subscribers. So, failure gets synchronized.
Or it could just be a coincidence. We as humans like to look for patterns in random events.
Interesting
Interesting. Never thought of it that way.
I get that message pretty much every time I visit a Community that I haven’t visited before (or maybe one that no-else from my instance has visited before).
It get fixed on a refresh (like the message suggests), so I’m guessing it’s a problem with time-out settings when my instance has to pull in a lot of new data.
It seem like it’s a bug which spread through more than a ddos.
Restarting the backend of an instance should fix the problem otherwise web users could try clearing the cookies
Only one working for me is
lemmelemm.ee. Lemmy.world and sh.itjust.works don’t workLemmy.world was only down for ten minutes
Everything is back up now. I used connect for Lemmy and it actually tells you why an instance is not working. Lemmy.world was under maintenance, and the other one was just down.
Overloaded servers (cpu,ram,disk), not enough bandwidth, endless possibillities…
Occuring on multiple instances at the same time? Unlikely.
idk, my server is in good shape 😅 But could be some bug in code which overload them.
For those affected by these outages in the larger servers and who’d be interested in helping spread around: my instance will be free for the first 250 registrations. There are ~220 spots still left.
The catch is that my registration process is (purposefully) difficult to avoid squatters/spammers/bots. So you need to do one of the following:
Edit: downvoters, please don’t be so cynical. I’ve been offering this even before the reddit blackout. What is so bad about it?
Thanks for sharing this, don’t mind the downvotes
I was facing issues with lemmy.world too, now seems to be okay