Hi all,

If you’re just now signing in for the first time in 12+ hours, you may just now be finding out that Lemmy World and other instances where hijacked. The hijackers had the full abilities of hijacked user, mod, and admin accounts. At this time, I am only aware of instance defacing and URL redirections to have been done by the hijackers.

If you were not forced to sign back in this morning, contact your instance admin to verify mitigations were completed on your instance.

How?

This occurred due to an XSS attack in the recently added custom emojis. Instance admins should follow the issue tracker on the LemmyNet GitHub, as well as the Matrix Chat. Post-Incident Activity is still on-going.

Currently, it is likely that just your session cookie was stolen, with instance admins being targeted specifically by checking for navAdmin, an HTML element only instance admins had. I do not believe this to affect users across instances, but I have yet to confirm this.

What happens next?

As I am not the developers or affected instance admins, I cannot make any guarantees. However, here is what you’ll likely see:

  1. Post Incident investigation continues. This will include inspecting code, posts, websites, and more used by the hijackers. An official incident writeup may occur. You should expect the following from that report:
  • Exactly what happened, when.
  • The incident response that occurred from instance admins
  • Information that might have helped resolve the issue sooner
  • Any issues that prevented successful resolution
  • What should have been done differently by admins
  • What should be improved by developers
  • What can be used to identify the next attack
  • What tools are needed to identify that information
  1. A CVE is created. This is an official alert of the issue, and notifies security experts (and enthusiasts), even those not using lemmy, about the issue.

  2. A code security audit is done. This will likely just be casual reviews by technical lemmy users. However, I will be reaching out to the Mozilla Foundation and Cure53 as they recently did an audit of Mastodon. If there is interest in an external audit of lemmy and the costs are affordable, I’ll look into crowdfunding this cost.

@[email protected]
link
fedilink
English
302Y

Was lemmy.ml affected ?

@[email protected]
link
fedilink
English
472Y

Only Lemmy instances with custom emoticons were affected based on the Recap of the Lemmy XSS incident. So if Lemmy.ml doesn’t have these it should not have been affected.

@[email protected]
link
fedilink
English
112Y

I think generally the less flashy features (pun intended) the better. Text and links (and well sanitized) is all we need.

@[email protected]
link
fedilink
English
-22Y

Yeah. Hopefully the devs will learn to wait before pushing new technologies that don’t really matter. (they won’t)

Wintermute
link
fedilink
English
62Y

I agree. Sadly it seems the rest of the world does not. Hopefully as Lemmy matures we can get to a point where features are not pushed put half-baked because there aren’t enough people willing and able to give thorough code reviews.

@[email protected]
link
fedilink
English
52Y

Well, I’m glad I didn’t bother to add them.

@[email protected]
link
fedilink
English
-12Y

Less is more.

@[email protected]
link
fedilink
English
262Y

also, I cannot properly login into my lemmy.world account anymore. username/password work, but when I try to upvote it tells me i gotta be logged in. Tried apps and web.

𝐘Ⓞz҉
link
fedilink
English
142Y

Clear cache of the app

@[email protected]
link
fedilink
English
42Y

Thank you so much! I had the same problem but didn’t think of clearing the cache…

𝐘Ⓞz҉
link
fedilink
English
32Y

Cheers. Happy to help

@[email protected]
link
fedilink
English
72Y

For me it was cache and storage. Just cache didn’t work. (Using Liftoff)

@[email protected]
link
fedilink
English
22Y

sometimes just logging in again works. if you have it set to automatically log in, that happens sometimes.

@[email protected]
link
fedilink
English
42Y

I had to clear the app cache and local storage before it would work. Cache alone didn’t work.

@[email protected]
link
fedilink
English
112Y

Try clearing cookies (or app cache, on app), that will most likely resolve the issue.

@[email protected]
link
fedilink
English
72Y

I had the same problem on the liftoff app. Had to delete the account then sign back in and everything worked perfectly again. Personally I also changed my password because it made me feel better.

@[email protected]
link
fedilink
English
42Y

For liftoff, you can just add the same .world account again and it’ll work, it worked for me.

jcrabapple
link
fedilink
22Y

Sometimes you just have to learn the hard way.

Sure thing. So happy to see all good aga8n.

uhvayga
link
fedilink
162Y

Next, someone somehow finds a protocol-level vulnerability of ActivityPub. This makes the Fediverse and all its related services using ActivityPub (Mastodon, Lemmy, KBin, etc) gone in a night. Then Fediverse refugees must buy the last haven again from El*n or Sp*z in exchange for their souls. Finally, they stay in the haven until they die without something important they had past… – Fin –

Internet dystopia will come! 😱

I’d just log off lol

Toxic social media < no social media

Exactly 👍.

Uriel-238
link
fedilink
English
992Y

Having recently migrated from Reddit (and kept up with commercial social media hacks) I’m used to Nothing To See Here! We totally didn’t store your personal information in plaintext for hackers to snatch. Oh and maybe please change your passwords. All Part Of The Show!

So, by comparison, the response here is downright heartwarming.

@[email protected]
link
fedilink
English
82Y

Whoops! Reddit spazzed out and couldn’t send your post because it hurts spez’s feelings!

monotremata
link
fedilink
72Y

Does this affect kbin.social? I know we federate with Lemmy, but I think there are also code differences.

DarkenLM
link
fedilink
62Y

It’s a completely different codebase. Lemmy is written in Rust, while KBin is written in PHP. At least for now, there hasn’t been any issue this bad.

In the end, Magic Quotes will save us all.

monotremata
link
fedilink
3
edit-2
2Y

Thanks for clarifying that! I didn’t have to sign in again, but my account is on kbin.social, so I guess that’s why.

DarkenLM
link
fedilink
22Y

No problem, fellow KBinaut!

FaceDeer
link
fedilink
152Y

kbin is a completely different piece of software so I’d be surprised if there were many code similarities.

Still, that could just mean that there are kbin-specific hacks lurking in the future.

@[email protected]
link
fedilink
English
-772Y

Friendly remind that Lemmy.world is considering federating with Meta while Lemmy.ml will not.

I would strongly suggesting ditching Lemmy.world for Lemmy.ml before they can even get started with that stupidity.

NullDrive
link
fedilink
English
102Y

deleted by creator

frozen
link
fedilink
English
92Y

Or a smaller instance! lemmy.ml wasn’t meant to be the main Lemmy hub, and it would help the load on the larger instances to spread users out.

@[email protected]
link
fedilink
English
-22Y

I ran screaming from FB. Like, hair on fire running. That place is a roiling pot of piss. I would 100% not interact with ANY instance that has federated with them. They are a cancer that will infect this next big thing

RCMaehl [Any]
creator
link
fedilink
English
92Y

While this is definitely a discussion to be had (I’ve created a few accounts on different instances). Posting that here is just adding onto the drama.

@[email protected]
link
fedilink
English
-252Y

Not sure how you view it that way.

It’s just good advice for people who want to stay the hell away from a company like Meta that was willing to hand over private data to assist authorities in arresting a woman for exercising her reproductive rights.

Really not thrilled with the Lemmy.world users trying to downplay just how ridiculous the idea of federating with Meta is.

Is it possible to block .world entirely yet?

@[email protected]
link
fedilink
English
132Y

Look, Lemmy.world takes the rational way where it says there is no use to panic. Let’s be rational and analyze the situation rationally and go from there.

I support Lemmy.world. Let’s see how the situation develops and let’s make a rational decision with cool heads.

Baron Von J
link
fedilink
English
52Y

User-level instance blocking is not yet implemented. You could use an instance that is defederated from lemmy.world, such as beehaw.org.

RxBrad
link
fedilink
English
172Y

“Ma’am… this is an Arby’s.”

Threads is not a Lemmy issue. Mastodon is where the concern lies.

@[email protected]
link
fedilink
English
12Y

NOT A LEMMY ISSUE IDIOTS!

@[email protected]
link
fedilink
English
23
edit-2
2Y

Maybe ditching Lemmy.world makes sense but signing up on Lemmy.ml doesn’t. Pick smaller instances and spread out guys. That’s the entire point. :)

It’s a bit funny that 100k users signed up on the same instance… :)

@[email protected]
link
fedilink
English
-52Y

yeah, Lemmy is instilling a ton of confidence when instances disappear overnight (vlemmy) and others get all their data hacked and close off all communication with the largest user base that will exist in the fediverse. Y’all really about to make people go back to Reddit

@[email protected]
link
fedilink
English
-92Y

Right… so why open up the gates to Meta then?

That will push people right back to Reddit.

@[email protected]
link
fedilink
English
62Y

It’s already over, actually kind of boring. Please hack again.

@[email protected]
link
fedilink
English
562Y

Dumb take. Lemmy.world devs simply said it’s not likely Threads will federate with Lemmy anytime soon anyways, and they’ll make a decision when there is actually a decision to be made.

@[email protected]
link
fedilink
English
-472Y

removed by mod

@[email protected]
link
fedilink
English
382Y

I’ve made many comments saying why we should defederate with Threads but I still agree with the admins that there’s no point in doing a knee-jerk reaction on a threat that

A) won’t happen until at least a few months later

B) likely won’t federate with Lemmy anyways

C) isn’t actually a threat to Lemmy, but could pose a threat to microblogging websites like Mastodon

So yeah, waiting and seeing isn’t stupid.

@[email protected]
link
fedilink
English
-402Y

“Wait and see” is fine.

Federating in the meantime is not.

Stop trying to dance around the heart of the issue.

TheSpookiestUser
link
fedilink
English
112Y

Correct me if I’m wrong, but I don’t think Threads even has the ability to federate yet?

@[email protected]
link
fedilink
English
352Y

Federating in the meantime is not.

Dude, Threads isn’t federating with anything right now. That’s the point, we’re not federated so there’s no reason to make a decision on something that won’t happen yet.

@[email protected]
link
fedilink
English
-252Y

I’m not sure you understand the takeaway from the admin post last night.

NullDrive
link
fedilink
English
12Y

deleted by creator

@[email protected]
link
fedilink
English
172Y

https://lemmy.world/post/1274909

Conclusion:

From the points discussed above, the possible lack of moderation alone justifies considering defederation from Threads. However, it remains to be seen how Meta will handle moderation on such a large scale. Additionally, the inability of individuals to block an instance means we have to do what is best for the community.

Where does it say in the admin post that they are default federating with Threads?

@[email protected]
link
fedilink
English
72Y

They haven’t made a decision, as it’s no issue as there is no federation to threads and won’t be for the forseeable future.

Making a fuss about it now is as usefull as making a fuss over the sun dying in several million years.

@[email protected]
link
fedilink
English
22Y

I would strongly recommend making decisions for yourself.

@[email protected]
link
fedilink
English
172Y

XSS, classic

Thanks for your hard work, fuck the trolls that always have to poke holes in shit

Remy
link
fedilink
English
32Y

thanks for the update

@[email protected]
link
fedilink
English
392Y

Honestly, I see it as a win.

The people that did this didn’t really act out in a coordinated attack. They were just kind of playing around, redirecting to lemonparty, changing page elements.

It could have been a lot worse. The site could have been redirecting to malicious websites, downloading trojans, doing a lot of bad things. Instead, we got direct attention to the security vulnerabilities in question, and they’re being worked on and patched out relatively quickly. Helps that a lot of those on these communities are focused in programming and cybersecurity.

@[email protected]
link
fedilink
English
102Y

That’s fair, I was a little harsh. Not all hackers are looking to cause damage!

drdaeman
link
fedilink
English
142Y

The fundamental issue is not that emoji XSS (that’s just a vector), but how JWTs are implemented and [not] secured. I’ve read that it was reported at least this January (https://akkoma.nrd.li/notice/AXXhAVF7N5ZH1V972W).

So, developers were already aware, yet - as I’m checking 0.18.1 - they have not fixed the unsafe-inline and unsafe-eval CSP, haven’t made jwt cookie HttpOnly, and haven’t done anything about exp and jti in the JWTs. I hope the recent events will make them do to so, and not just patch this particular XSS.

P03 Locke
link
fedilink
English
22Y

Kegan may have been better off submitting a GitHub issue. And if issues haven’t been open for these critical security flaws, you should, too.

@[email protected]
link
fedilink
English
12Y

Yes, exactly. This could’ve been way worse.

@[email protected]
link
fedilink
English
82Y

Agree completely. In the grand scheme of things the damage that appears to have happened here is small potatoes, but it brought attention to the vulnerability so it was patched quickly. Going forward now, the authors and contributors to the project might be a bit more focused on hardening the software against these types of vulnerabilities. Pen testing is invaluable on wide user-base internet accessible platforms like this because it makes better, more secure software. Unfortunately this breech wasn’t under the “ethical pen testing” umbrella but it sure as hell brought the vulnerability to the mindshare of everyone with a stake in it, so I view it as a net win.

@[email protected]
link
fedilink
English
92Y

Is the sudden unannounced disappearance of Vlemmy related to this ? I had my main account there and it just disappeared

@[email protected]
link
fedilink
English
92Y

The most probable theory is that the server owner found illegal federated content on their server and decided to nuke the place entirely.

@[email protected]
link
fedilink
English
46
edit-2
2Y

The way the hack was utilized is honest very creative and interesting; either way, if all big communities could crowdsource money for security audits, I believe that it could help prevent something like this in the future.

Edit for clarification: I am NOT talking about how the exploit itself was created or achieved, I am talking about the act of having the power they did and just doing shock crap.

@[email protected]
link
fedilink
English
42Y

HTML injection / XSS vulnerabilities tend to be a sign of amateur hour to be honest. Made me a bit worried that I’m hitching my reddit escape wagon to the wrong technology. But sounds like this was due to instance-specific customizations rather than the core Lemmy tech, so hopefully we’re still on solid ground.

P03 Locke
link
fedilink
English
02Y

No, Bobby Tables is really not that creative at all. It is the most basic, entry-level exploit.

@[email protected]
link
fedilink
English
42Y

The way the hack was utilized is honest very creative and interesting;

That’s often the case with exploits.

@[email protected]
link
fedilink
English
62Y

Your not wrong, however, most of the time when these events happen it’s meant for something more insidious than lemon party.

@[email protected]
link
fedilink
English
0
edit-2
2Y

Sure. Not sure how that’s relevant though?

In general, finding an exploit requires looking for little tiny details that could exist in, really, any area of a given system; looking for a bug, and then exploiting that bug by understanding how input data can be used to create a deterministic chain of events.

This almost always requires thinking outside of the box.

There are people who are also paid to find these before malicious actors do.

It’s always going to be creative in some way, at least in the beginning.

It’s like when people first discover Quake’s fast inverse square root. Sure, the first time around it seems genius. In reality, code like that is actually everywhere, and there is a somewhat trivial aspect to optimizing those kinds of problems.

@[email protected]
link
fedilink
English
198
edit-2
2Y

This incident made me realize not to use an admin account for my primary lemmy account in my personal instance. I setup another account for instance admin purpose (with 2FA enabled) and keep it logged out, then remove my primary account from the instance admin list.

BarterClub
link
fedilink
English
62Y

Yup. Basics of running a server for anything. Never use your admin account and make a default backup with 2 factor.

@[email protected]
link
fedilink
English
762Y

This is a good mindset in general, when working in AWS you are not supposed to use your root account unless it’s absolutely necessary even if you are the only user. Hosting a Lemmy instance should be no different.

@[email protected]
link
fedilink
English
112Y

Yeah, even in Windows or what-have-you, you should always keep your admin account separate from your daily driver account for exactly this reason.

@[email protected]
link
fedilink
English
2
edit-2
2Y

I just setup a VPS for a Minecraft server for some friends and did exactly that. I was under a bit of a time crunch, but still took the time to think through those challenges in access for everything. Created an unprivileged user to run the server as, created a seperate unprivileged user for another service. Disabled password-based SSH login, etc.

I should probably setup a dedicated non-root admin account for administrative functions but that’s a problem for after work

@[email protected]
link
fedilink
English
62Y

most, if not every, linux distro work that way

@[email protected]
link
fedilink
English
172Y

I work with 2 factor, Oath, SAML, etc. all the time for work, and for the life of me I can’t get it working properly with Lemmy.

@[email protected]
link
fedilink
English
72Y

I use bitwarden, so it’s pretty simple: just copy the link from the 2fa button in Lemmy into the TOTP field in your account editor in bitwarden, and it’ll automatically recognize the format.

@[email protected]
link
fedilink
English
12Y

Yeah, I’m using Google Auth, and I’m getting codes just fine, but none of them work, regardless of how I try to use it.

@[email protected]
link
fedilink
English
62Y

Google Authenticator doesn’t seems to support SHA256: https://github.com/google/google-authenticator-libpam/issues/11

Maybe try FreeOTP instead: https://freeotp.github.io/

Notorious
link
fedilink
English
212Y

Lemmy decided to go with SHA256 for TOTP seed. This is a very odd move since many 2FA apps don’t support SHA256. I actually had to write a quick python script to spit out my 2FA code since Bitwarden doesn’t support it. Hopefully either Lemmy will change to SHA-1 or Bitwarden will start to support SHA256 seeds.

@[email protected]
link
fedilink
English
122Y

SHA-1 has been broken already.

@[email protected]
link
fedilink
English
22Y

Doesn’t matter too too much for a TOTP though. It just needs to be pseudorandom enough for a relatively difficult to guess 6 digit (or longer) code that changes every 30 seconds. Much more likely for someone to get phished for their TOTP than for someone to reverse engineer someone’s TOTP seed. Plus how would that even happen? The attacker would need to have the device and / or track the user’s 6 digit codes for quite a while at which point the game’s already up.

@[email protected]
link
fedilink
English
102Y

Preferably the latter, nobody should really be using SHA1 for anything security focused and new

@[email protected]
link
fedilink
English
17
edit-2
2Y

Wait, I use bitwarden and it works just fine. And yes, it’s using SHA256.

Screenshot:

@[email protected]
link
fedilink
English
32Y

Bitwarden definitely works with SHA256 and even SHA512, I believe. And with codes longer than 6 digits also. Oh and even Steam codes if you really want.

@[email protected]
link
fedilink
English
12Y

You can put steam OTP there too? I’m tired getting locked up from steam whenever I format my phone. If I can put steam OTP into bitwarden, that’ll be really great.

@[email protected]
link
fedilink
English
1
edit-2
2Y

Not officially (you have to trust some third party desktop app to do so), but it is technically possible. I think there’s a guide on the Bitwarden help articles.

Edit: link - https://bitwarden.com/help/authenticator-keys/#steam-guard-totps

@[email protected]
link
fedilink
English
22Y

wait wait wait. how did you get lemmy 2FA into bitwarden. I can only get it to set up in authy or google auth

@[email protected]
link
fedilink
English
22Y

You’ll need to either subscribe to bitwarden premium ($10 / year), or deploy vaultwarden in your own server and have your bitwarden extension/app use that vaultwarden server instead of the official server.

@[email protected]
link
fedilink
English
12Y

Yeah I have premium and every account I have that offers 2fa is set up but Lemmy will only let me set it up in authy or Google auth. It won’t give bitwarden as an option or give me the code to manually add it to bitwarden.

@[email protected]
link
fedilink
English
22Y

Just right click at the 2fa button and select “copy link”, then paste the link into bitwarden TOTP field. The link you copied should already be in the format otpauth://totp/xxxx which will be recognized by bitwarden.

@[email protected]
link
fedilink
English
242Y

How do I contact my instance admin?

I went ahead and logged out and back in on my own, but I was not prompted to do so by Lemmy and I don’t know enough about this stuff to say whether or not doing it on my own would’ve helped. I’m still pretty overwhelmed trying to figure this site out, so I apologize if this is a dumb question, just want to make sure I’m doing what I can to protect my own privacy.

I don’t want to fall for the millennial version of a Nigerian prince scam, lol.

@[email protected]
link
fedilink
English
3
edit-2
2Y

I wonder which instances required logins. I had to relog into my .world account, but my .ee account was still logged in.

@[email protected]
link
fedilink
English
12Y

I only have a .world, so I’m not sure. I’d be interested to hear from others!

@[email protected]
link
fedilink
English
22Y

I heard lemm.ee wasn’t vulnerable, so logging users out shouldn’t be necessary. To be vulnerable there would need to be custom emotes defined on the local instance by the admins, so I’m guessing they had none.

@[email protected]
link
fedilink
English
3
edit-2
2Y

You’re on the lemmy.world instance, so you can reach the admins by emailing [email protected], or posting in the support forum [email protected]

Now to answer whether there’s a difference between being promoted and doing it yourself - In this case, it’s suspected that session tokens were compromised. You know how when you enter some events, they vet you/your ticket once at the door and then put a stamp on your hand? If you go out and want to get back in, you don’t have to do the whole verification song and dance again, just show them your stamp? Well, that’s pretty much what a session token is - Lemmy vets your password once when you log in, and gives an unique session token to whatever browser or app you used to log in. That way, when you reopen Lemmy, you don’t have to enter your password again.

Now that token is compromised, you have to assume a hacker has your unique token. When you logged yourself back in, Lemmy did the whole validation process again and gave your browser/app a new, unique session token - that’s just how logging in works. But the important question is, did it invalidate the old session token when you logged out? Otherwise the hacker can still show the old token and pretend to be you.

Now if your browser/app prompted you to log-in today, you can be sure that your browser/app tried to get into Lemmy and was denied access. That means you can be sure your old stamp/token is now invalid. Logging out and in yourself doesn’t give you the same guarantee - you will have to check Lemmy code (or run some experiments) to know if logout does actually invalidate the old token. I haven’t validated Lemmy’s code, but I will say most half decent software will invalidate your token when you log out. If you want an extra layer of protection, change your password as well - even the software devs that forget to invalidate tokens on logout usually remember to invalidate them on password changes.

CommunityLinkFixerBot
bot account
link
fedilink
English
12Y

Hi there! Looks like you linked to a Lemmy community using an URL instead of its name, which doesn’t work well for people on different instances. Try fixing it like this: [email protected]

@[email protected]
link
fedilink
English
122Y

This post is weird. You’re typing like you’re in charge of things, but you’re apparently not.

It’s one thing to show some initiative, but you’re literally demanding a full report like the Lemmy devs work for you. You sound like someone who does this kind of thing for a living and felt the need to flex. Because otherwise, what the hell are you even doing?

Setting neurotically-specific demands for the developers makes sense if you represent a big instance or something, but you’re literally just a dude. You could have framed this entire post in a different way and gotten away with it. Right now, it’s creepy to anybody who actually reads the entire thing.

RCMaehl [Any]
creator
link
fedilink
English
2
edit-2
2Y

These aren’t demands, but I can definitely see how they can come off that way. These are industry standard post cybersecurity incident review questions by defined by NIST (NIST SP 800-61 Rev 2 Section 3.4.1) slightly rephrased.

@[email protected]
link
fedilink
English
-192Y

So much drama

m3t00🌎
link
fedilink
English
12Y

had a login/not logged in loop for a while yesterday. reset lemmy.world site data seems to have fixed

Create a post

A community to talk about the Fediverse and all it’s related services using ActivityPub (Mastodon, Lemmy, KBin, etc).

If you wanted to get help with moderating your own community then head over to [email protected]!

Rules

  • Posts must be on topic.
  • Be respectful of others.
  • Cite the sources used for graphs and other statistics.
  • Follow the general Lemmy.world rules.

Learn more at these websites: Join The Fediverse Wiki, Fediverse.info, Wikipedia Page, The Federation Info (Stats), FediDB (Stats), Sub Rehab (Reddit Migration), Search Lemmy

  • 1 user online
  • 134 users / day
  • 2 users / week
  • 158 users / month
  • 647 users / 6 months
  • 0 subscribers
  • 389 Posts
  • 12.9K Comments
  • Modlog