Any service I have that is public facing is proxied through Cloudflare. I run a firewall on the host that only allows traffic from Cloudflare IPs. Those IPs are updated via a cron job that calls this script: https://github.com/Paul-Reed/cloudflare-ufw I also have a rule set up in Cloudflare that blocks traffic from other countries.
For WAF, I use modsecurity with nginx. It can be a little time consuming to set up and weed out false positives, but it works really well when you get it configured properly.
Some of my applications are set up with Cloudflare Access. I use this with Azure AD free tier and SAML, but it could be set up with self hosted solutions like authentik.
I really wanted to use this and set it up a while ago. Works great but in the end I had to deactivate it, because my nextcloud instance would cause too many false positives (404s and such) and I would ban my own up way too often.
I do client ssl verification.
Nobody but me or my household is supposed to access those anyway.
Any failure is a ban (I don’t remember how long for).
I also ban every IP not from my country, adjusting that sometimes if I travel internationally.
It’s much easier when you host stuff only for your devices (my case) and not for the larger public (like this lemmy instance).
Some apps now have support for client certs (home-assistant ❤).
Nextcloud is one of the only apps that’s open without client ssl because it’d be highly inconvenient to share a file link with someone if I had to install a cert on their devices.
Plex app never works right for me so I just use the browser. My TV is too old to have old built-in so I have a VM in which I use a browser to watch plex.
How do you have this set up? Is it possible to have a single verification process in front of several exposed services? Like as part of a reverse proxy?
Yup, there are many ways of doing that. Most reverse proxies should support basic auth (easy, but browser UX is terrible and it breaks websockets) or TLS client auth (even worse browser UX, phones are awful).
The best thing is do something like Caddy + Authelia (which is what I currently do with most things, with exceptions for specific user agents and IPs for apps that require it, aka non-browser stuff like Jellyfin),
Yes it’s running in my reverse proxy.
Nginx is doing my “client ssl verify” in front of my web services.
You can even do this on a per uri/location.
For example, my nextcloud is open without client certs so I can share files with people, but the admin settings path is protected by client ssl.
I only expose services on IPv6, for now that seems to work pretty well - very few scanners (I encounter only 1 or 2 per week, and they seem to connect to port 80/443 only).
I still have firewall (that blocks almost all incoming connections) and sshguard setup. I also check the firewall logs daily, blocking IPs that I find to be suspicious.
I could probably do better, but with so few scanners connecting to my home server, I have managed to sleep way better than back when I setup a server on IPv4!
Also, even if my home server gets attacked, at least I know that my other devices aren’t sharing the same IP with them… NAT-less is a godsend.
Lol, I have heard some ISP horror stories from the Down Under.
I am fortunate enough that my country’s government has been forcing ISPs to implement IPv6 in their backbone infrastructure, so nowadays all I have to really do is to flick a switch on the router (unfortunately many routers still turn off IPv6 by default) to get an IPv6 connection.
Yeah the internet services here are really stuck in the past. Hard to tell if theyre taking advantage of the scarcity of ipv4 addresses to make more money somehow, or of theyre just too fuckn lazy
I mean, it’s not a big deal to have crawlers and bots poking at our webserver if all you do is serving static pages (which is common for a blog).
Now if you run code on server side (eg using PHP or python), you’ll want to retrieve multiple known lists of bad actors to block them by default, and setup fail2ban to block those that went through. The most important thing however is to keep your server up to date at all times.
I stopped messing with port forwarding and reverse proxies and fail2ban and all the other stuff a long time ago.
Everything is accessible for login only locally, and then I add Tailscale (alternative would be ZeroTier) on top of it. Boom, done. Everything is seamless, I don’t have any random connection attempts clogging up my logging, and I’ve massively reduced my risk surface. Sure I’m not immune; if the app communicates on the internet, it must be regularly patched, and that I do my best to keep up with.
I’ve been using crowdsec with swag for quite some time. I set it up with a discord notifier. It’s very interesting to see the types of exploits that are probed and from each country. Crowdsec blocks just like fail2ban and seems to do so in a more elegant fashion.
Waf is the way to go I think. Fail2ban has had it’s own issues over the years, and if you use keys then you can forget about the constant SSH attempts. The ‘AllowUsers’ option in your SSH config is a good place to start too.
I just find all of these “lock down port 22” posts to be so noobish. Declarative waf is the way to go
Edit: Red Hat Identity Management (IdM) + Hashicorp Vault if you really care about SSH. Rotate your keys and create new users automatically
Nothing too fancy other than following the recommended security practices. And to be aware of and regularly monitor the potential security holes of the servers/services I have open.
Even though semi-related, and commonly frowned upon by admins, I have unattended upgrades on my servers and my most of my services are auto-updated. If an update breaks a service, I guess its an opportunity to earn some more stripes.
All the legit reasons mentioned in the blog post seem to apply to badly behaved client software. Using a good and stable server OS avoids most of the negatives.
Unattended Upgrades on Debian for example will by default only apply security updates. I see no reason why this would harm stability more than running a potentially unpatched system.
Even though minimal, the risk of security patches introducing new changes to your software is still there as we all have different ideas on how/what correct software updates should look like.
I use Caddy as a reverse proxy, but most of this should carry over to nginx. I used to use basic_auth at the proxy level, which worked fine(-ish) though it broke Kavita (because websockets don’t work with basic auth, go figure). I’ve since migrated to putting everything behind forward_auth/Authelia which is even more secure in some ways (2FA!) and even more painless, especially on my phone/tablet.
Sadly reverse proxy authentication doesn’t work with most apps (though it works with PWAs, even if they’re awkward about it sometimes), so I have an exception that allows Jellyfin through if it’s on a VPN/local network (I don’t have it installed on my phone anyway):
@notapp {
not {
header User-Agent *Jellyfin*
remote_ip 192.160.0.0/24 192.168.1.0/24
}
}
forward_auth @notapp authelia:9091 {
uri /api/verify?rd=https://authelia.example
}
It’s nice being able to access everything from everywhere without needing to deal with VPNs on Android^ and not having to worry too much about security patching everything timely (just have to worry about Caddy + Authelia basically). Single sign on for those apps that support it is also a really nice touch.
^You can’t run multiple VPN tunnels at once without jailbreaking/rooting Android
I use ACL where I add my home/work IPs as well as a few commonly used VPNs IPs as well.
Cloudflare clocks known bots for me. Don’t see anything in the server logs, but I do see attempts on the CF side.
I am actually thinking about going back to Cloudflare tunnels. The only reason that I am hesitant is that I do use a fair amount of bandwidth as I host a mastodon server as well as a lemmy one. I don’t want to be stuck with a huge bandwidth bill.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: [email protected]
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
Any service I have that is public facing is proxied through Cloudflare. I run a firewall on the host that only allows traffic from Cloudflare IPs. Those IPs are updated via a cron job that calls this script: https://github.com/Paul-Reed/cloudflare-ufw I also have a rule set up in Cloudflare that blocks traffic from other countries.
For WAF, I use modsecurity with nginx. It can be a little time consuming to set up and weed out false positives, but it works really well when you get it configured properly.
Some of my applications are set up with Cloudflare Access. I use this with Azure AD free tier and SAML, but it could be set up with self hosted solutions like authentik.
Is everyone using Cloudflare?
cloudflare is sweet I just switched to there from Google domains and it feels like a billion options have just opened up
also the https security radio buttons I always forget to change in new sites
Pretty much, strange in the self-hosted community to have stuff like that happen.
Ignore them, as long as your firewall is set up properly.
Fail2ban and Nginx Proxy Manager. Here’s a tutorial on getting started with Fail2ban:
https://github.com/yes-youcan/bitwarden-fail2ban-libressl
Crowdsec is more advanced
Does it integrate with NPM?
Yes it does! You find everything on the site. It is very well documented.
I really wanted to use this and set it up a while ago. Works great but in the end I had to deactivate it, because my nextcloud instance would cause too many false positives (404s and such) and I would ban my own up way too often.
I do client ssl verification.
Nobody but me or my household is supposed to access those anyway.
Any failure is a ban (I don’t remember how long for).
I also ban every IP not from my country, adjusting that sometimes if I travel internationally.
It’s much easier when you host stuff only for your devices (my case) and not for the larger public (like this lemmy instance).
That sounds like an excellent solution for web based apps, but what about services like Plex or Nextcloud that use their own client side apps?
Some apps now have support for client certs (home-assistant ❤).
Nextcloud is one of the only apps that’s open without client ssl because it’d be highly inconvenient to share a file link with someone if I had to install a cert on their devices. Plex app never works right for me so I just use the browser. My TV is too old to have old built-in so I have a VM in which I use a browser to watch plex.
How do you have this set up? Is it possible to have a single verification process in front of several exposed services? Like as part of a reverse proxy?
Yup, there are many ways of doing that. Most reverse proxies should support basic auth (easy, but browser UX is terrible and it breaks websockets) or TLS client auth (even worse browser UX, phones are awful).
The best thing is do something like Caddy + Authelia (which is what I currently do with most things, with exceptions for specific user agents and IPs for apps that require it, aka non-browser stuff like Jellyfin),
Thanks. Authelia looks promising, but I can find anything about tls client auth.
Edit: actually maybe caddy supports this directly? https://caddyserver.com/docs/json/apps/http/servers/tls_connection_policies/client_authentication/
Yes it’s running in my reverse proxy.
Nginx is doing my “client ssl verify” in front of my web services.
You can even do this on a per uri/location.
For example, my nextcloud is open without client certs so I can share files with people, but the admin settings path is protected by client ssl.
I only expose services on IPv6, for now that seems to work pretty well - very few scanners (I encounter only 1 or 2 per week, and they seem to connect to port 80/443 only).
Isn’t that akin to security through obscurity… you might want one more layer of defense
I still have firewall (that blocks almost all incoming connections) and sshguard setup. I also check the firewall logs daily, blocking IPs that I find to be suspicious.
I could probably do better, but with so few scanners connecting to my home server, I have managed to sleep way better than back when I setup a server on IPv4!
Also, even if my home server gets attacked, at least I know that my other devices aren’t sharing the same IP with them… NAT-less is a godsend.
Must be nice living in a post 1995 country… theres only 1 or 2 ISPs in Australia that support ipv6…
Lol, I have heard some ISP horror stories from the Down Under.
I am fortunate enough that my country’s government has been forcing ISPs to implement IPv6 in their backbone infrastructure, so nowadays all I have to really do is to flick a switch on the router (unfortunately many routers still turn off IPv6 by default) to get an IPv6 connection.
Yeah the internet services here are really stuck in the past. Hard to tell if theyre taking advantage of the scarcity of ipv4 addresses to make more money somehow, or of theyre just too fuckn lazy
I’m guessing they’re on CG-NAT and someone upstairs thinks staying ipv4 reduces customer support costs.
Being put on CGNAT without IPv6 is terrifying.
Aussie supports full IPv6 and provide a /48
Don’t have vulnerable shit and ignore them.
Those are just weather.
I mean, it’s not a big deal to have crawlers and bots poking at our webserver if all you do is serving static pages (which is common for a blog).
Now if you run code on server side (eg using PHP or python), you’ll want to retrieve multiple known lists of bad actors to block them by default, and setup fail2ban to block those that went through. The most important thing however is to keep your server up to date at all times.
I map them every day.
I stopped messing with port forwarding and reverse proxies and fail2ban and all the other stuff a long time ago.
Everything is accessible for login only locally, and then I add Tailscale (alternative would be ZeroTier) on top of it. Boom, done. Everything is seamless, I don’t have any random connection attempts clogging up my logging, and I’ve massively reduced my risk surface. Sure I’m not immune; if the app communicates on the internet, it must be regularly patched, and that I do my best to keep up with.
I’ve been using crowdsec with swag for quite some time. I set it up with a discord notifier. It’s very interesting to see the types of exploits that are probed and from each country. Crowdsec blocks just like fail2ban and seems to do so in a more elegant fashion.
Depends on what kind of service the malicious requests are hitting.
Fail2ban can be used for a wide range of services.
I don’t have a public facing service (except for a honeypot), but I’ve used fail2ban before on public ssh/webauth/openvpn endpoint.
For a blog, you might be well served by a WAF, I’ve used modsec before, not sure if there’s anything that’s newer.
Waf is the way to go I think. Fail2ban has had it’s own issues over the years, and if you use keys then you can forget about the constant SSH attempts. The ‘AllowUsers’ option in your SSH config is a good place to start too.
I just find all of these “lock down port 22” posts to be so noobish. Declarative waf is the way to go
Edit: Red Hat Identity Management (IdM) + Hashicorp Vault if you really care about SSH. Rotate your keys and create new users automatically
Nothing too fancy other than following the recommended security practices. And to be aware of and regularly monitor the potential security holes of the servers/services I have open.
Even though semi-related, and commonly frowned upon by admins, I have unattended upgrades on my servers and my most of my services are auto-updated. If an update breaks a service, I guess its an opportunity to earn some more stripes.
Why is unattended upgrades frowned upon? Seems like I good idea all round to me?
Mostly because stability is usually prioritized above all else on servers. There’s also a multitude of other legit reasons.
All the legit reasons mentioned in the blog post seem to apply to badly behaved client software. Using a good and stable server OS avoids most of the negatives.
Unattended Upgrades on Debian for example will by default only apply security updates. I see no reason why this would harm stability more than running a potentially unpatched system.
Hell, debian is usually so stable I would just run dist-upgrade on my laptop every morning.
The difference there is that I’d be working with my laptop regularly and would notice problems more quickly
Even though minimal, the risk of security patches introducing new changes to your software is still there as we all have different ideas on how/what correct software updates should look like.
Fair, I’d just rather have a broken system than a compromised one.
I use Caddy as a reverse proxy, but most of this should carry over to nginx. I used to use basic_auth at the proxy level, which worked fine(-ish) though it broke Kavita (because websockets don’t work with basic auth, go figure). I’ve since migrated to putting everything behind forward_auth/Authelia which is even more secure in some ways (2FA!) and even more painless, especially on my phone/tablet.
Sadly reverse proxy authentication doesn’t work with most apps (though it works with PWAs, even if they’re awkward about it sometimes), so I have an exception that allows Jellyfin through if it’s on a VPN/local network (I don’t have it installed on my phone anyway):
It’s nice being able to access everything from everywhere without needing to deal with VPNs on Android^ and not having to worry too much about security patching everything timely (just have to worry about Caddy + Authelia basically). Single sign on for those apps that support it is also a really nice touch.
^You can’t run multiple VPN tunnels at once without jailbreaking/rooting Android
I use ACL where I add my home/work IPs as well as a few commonly used VPNs IPs as well. Cloudflare clocks known bots for me. Don’t see anything in the server logs, but I do see attempts on the CF side.
I am actually thinking about going back to Cloudflare tunnels. The only reason that I am hesitant is that I do use a fair amount of bandwidth as I host a mastodon server as well as a lemmy one. I don’t want to be stuck with a huge bandwidth bill.
sometimes I grab popcorn and “tail -f /var/log/secure”