[jonssons]

pov of sysadmin:

- reverse proxy to backend webserver

- run checks on each request on the reverse proxy

- close all ports except what needs to be open

- only allow access to what needs to be seen

- perform frequent security updates (script it, automate it, run it hourly)

- never install more software then strictly required

- never allow SSH directly to your public address

- for ssh, use custom ports, cert (4096b) with passwords

- install logwatch or something else that will inform you about logs (elk stack)

- automate checks in logs for tell-tale signs of scriptkiddies (attempted access to /wp-admin or phpmyadmin for example) => automate adding source IP's to firewalls

- frequently check for malicious code (clamav, rkhunter, ...)

- only use stable and tested software

- only use stable and tested OS (Centos/redhat. DO NOT run production on anything else, especially ubuntu.)

- do not use ubuntu for production. it is not suited.

- install failtoban

from dev pov:

- implement measures to prevent sql injection

- don't require production to install dev libraries

- don't require production to install packages only available on ubuntu

- log strange behaviour to a file, make sure actions can be automated (iptable drop)

I'm no dev, but those are things I'd like dev's to do with their software.

[Jonnax]

Why is Ubuntu not suited to production?

I keep hearing it but it's starting to sound like Red Hat marketing like Macs don't get viruses.

[yellowapple]

An LTS Ubuntu release should be fine, though I'd hold off on using it in production until a year or so after release (to give Canonical time to shake out bugs).

A non-LTS Ubuntu release should be avoided, since they tend to be less stable.

If you're using something in the Debian family, Debian itself is probably the much better choice unless you're looking to use Canonical's enterprise products.

[kusmi]

I think it can be used in production but it's more work. Red hat and CentOS can be hardened following STIG security profile on first boot from fresh install which includes SElinux checks, firewall and service configurations, etc. I'm sure there's more to it, but reading documentation on these alone is enough to overwhelm me.

[jonssons]

In my experience:

- Ubuntu is not stable

- it is not hardened out of the box

- it can contain dangerous configuration out of the box

- it calls home

- it will tempt you to install dev packages or other packages unfit for production (because devs dev on ubuntu and they require the dev packages)

But please, don't mind me. Other people are better at explaining this. I'm biassed.

[ge96]

Curious if you're not a dev how do you know this (no offense meant).

Surprised about Ubuntu, what about Debian? Share same... Hmm...

A lot of these I have to tick off.

[rpod]

As a sysadmin, you'd have to have first-hand experience on what can go wrong on the dev side.

Also curious as to why Ubuntu isn't suitable for production.

[ge96]

Maybe I have been lucky I am using nginx now on top of centos for one site but the reverse proxy for example not using that.

I am on an Amazon EC2 in this particular case, also curious that Apache was also on there but only nginx is running. Had a long TTFB was trying to figure out what was wrong. Upon restart (new instance) I guess I got lucky, now back to regular site speed around 2 seconds load (full page).

Sorry rambling, cool stuff. Been on VPS, I guess EC2 is still the same thing but next would be serverless I guess.

[jonssons]

I am a sysadmin with approx 15 years of experience running linux systems and open source software. When I set things up, I tend to form things my way. I usually have little resources for monitoring or staff, so automation is the best to do. A server serves, it may produce errors and you can act upon that (error 404 for not finding phpmyadmin (which should _never_ be public => add ip to drop table). There are scripts that can do this for you.

However, the application which is being served, may produce different errors (failed login attempt, attempt for sql injection, cross site attack, ...). Some things are better logged by the application, Then the application needs to produce these logs. An awful lot of applications do nothing when presented with any of these type of attacks, making them extremely vulnerable. It's the dev's job to provide logging.

Debian: fine if you like pain :) it's stable.

[ge96]

How do you make PHPMyAdmin not public aside from say using one of those directory passwords.

Also it is somewhat of a similar question as to how Git is public but private.(Not GitHub your own setup)

[jonssons]

There are lots of possibilities. For example: Only allow web connections through localhost and start an x session through ssh. This would require you to install a webbrowser though, so that's another security risk. Perhaps you could set up a VM with a dedi IP and only allow connections from there. (secure it well)

Another possibility would be to first pass a reverse proxy that will request a client-side certificate. That'd make it pretty secure.

A password protected directory is just asking for a brute force through a botnet. Your failtoban won't even block anything because the attacker would only let 1 host attempt a password once per 30 min or per hour. But if he has thousands of bots to his disposal, he can still bruteforce a couple times a second.

That's OK you say. I have a strong password. Don't forget: Each connection and failed login attempts is logged. Your logfile will quickly grow. If you did not set a seperate VAR partition on your disk, your root disk will quickly get full and server will crash. Additionally, it will fill up your bandwith and be as effective as a ddos attack.

Don't use passwords. Use certificates. easier for you, impossible to hack.

There are other methods of course, but these would be my preferred methods. Setting phpmyadmin publically is asking to get your data stolen. Do not assume there are not exploits in the latest phpmyadmin. But don't forget to update it within the first 10 minutes when a security patch comes along, otherwise your data may well already be compromised.

[ge96]

Oh man I feel so dumb, you know despite what I can do. I guess that's the problem of trying to do everything you can only get so deep.

Use certificates? What is that not keypairs?

I logged into this Digital Ocean droplet and was surprised to see 15,000+ failed login attempts to SSH. I hadn't seen that before.

I don't know if using PHPMyAdmin is noobish. I still primarily use MySQL/Maria (only).

Thanks for the tips. I learn a lot though HN.

[jonssons]

using certificates is getting advanced. Here you'll allow the server to request a client certificate from the visitor. The visitor will have to supply the certificate (which is 4096b if you take security serious).

Easy (read easier) to do with nginx: https://www.google.de/search?q=nginx+client+side+certificate...

It requires quite a bit of steps, but is as secure as it gets what web access is concerned.

You should not be surprised to see 15k+ failed login attempts on ssh with popular ISP's. As I said, a system is the most vulnerable when it has just been installed and a/the default root password has not yet been changed. Simply disallow password login on ssh, change the port and only allow non-root users you need to allow. I have on my systems only 1 user allowed to login, authenticated by a 4096b key. There is no way an attacker can use ssh if not using an ssh exploit. The system is updated automatically every hour. This way known exploits are very quickly taken care of.

For me, server security has been a practice over many years and it takes many years to perfect your 'secure server setup'. It's depressing how many companies do not adhere good security practices and just leave their systems unprotected. Especially mail servers.

[no1youknowz]

> - reverse proxy to backend webserver

Can you give more clarification? Do you mean LB -> Firewall -> Web Server?

> - run checks on each request on the reverse proxy

Can you give more information on how to do this please.

Thanks

[jonssons]

> - reverse proxy to backend webserver

Using a reverse proxy allows you to cache static content, decreasing load. I'm not talking about load balancers. These days you need them when your website pulls _a lot_ of requests (1000/s) Your reverse proxy will also stop unwanted requests and indeed will have a firewall incorperated.

> - run checks on each request on the reverse proxy

post => clamav

post => sqlinjection

get => blacklisted address?

any => iptables (let any connection go through your firewall. It's a check)

don't terminate SSL on reverse proxy though. It's a bad idea, unless you are a bad person and like to read your visitors data they think is secure.

There are plenty of other ways of using security on a reverse proxy. Have a look around on google.

[corobo]

No backups? Security involves the ability to recover from a breach too! :)

[jonssons]

if you wanna go there. I use a custom script to make backups, encrypt them using gpg and store them on public ftp boxes. Automated retrieval of said backup in case of problems and automated server recovery in case of disaster.

[DKnoll]

> - only use stable and tested OS (Centos/redhat. DO NOT run production on anything else, especially ubuntu.)

I'd tend to agree about Ubuntu because... why aren't you just using Debian? I would generally opt for CentOS for a prod server, but I am now starting to think about using OpenBSD in production environments.

Ultimately it comes down to the way you configure the server, not the distro you use. The only benefit of Enterprise Linux (CentOS, Red Hat, Fedora, Oracle) over Debian out-of-the-box (that I am aware of) is the presence of SELinux, which you can still set up on Debian. All have insecure configs by default.

I will say that the live kernel patching in Ubuntu Server now is cool. I used KSplice on CentOS before it was discontinued (only on Oracle Linux now).

[jonssons]

I don't use debian because I don't like apt. The most vulnerable moment of my systems is when the ISP sends the password in plain text over e-mail. Until the password has been changed and the system has been sanitized(), I consider it compromised.

() updated, rkhunter, clamav, ...

[LinuxBender]

For my own sites, I do all of the things jonssons said, but I also use haproxy stick table to limit the number of errors a user may generate. I also route certain URI patterns to a tarpit.

At work, I show developers, ops and infrastructure folks these things and let them decide if they want to test and implement any of it.

[atmosx]

If you don't use FIM there is no way in hell the above list will protect you from targeted attacks.

[jonssons]

rkhunter does fim in a limited way. I hope OP is not talking about sites with sensitive data. ossec is usually great for storing more sensitive data.

The above list will help you from most scriptkiddy attacks, which are 99% of the attacks I have experienced. It's a good idea not to piss off anybody on the net or do it with a sufficiently obscure account.

[atmosx]

Agreed. I would add "auditd" and "acct on" to your list.