The way to make a server absolutely safe is to unplug its network cable. If you are connected to the Internet, you are connected to a world that is filled with miscreants who have a desire to find a chink in your server's armour.

The topics below consider some aspects of server security, considering risks and mitigations from my personal experience.

In recent years the number of port scans and URL scans has increased significantly.

With URL scans, I have made it a practice of blocking unknown robots. Some of these are from “research” groups in universities or large enterprises or unknown organisations who are using my resources for free and then being paid for their research without sharing any of the income that my costly resources helped them obtain. A lot of robot activity comes from inside of large clouds like Amazon and Azure that is not the main corporation's search engine, but rather is an unidentified robot that just has an agent string such as zgrab or simply python or curl. Again, they are using my resources and providing me with no compensation or benefit that I can discern, so I tend to block such activity.

Modern web sites are portals into sensitive information. The days of static web site that have no hacking potential disappeared with the last millenium.

If someone hacks credentials for some Mega Shopping Web Site they will likely expose personal details that could include partial or full credit card data. If they manage to work their way into a database the potential to steal the personal data of millions of people becomes possible. By now everyone should know to follow PCI-DSS guidelines for credit card details, encrypt all personal data, and never use a web-based GUI to manage a database.¹⁾

I see frequent robot scans for both web sites and IP addresses—e.g., http://10.1.2.3 as well as http://somedomain-that-I-host.com—that include common administration components in the URL, such as wp-login, wp-admin, or admin. For those annoyances the fail2ban tool is very helpful, although it needs to be configured and maintained. For example, when you add new hosted domains you need to add their log files to the list of ones that fail2ban will scan, and if some domains use frameworks like WordPress then you will need to carefully choose the rules that are applied to each domain. See my Fail2ban page for some ideas.

Checking the kinds of 404 errors you get can be helpful in spotting hackers. I have my web servers check the origin of the traffic and apply it to a list of allowed countries; any that do not match generate an e-mail to me to help me determine whether I want to allow or block it.

When it comes to port scans, I consider the origin of the traffic. If the origin is a country where none of my clients do business then I will likely place the entire subnet in the firewall to block all traffic, which may reduce a lot of spam traffic as well. This is an extreme measure and, of course, needs to be taken with sensitivity to the clients hosted on a given server, but I think by and large the majority of North American enterprises do not care about traffic from Asia or Africa. If you got an e-mail from Nigeria saying they wanted to place an order for whatever-it-is that you sell, would you even take it seriously or want to deal with the paperwork, expense, and complication of exporting? Probably not.

The only 100% sure security is to unplug the server. If you have people hacking away at your server, scanning for vulnerabilities without any limitation from the administrators, there is a reasonable chance that they could find a weakness one day, especially if they and your server are connected via gigabit fibre-optic networking that spans the globe so that someone in China has virtually the same speed of access to your server in Montreal as you do from your desk in Vancouver.

If I see traffic from 180.152.123.123 hitting my network and whois tells me that 180.152.0.0/13 is in a location I have never done or plan to do business, I add that subnet to my firewall. That part is easy.

There are many difficult aspects of maintaining such security, however.

One is the sheer volume. There are 65,533 possible /16 subnets and most ISPs are going to have /20 or smaller subnets so the number of origins quickly becomes millions and quite unmanageable by hand.

Another is the changing nature of the Internet. I have spotted subnets change hands, from one ISP in country X to another ISP in country Y, from one month to the next. It seems that people buy and sell subnets like they do stocks.

Sometimes you will find a subnet with multiple countries listed. For example, a subnet might be managed by LACNIC, but is registered to a company in North America or Europe. Some ISPs have POPs in foreign locations—i.e., when m247, a Romanian ISP, has a POP in France, do you consider that traffic to be French or Romanian? It's hard to say! I have seen whois records with a dozen or more countries all listed.

I know of some administrators in large organisations that feel that server e-mail is an annoyance to be avoided. They configure the servers to black-hole the mail. In doing so they have put a blindfold on themselves. In an Intranet of a small or moderately-sized enterprise this may be an acceptable risk, but in a large enterprise or where servers are connected to the external Internet this is an unacceptable risk, in my opinion.

Server monitors, like OEM or Zabbix, are wonderful tools, but as with fail2ban they are only as good as the admin who configures them.

Configure your servers to send mail to an administrators' mail group so that you can be quite sure that at least one of them will see something important. One role I had in a global enterprise sent around five thousand e-mails per day to the administrators, and our team of ten were expected to review them all. It seems overwhelming at first, but you can get used to seeing the ones that report common issues that can be put off so that you can focus on the ones that really need immediate attention. When you spot something big that could have been a disaster if it went unnoticed, you may be glad for the effort you made to read the server e-mails.

As with Apache, a tool like fail2ban is very handy in spotting and blocking robots that are scanning for vulnerabilities, but those tools are only as perfect the administrator who configures them. You need to configure fail2ban to scan your SSH logs, for example.

Relying on tools alone can leave you facing a disaster unaware, so I caution myself and any other administrator to personally scan a server's activity manually from time to time to see if there is anything going unnoticed.

The amount of criminal activity on the Internet is sure to keep increasing, which makes every administrator a kind of informal cyber-cop. Sometimes it's so frustrating that you wish those annoying people behind the activity would do something productive with their lives, but it can be exciting and rewarding at times, too.

We all make mistakes and overlook things from time to time, so I make it a habit of checking the status of my monitors and manually scan logs every month or two for unusual activity that the monitors may be missing. I have scripts that I use for these purposes that I will share on these pages under the topics above as I write about the things I've learnt and experienced.