In July 2008, some security problems with a Linux system required a trip to the customer's site. Following are some notes from that time.
I'm about to make an unscheduled trip to beautiful Omaha, all because a Linux system got compromised by Internet attack. Fortunately, the system is running well enough to keep the application going (A-Shell being nearly invincible!), but it's otherwise nearly crippled. Can't execute simple commands like ps, rpm, vi (vue works!), all the networking commands are broken, the backup program user interface is broken, etc. To compound matters, the problem started happening so long ago that it isn't even clear how far back we need to go (or just reload the OS). But I thought it might be nice to share a few warnings/thoughts/reminders about how to avoid this kind of situation:
| • | Don't expose Telnet to the Internet. Use SSH instead. Same goes for FTP; use SSH (SFTP) instead and block the Telnet and FTP services from the Internet. You need to leave Telnet running for local connections in order to use SSH tunneling though. |
| • | Make sure that you don't have obvious login/passwords (like user/user, staff/staff, etc.) We're not actually sure if this is how they broke in, but we can see from the logs hundreds of attempts to log in with various typical usernames (user, staff, admin, john, fred, etc.) It should go without saying that the root password better be secure. |
| • | Use a firewall, and configure it properly. In this case, they had a Cisco firewall, and it was supposed to be blocking outside incoming connections except for a few special IP addresses (like mine), but apparently a lightening strike caused it to lose its rules and revert to wide-open incoming access. If possible, require all the remote users to come from fixed IP addresses (which are easy to configure in even the simplest Internet routers). |
| • | Employ some kind of log-checking mechanism. RHEL includes a thing called LogWatch, which can be used to display or output a semi-consolidated listing of all or selected log activity, but you still have to arrange to have it execute on a daily basis and for someone to review the logs. Better still would be to use software to dynamically look for exceptions, but there doesn't seem to be clear, obvious, and simple-to-configure choice that I can see. There are commercial utilities such as TripWire, and there are open source utilities such as swatch, but they all require a fair amount of time spent figuring out what you want them to look for and how to get them to do it. (In this case, merely reporting failed login attempts would probably have been sufficient to raise the red flag.) |
| • | Use a "real" backup tool (I still like BackupEdge) that allows you to fully restore the root filesystem. This is probably what I will end up using. But even this is not helpful if you don't have a good rotation of backup tapes, including week-end or month-end tapes going back at least a couple of months. (In this case, we know the problem started more than a month ago, but aren't sure because that's how far back the logs go.) |
| • | Tip: edit /etc/logrotate.conf to change the default rotate 4 to a higher value to increase the number of archived copies of the logs). |
| • | Configure your disks so that the application is in a separate filesystem (e.g. /vm or /u). Not only does this isolate the application and the root from problems originating in the other, it also makes it a lot easier to do an emergency restore (or reload) of the root filesystem, without exposing the application to risk of unwanted over-writing. |
| • | Install security patches. There was a time when Linux was considered reasonably immune from exploitable security flaws (and still is much much better than Windows, which requires updates on a near daily basis to remain secure), but if your server is exposed to the Internet, you need to think about updates. This is one of the advantages of going with a commercial distribution, like RHEL, which comes with a subscription for patches. |
| • | Turn off any services you aren't using - such as httpd, etc. |
This discussion was originally posted on the A-Shell forum under the title "Linux System Cracked - Lessons", and may have been updated more recently than the version here.
