Ouch! Details Of How Kernel.org And Other Linux Servers Were Compromised

OMG! It appears that thousands of Linux servers have had their passwords stolen and an organized crew has gone about compromising them to spread spam and malware far and wide. There aren’t any special cures except rebuilding the servers from scratch and the authours of this report suggest two-factor authentication become the norm. Passwords are just too easily lost/stolen/guessed. This is not about weak passwords but loss/theft. They do supply a test which my servers have passed…

Sad to say that kernel.org promised a full report and these guys are the best we have years later by careful analysis of the universe of IT. Shame on the Linux Foundation for not being more open and setting a better standard. Shame on the Linux Foundation for allowing the bad guys to besmirch the reputation of Linux for security. They were near the tip of the iceberg and didn’t bother to save others from their fate. There’s more to Linux than cranking out code.

See Operation Windigo.

See also Ebury FAQ

About Robert Pogson

I am a retired teacher in Canada. I taught in the subject areas where I have worked for almost forty years: maths, physics, chemistry and computers. I love hunting, fishing, picking berries and mushrooms, too.
This entry was posted in technology and tagged , , , . Bookmark the permalink.

12 Responses to Ouch! Details Of How Kernel.org And Other Linux Servers Were Compromised

  1. oiaohm says:

    DrLoser exactly why worry. The target is not Linux Desktops or Chrome OS devices or Android Devices. Sorry as Linux users we are not required to fell mercy for Windows users.

  2. DrLoser says:

    From the original pdf:

    Over half a million visitors to legitimate websites hosted on servers compromised by Windigo are being redirected to an exploit kit every day.

    You’re right, Robert. I wouldn’t trouble yourself over trivialities like this when there’s hunting season to enjoy and a brand new Pascal ranging calculator to experiment with.

  3. dougman says:

    keepass uses mono now btw

    Also, once a Win-Dohs system is infected I tell clients a complete reinstall is required, as unknown code has been allowed to taint the system.

  4. Mats Hagglund wrote, ” there is still one thing i have to learn: use much better password. OMG how terrible passwords am i still using. Mostly because my wife hates so much complicated (!AD?@kR$M#z!) passwords.”

    I gave up trying to keep up with passwords a decade ago. I use a password bank now so I only have to remember one but can cut-and-paste the ~hundred others… With good crypto, a randomly generated password is truly a beast. Let’s just say my passwords are much longer than 10 characters and I include all types of characters, so guessing is not an option. The password I use to open the vault is long and strong, but memorable. I use passwordless logins via SSH as often as possible.

  5. dougman wrote, “ssh -G 2>&1 | grep -e illegal -e unknown > /dev/null && echo “System clean” || echo “System infected””

    Apparently, that command spots a difference in the response of the trojan horse SSH executable. There are other commands in the articles to spot changes in the data-structures in memory and so on. The problem becomes that like that other OS, once a system you rely upon is corrupted, there’s no way to undo the damage that it did nor to be sure you fixed everything. Some go so far as to replace the whole computer… That’s a Hell of a lot of trouble for having weak passwords/controls. It’s a foot-in-the-door thing. If a computer is compromised and you connect to/through it with SSH, so is just about every computer accessible from that one. Weak passwords/controls makes any system a house of cards. I suppose this is one of the primary reasons to forbid logins as root. Pain.

  6. dougman says:

    KUKU…..LOL If one has the keys to the palace, nothing is safe, but MORE importantly Win-Dohs is unsafe regardless.

    ESET is damn fine AV solution, sadly I no longer need their software at the home office.

    ssh -G 2>&1 | grep -e illegal -e unknown > /dev/null && echo “System clean” || echo “System infected”

  7. Mats Hagglund says:

    Over 6 years using Linux and there is still one thing i have to learn: use much better password. OMG how terrible passwords am i still using. Mostly because my wife hates so much complicated (!AD?@kR$M#z!) passwords. Perhaps some day when me and she is using both our own personal computers that will change. She has to know the password to get some things done with our single desktop.

  8. JD says:

    Btw all ~20 of my servers seem clean, key based auth ftw. There really is no excuse to have SSH password auth enabled anymore unless it has been less than 3 minutes since your system was installed.

  9. JD says:

    To me, it looks like these guys put in a lot of work for very little gain. I mean, their untimate hack which kurkosdr so spectacularly failed to understand (try reading the article / links first next time before plagiarizing nursery rhymes) was a brute force attack.

    If criminals would stoop to such lows to send spam / generate some ad revenue, I can hardly imagine the chaos that awaits us once the XP 0 days start rolling in. Full Disclosure couldn’t have picked a more interesting time to close their ports.
    http://www.theregister.co.uk/2014/03/19/full_disclosure_closes/

  10. kurkosdr wrote, “Linux Security sat on a wall…”

    This is not about Linux security but security of a GNU/Linux server. No system that has passwords held by an intruder is going to be secure. Here, give me the keys to your house. We shall see how secure yellow houses are…

  11. oiaohm says:

    kurkosdr as normal you are wrong. Linux as an OS does include formal methods to keep logs that cannot be deleted by root user. The problem is it cost dollars to have them.

    dmesg for starts can be sent while a system is going to a serial port into a data logger. This is out of the root users hands. Also if you have nvram last section dmesg goes into nvram root cannot turn this off.

    Above that you have your normal syslog system. It also includes an output to a serial port/printer. The default in most distributions for syslog is rsyslog that includes send to another server in network.

    Windows event reporter is also only secure if you have a data logger connected to a serial port.

    No matter how you are logging if the system is x86 and many other arch it can be disabled. If you want logging that cannot be disabled by software you need it in the hardware.

    So kurkosdr formally everything to protect logs up until the point they get disabled exists in Linux and Windows. The problem is most people don’t pay for the data logger so the functionally cannot be enabled.

    kurkosdr for Windows and Linux there is a formal process to audit a comprised system it does not require formatting and starting over but its prity darn close. You need to be able to validate everything on the system.

    Basically download all the packages binary compare all the packages to what is installed. Locate all the packages that don’t own to a package or are from a package that were generated archive those. At this point you have all the data you are going to get you hope.

    The big problem these days is if you get to root or windows administrator you can have altered the firmware of the motherboard.

    If by nuke everything you meant destroy the computer motherboard and other items with long term means to store firmware alterations and start with fresh hardware you are right. If you meant just format the OS you were badly wrong. Basically its better to not to get compromised in the first place. Most places are not like kernel.org that can ring the maker of the boards and send over a person to use the program interface to nuke and redo all the firmware.

    kurkosdr with kernel.org a lot of the trace-back was possible because their cluster of servers was running a decanted log server. So they knew when logging seams to become bogus then could investigate who was using the system at that time.

    There is a issue with most hosting. Most hosting does not provide secure logging servers or option of serial data loggers.

  12. kurkosdr says:

    “There aren’t any special cures except rebuilding the servers from scratch”

    Linux Security sat on a wall.
    Linux Security had a great fall.
    All the king’s horses,
    And all the king’s men,
    Couldn’t get Security back together again.

    With Linux not going through the formality of keeping logs that can’t be changed by a root user, there is no way an audiit about what really happened to be conducted, so indeed, the best way is to nuke everything and start over.

    “Re-securing a compromised Linux system is very difficult. Intruders usually
    leave startup traps, trap doors, and trojan horses in their wake. After a security
    incident, it’s often easier to reinstall the operating system from scratch,
    rather than pick up the pieces.”

    “Attackers trivially hide their tracks. Once an attacker breaks into a Linux,
    she edits the log files to erase any traces of her incursion. Many system
    operators examine the modification dates of files to detect unauthorized
    modifications, but an attacker who has gained superuser capabilities can
    reprogram the system clock—they can even use the Linux functions specifically
    provided for changing file times.”

    Homework: Find the source of the quotes.

Leave a Reply