Putting My Money Where My Mouth (well Blog) Is
A couple of days ago I extolled the virtues of an online friend who does contract SysAdmin work, Demitrious, and I thought to myself "DOH! If he's good enough to write about then why don't you use him yourself?" since I'm currently locked in a seemingly infinite loop of:
Work ==> Code ==> SysAdmin ==> Work with Client / Partner / X ==> Code ==> SysAdmin ==> Repeat from start, do not pass go, do not sleep well, do not collect the proverbial $200.
So I IM'd Demitrious, went over my concerns -- basically I needed a security review of my www.RackSpace.com box, to get QMail going and a rebuild of PHP and Apache that I could trust (he's done this a lot more than I have). He was very cool with helping me, I gave him Root access (That's Trust) and I kind of just sat back and answered questions. The result?
I'm Happy. Happier than I actually expected to be.
He found a bunch of security holes and other system level issues. Some that I / another person had induced, some from the default www.rackspace.com setup (basically gid / uid file attribute issues / an outdated SendMail installed by RackSpace / an obsolete Kernel and wrong ATA IDE drivers among others ). He got QMail up in, well less than 30 minutes, setup sqwebmail and did the Apache / PHP the next night. The QMail was particularly impressive since QMail is hard to get going. It's absolutely wonderful once it's up but hard to get installed.
So I've taken my own advice. I've tested the waters and I'm pleased. I'd also point out to everyone that even when you have a top tier provider like RackSpace it's not necessarily true that everything is 100% correct with just a default configuration. Here were his comments:
1: old kernel (2.4.9-31.3RS)
2: have ata100 controllers and ata 33 hard drives (or improper
controller drivers)
VP_IDE: VIA vt8233 (rev 00) IDE UDMA100 controller on pci00:11.1
ide0: BM-DMA at 0xb400-0xb407, BIOS settings: hda:DMA,
hdb:pio
ide1: BM-DMA at 0xb408-0xb40f, BIOS settings: hdc:DMA,
hdd:pio
hda: MAXTOR 6L020J1, ATA DISK drive
hdc: MAXTOR 6L020J1, ATA DISK drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
hda: 40132503 sectors (20548 MB) w/1819KiB Cache,
CHS=2498/255/63, UDMA(33)
hdc: 40132503 sectors (20548 MB) w/1819KiB Cache,
CHS=39813/16/63, UDMA(33)
Yes, that's right, I asked him to log what he did so I had a record and he did it. Tres cool.
Disclaimer: I haven't talked to RackSpace about this yet. Maybe there are valid reasons, maybe not. It's unclear to me. I'll let you know when / if I do. I'm not bashing RackSpace on this but it's all about trust for me and I trust Demitrious.
The scariest thing of course is that fixing the problem he noted above requires a kernel rebuild which is dangerous and can take down the whole machine. Here was his approach to handling it:
fuzzygroup: What do you think I should do here ? DemitriousK: latest stable kernel is 2.4.18 DemitriousK: you do the math fuzzygroup: Ouch fuzzygroup: Here's my concern fuzzygroup: I know that RackSpace customizes their RH build fuzzygroup: I don' t know what they do exactly fuzzygroup: but they are generally pretty good fuzzygroup: although they do screw up as well DemitriousK: everything they do for the kernel is in /usr/src/linux-2.4/.config fuzzygroup: My concern is that we get going on it and then the box goes down fuzzygroup: Ok. Good. DemitriousK: and if we inform them of the reboot and have them on call i can make sure that the old kernel is available as a boot menu option with a default delay of 5 secs and defaulting to the new kernel fuzzygroup: Ok fuzzygroup: So you would go for the new kernel then., right ? DemitriousK: if thats an acceptable game plan. max of like 5 mins downtime unless i screw up massively DemitriousK: which i havent done in years, but dont discount it fuzzygroup: that's fine then DemitriousK: plus a couple of sec patches, yes fuzzygroup: Everyone makes mistakes. It's ok. fuzzygroup: I rarely get angry -- I just want stuff fixed when there are errors. DemitriousK: yea. and if we have rackspace watching local all will be well very quick fuzzygroup: (It takes a lots and lots and lots to get me angry) DemitriousK: which is why i wouldnt do it like.... NOW
When I pointed out that I had a major client demo the next day even though he was ready to do this, he wisely recommended that we wait. Wise beyond his years, IMHO.
A Note About Security Audits
I've had security audits / security reviews done for me before and I've done them myself (not nearly as well as Demitrious did). Here's what happens:
-
The audit starts.
-
The auditor finds LOTS more than you knew about.
-
You think "I don't really need to fix all of this".
Don't. Fix stuff. Even if it's inconvenient. Remember -- if you don't fix things, the great cosmic karma weenie will hit you in the head HARD. Example -- As soon as I had the audacity to write about backup, I started having data loss. It's almost guaranteed that if you have a security audit down that you'll have problems if you choose not to fix things. Keep it in mind.
5:56:22 AM Google It!
IM Me About This
|