Archive for December, 2006

Security is hard, part 3

Thursday, December 28th, 2006

A few weeks ago, a great paper was published on side-channel attacks for RSA using the Pentium 4’s branch prediction logic. They were able to get nearly all of the bits of a 512-bit key for a (slightly) reduced-strength RSA implementation. This isn’t an algorithm break per se; it’s a generally applicable technique that can be used by one thread to spy on another thread’s actions, assuming it has the code that the other thread is running.

Hmm, I wonder what would happen if you had an account on a virtual server (e.g. Linode) and wanted to get an RSA key on the host? Hmmm…

If you’re interested in CPUs or security or both, the paper is worth reading.

Security is hard, part 2

Tuesday, December 26th, 2006

I’m sure you’ve all seen iTunes gift cards for sale in the stores this holiday season. You may also have heard that there is a bit of a security problem going around with them.

It works like so: the attacker writes down the activation number of the next card in line to be purchased. He then goes home and waits for a few days, and then tries logging into iTMS using the stolen code. If, in the meantime, a customer has purchased the card, the activation will work, and the attacker can steal the value off of the card.

Apple knows about this problem, and has a FAQ entry urging users to not buy cards with the code scratched off. If every minimum-wage cashier in the world checks these cards religiously, then it’s a perfect plan. :-)

Engineering and Christmas lights

Friday, December 22nd, 2006

One of the best Christmas movies of all time has to be Christmas Vacation with Chevy Chase. I just caught it on cable last night while I was trying (not) to get some code working.

I couldn’t help but be struck by the Christmas light scene, where he wires together a zillion strings of lights and then gets stuck troubleshooting it. I think programmers everywhere can sympathize - the code is perfect, why doesn’t it work? I’d never noticed it before, but looking at those scenes through the eyes of a debugger is hilariously familiar.

Security is hard

Friday, December 22nd, 2006

I’m going to take a few posts to talk about security, and specifically how hard security can be sometimes. I enjoy collecting tech (and non-tech) security stories, so I’m going to share some of them here over the next few posts.

So to kick this series off, here’s an interesting brute force attack against the keyless entry codes used to unlock some cars. I saw this linked from kottke.org. The problem here is that the attack isn’t nearly brute-force enough: in only 3,129 keypresses, you can open any car door using a standard 5-button keypad. The article estimates that it will take you 20 minutes to enter. And, in fact, that’s the max length of time it’ll take; the average would be 10 minutes.

There are a couple of problems here. First, the 5-digit code would seem to be chosen from a keyspace of 100,000 potential keys, but unfortunately, there are really only 5 buttons labeled 1/2, 3/4, 5/6, 7,8, 9/0, so it’s really 5^5 possibilities, or a mere 3,125. Furthermore, instead of it being 5 times 3,125 button presses, as the article points out, you wind up needing only one button press to test a code, leading to an 80% reduction in brute force time.

The other problem here is that the computer allows keys to be tested as fast as the attacker wants, with no regard for the number of missed key tries. Most secure login systems either lock the user out after a number of unsuccessful attempts or at least add enough delay between attempts to really slow brute force attackers down.

So, is 10 minutes, on average, enough time to make you comfortable that someone would notice the attack in progress?

Path MTU

Tuesday, December 19th, 2006

Anyone who has implemented a VPN has probably had to deal with MTU issues (unless you use a managed service). We’ve had code in our products for years to handle various MTU-related cases, and I’m going through some of it now. In double-checking our implementation, I took a peek at the relevant RFC, RFC 1191. In the introduction can be found this wonderful prediction:

It is expected that future routing protocols will be able to provide accurate PMTU information within a routing area, although perhaps not across multi-level routing hierarchies. It is not clear how soon that will be ubiquitously available, so for the next several years the Internet needs a simple mechanism that discovers PMTUs without wasting resources and that works before all hosts and routers are modified.

Woulda been nice…

At any rate, this is one of those IP features that is often handled incorrectly by firewalls. If you don’t allow Datagram Too Big messages (ICMP Type 3, Code 4) through, you are effectively killing performance (or even connectivity) for any paths with a Path MTU of less than your NIC’s MTU.

This is becoming more important: tunnel interfaces are proliferating, and as security consciousness increases, we’re only going to see more of this. For non-TCP-based tunneling protocols (IPSec, PPTP, GRE, L2TP, L2F, …) the carried packets are interface MTU minus overhead, so if you add, say, 100 bytes of overhead per packet, you wind up with a tunnel interface MTU of only 1400 bytes. Unless you want to have tons of frags, you have to listen for and handle Path MTU messages.

There are two common symptoms to path MTU problems:

  1. Connection speeds are bad and/or highly variable, caused by the fact that routers have to bounce your packets off of the line card and up to the CPU to fragment them (much slower), and by the fact that you’re doubling the number of packets to send. This also introduces the possibility of out-of-order delivery and the consequent fragment queuing. And to top everything off, fragmentation paths are never as well tested as non-frag paths, so there may be OS-level perf or security bugs you’re going to tickle.
  2. Connections suddenly quit working. While there are sometimes other explanations for this, a big cause of this problem is dropped datagrams due to MTU. TCP (usually) sets the DF bit on outgoing segments. If a router receives a segment that’s too big for the egress interface, and that segment has the DF bit set, it has no choice but to drop it. It then sends back its PMTU ICMP message, but the firewall drops that message, so the sender simply thinks that that the receiver has dropped off the face of the earth. Retransmissions happen and eventually the sender gives up. This generally happens with the first big packets in a session, such as (for example) when a terminal services client is painting the desktop.

So, if you’re an implementor of a firewall (or similar), pay attention to proper handling of Path MTU messages. And if you’re an end user and notice any of these symptoms, try removing firewalls and NAT devices and see if the problem disappears. It could be due to dropped PMTUD messages.

Slow hard drives

Wednesday, December 13th, 2006

I went to Micro Center this week to buy a new external hard drive; my 120G internal laptop drive is too full. I thought I’d try to get the fastest one I could, so I got a FireWire 800 enclosure and a new Segate 320G 7200RPM drive with a 16MB cache. Since I spent a little extra on fast parts, I thought I’d benchmark a bit. After timing some drag-and-drop operations, I was coming out with around 30MB/sec, which seemed too slow for this equipment.

I’m not a storage guy, but I seem to remember 33Mb/sec as an old transfer rate speed to ATA drives. A great Wikipedia article confirmed my hunch. I was instantly irritated with the vendors of this drive enclosure, so I wrote up a quick program to benchmark this thing.

The program confirmed that I was getting right about 30MB/sec copying a 2G file from my laptop to the external drive. So, just for the sake of comparison, I changed the program to simply writing 1GB worth of 0’s and pointed it at my internal hard drive.

30MB/sec.

I use a 17″ MacBook Pro for most of my dev work; I work in one of several VMs depending on the environment I’m developing for. My System Profiler tells me that I have a 1.5Gb/s (192MB/s) SATA controller with an attached Segate 120G 5400RPM drive. No obvious reason it would come in at 30MB/s sustained write.

I then tried my program on my new external drive. 57 MB/sec. Waaaay better. This is about what I’d expect out of this drive.

So, I have no idea why my internal drive’s sustained transfer rate is roughly 1/2 of my external drive’s. The data sheet says it should be capable of 42MB/s. Sure looks like it’s clipped at 33 though.

The true performance story, of course, has little to do with max sustained transfer speeds; seek time and bursts tend to dominate performance. Still, with a SATA controller and a reasonably fast hard drive in my laptop, I’m curious to know what’s up. The inescapable truth here is that it’s going to take 2x the amount of time to back up my iTunes repository that it should, or to copy VM’s around, which annoys me.

SMB don’t need no stinkin’ NAT

Monday, December 4th, 2006

Ken has a good write-up of a feature we found while working on a customer’s SMB networking issue: SMB is incompatible with (overload) NAT.

This is amazingly broken, and seems to me like an excellent opportunity for a DoS in a terminal server environment. Regardless, Ken’s walk-through of his debugging procedure is worth a read.

Don’t make a Rembrandt

Saturday, December 2nd, 2006

I can’t remember where I first heard the terms complexifiers and simplifiers, although it looks like they were originally coined in a blog post by Scott Berkun, but I’ve found them to be more and more useful in recent weeks.

My company is working on a Great New Service™ and we’re trying to finish up the first round of design. Virtually all of the intellectual energy I’m spending in these discussions is centered on factoring out complexity whenever possible.

Complexity is amazingly expensive:

  • It’s obviously more expensive to code.
  • It costs the business in terms of things like time-to-market.
  • It takes (much) longer to test.
  • You can wind up actually sacrificing the quality of the product you were trying to improve, by rushing the implementation process or short-changing QA (or, likely, both).
  • It presents bigger attack surface.
  • It makes security architecture harder.
  • It is more prone to bugs.
  • It makes individual bug fixes harder.
  • It requires smarter/better/more experienced developers.
  • It makes it very hard to replace those developers.
  • It’s harder to communicate your marketing message to your audience.
  • It can negatively impact scalability, performance, and responsiveness to demand changes.
  • It can kill your ability to adapt your product to changing market conditions.

I have a rule related to this: a system’s overall design is limited to what the lead technical person can fit into her head. Once your system is too complex to fit between one person’s ears (at a sufficient level of detail), you have to start dividing the architecture and scaling the team in ways that have yet more expenses built in, in terms of design, management, and programmer interactions.

Put another way: You have a limited amount of complexity that can be spread across your product. Spend it carefully.

Simplification is difficult, because complexity is the default route of most bright people that I’ve met. It’s fun (and on a deeper level, very satisfying) to build complex things. That’s why simplification has to be a going-in design goal. When you re-state simplicity as a goal, the team’s creative talents can be applied to the problem of simplification, which makes everyone happy.

There are tons of examples of people battling with this question, ranging from this week’s Vista vs. OSX shutdown wars (more here and here and here) to more abstract designs like Networking Truth #12.

This point of view may be an oversimplification :-) or you may already be years into a shipping product, with few chances to apply this principle. But if you’re in the design stage of a new product or service, like I am right now, do yourself a favor and (as my grandfather used to say) don’t make a Rembrandt!

Questions and suggestions

Saturday, December 2nd, 2006

This post is for any questions or suggestions you have. Feel free to post topic requests, feedback, etc. You can, of course, also e-mail me at dispensa - at - positivenetworks - dotcom.