There’s been a lot of discussion recently about the NSA eavesdropping programme, which reportedly has been surveilling US citizens without first getting a warrant. In one of these discussions, someone asked:<\/p>\n
What’s the worst case scenario? How big could it be?<\/p><\/blockquote>\n
That’s a really good question. It occurs to me that no one has really attempted to address this yet in layman’s terms, so here goes….<\/p>\n
<\/p>\n
The short answer is: ‘Big. Really really big. Think Google, then add a bit. But fairly simple, too.<\/strong>‘<\/p>\n
Let’s perform a quick – and necessarily broad – assessment of the state of the art in Information Retrieval and data managment technologies, then try to tease out a few reasonable suppositions.<\/p>\n
Before I go any further, I have to preface these comments with a few caveats:<\/p>\n
\n
- Everything that I’m about to write is based on supposition. I am not now, nor have I ever been, a member of any intelligence agency. I have no certain knowledge about any of the details below.<\/li>\n
- I do have experience working in the field of knowledge management, Information Retrieval and natural language processing. I’ve written software applications that have been used to search very large language-based information resources, commonly called ‘corpuses’.<\/li>\n
- Much of my professional work requires a fairly detailed knowledge of networking technologies and architectures and their inherent strengths and weaknesses.<\/li>\n
- I’m no expert, but my work requires a fairly solid theoretical understanding of computer and network security.<\/li>\n<\/ul>\n
Okay, full disclosure is complete. Let’s take a quick stroll through the state of computers, computer networks and the Internet today. We’ll start by looking at the public search engines.<\/p>\n
Google, Yahoo!, Microsoft and About.com all have fairly large and fairly well understood search engines. While they differ in implementation, they all perform essentially the same task:<\/p>\n
\n
- Periodically (typically once a week or so) visit every public website on the Internet, and take a copy of every HTML (web) page and image file.<\/li>\n
- Store all these files (and metadata<\/em>, information about these files) in a very large repository, often referred to as a database, but more accurately called an index.<\/li>\n
- Perform analysis of all the files and the metadata, and use that information to determine the relation of contents of these files to data contained in other files.<\/li>\n
- Respond to requests for information (searches) entered by users of their system. These searches are all based on key words<\/em> – important terms that the searcher thinks will occur more often in the content they’re interested in.<\/li>\n<\/ul>\n
All of the steps described here are really simple to perform. A moderately talented programmer could write her own simple searchable index in a fairly short time. The analysis tools of each of the search services vary fairly widely, and are closely guarded trade secrets, but we know enough about them to draw some simple conclusions:<\/p>\n
\n
- Google uses a system called PageRank, which in its simplest form states, ‘A lot of other people link to this page using these key words. If others think this is useful, someone searching for the same keywords will likely find it useful, too.<\/em>‘<\/li>\n
- For quite some time, Yahoo! relied heavily on people to sift through the web pages its crawler found, and to categorise the content. It worked for a while, but the Web quickly got too big for this process. It’s reasonable to assume, however, that they applied at very least the same philosophy of broadly categorising their data in order to reduce the number of pages that have to be searched.<\/li>\n
- All of the search engines rely on keyword searches.<\/li>\n<\/ul>\n
Let’s look at this last point in more detail. Why don’t search engines use something other than keyword searches? Well, the answer is that because there are so many web pages around (billions and billions, as Carl Sagan used to say), that they need to keep the processes simple. There simply isn’t enough computing power in the world to cope with really elaborate processes for that much data.<\/p>\n
You see, computers are fundamentally different from human brains. They’re restricted to what’s sometimes called classical logic, the kind of thing they teach you in freshman philosophy classes. Computers are really bad at doing things that human and most animals do really well: determining patterns from multiple sources, and contextualising the information extremely rapidly.<\/p>\n
An example: When you see a slight movement out of the corner of your eye, your brain evaluates the data and within a few tenths of a second, your body has put you in a warning stance, you focus on the movement and try to determine if the cause of the movement requires a flight or fight reaction, or whether there’s no cause for alarm. If the moving thing turns out to be a black cat, you might:<\/p>\n
\n
- Sigh with relief and relax, because that just bootsie, your house cat.<\/li>\n
- Feel vague alarm, because cats give you allergies.<\/li>\n
- Ponder for a moment, trying to remember if any of your neighbours owns a black cat, or if you’ve seen any lost cat posters in the neighbourhood recently.<\/li>\n
- Cross yourself and spit, because your grandma taught you that black cats are bad luck, and with the way your day has been going, more bad luck is the last thing you need.<\/li>\n<\/ul>\n
As you can see, each of the reactions is derived from a wealth of prior input that your brain has received. The fact that the process described above typically takes no more than a second or two is, from a computer science perspective, nothing short of a miracle. Your brain has sorted through millions of data points and resolved a very complex problem in a fraction of the time it would take a computer.<\/p>\n
Nobody quite knows how the brain manages this feat, but they do know that doing the same thing with a computer would require reviewing every single input ever received, and performing a costly (in terms of the number of instructions) comparison between it and the new input. Then it would have to permute all these comparisons to come to some kind of useful conclusion.<\/p>\n
It’s not so hard to do if the number of inputs are limited, but doing it with a lot of apparently unassociated inputs quickly increases the ‘problem space’ – the number of possible answers. A simple game of chess involves so many possibilities that the big computer makers use the game as a way of proving whose computer is, uh, bigger, if you know what I mean.<\/p>\n
So search engines like to keep things simple. They’ve got billions of data points to consider, so they can’t really afford to do much with each of them. The thing that makes them really impressive is that fact that they can do anything useful at all with the volume of information that they deal with every day.<\/p>\n
Now let’s consider the NSA. These guys are really good. They hire the most talented mathematicians, computer scientists and academic researchers they can find. They reward them by giving them the best toys – I mean, tools<\/em> that money can buy, and they motivate them by telling them they’re doing the single most important thing an American can do: Keep the country safe by keeping it one step ahead of the Bad Guys.<\/p>\n
Is it reasonable to think that the NSA could surveil the entire Internet, the same way Google and co. do? Sure it is. Google runs most of its operations on beefed-up PCs, not very different from the one I’m using to write this diary. The NSA has access to hardware and software that Google can only dream about. It’s reasonable to assume that they can do a much more efficient job than the public search engines.<\/p>\n
But there’s a lot more data than just the web to think about. The number of email messages, telephone calls and live computer links absolutely dwarfs the web, especially because it’s all happening in real time.<\/p>\n
So let’s assume that the NSA, in order to keep up with it all, has to be a few times the size of Google. Is that a reasonable assumption? Just barely. Published estimates on the number of computers Google uses to manage its data run from between 60-100,000 machines, spread out over a number of centres world-wide. They take up a lot of space. But that’s okay, if Google needs more elbow roompace, they just cut another check and pay a large service provider to give them more space in their server room.<\/p>\n
The NSA on the other hand has to work in secrecy, which means that they can’t plunk servers down all over heck’s half acre. They do have very large establishments to work in, but they’re constrained by the fact that the information has to be fed into a limited number of locations. Switches these days can send tens of billions of bytes of data a second, but the volume that the NSA might want to look at could easily choke even the biggest pipe.<\/p>\n
How big then, could the NSA reasonably be? Bigger than Google for sure, with the capacity to perform really simple searches on a significant percentage of the world’s communications. And there’s no reason to believe they’re not doing this. Remember: It’s the NSA’s job to keep an eye on the rest of the world. The only place where they face restrictions is when they listen to people in the US.<\/p>\n