Content by Cory Doctorow (best free novels txt) 📖

to be sure that everyone was pulling their weight and not, e.g., napping in the treetops instead of watching for predators, emerging only to eat the fruit the rest of us have foraged.

Keeping track of our social relationships is a serious piece of work that runs a heavy cognitive load. It's natural to seek out some neural prosthesis for assistance in this chore. My fiancee once proposed a "social scheduling" application that would watch your phone and email and IM to figure out who your pals were and give you a little alert if too much time passed without your reaching out to say hello and keep the coals of your relationship aglow. By the time you've reached your forties, chances are you're out-of-touch with more friends than you're in-touch with, old summer-camp chums, high-school mates, ex-spouses and their families, former co-workers, college roomies, dot-com veterans... Getting all those people back into your life is a full-time job and then some.

You'd think that Facebook would be the perfect tool for handling all this. It's not. For every long-lost chum who reaches out to me on Facebook, there's a guy who beat me up on a weekly basis through the whole seventh grade but now wants to be my buddy; or the crazy person who was fun in college but is now kind of sad; or the creepy ex-co-worker who I'd cross the street to avoid but who now wants to know, "Am I your friend?" yes or no, this instant, please.

It's not just Facebook and it's not just me. Every "social networking service" has had this problem and every user I've spoken to has been frustrated by it. I think that's why these services are so volatile: why we're so willing to flee from Friendster and into MySpace's loving arms; from MySpace to Facebook. It's socially awkward to refuse to add someone to your friends list -- but *removing* someone from your friend-list is practically a declaration of war. The least-awkward way to get back to a friends list with nothing but friends on it is to reboot: create a new identity on a new system and send out some invites (of course, chances are at least one of those invites will go to someone who'll groan and wonder why we're dumb enough to think that we're pals).

That's why I don't worry about Facebook taking over the net. As more users flock to it, the chances that the person who precipitates your exodus will find you increases. Once that happens, poof, away you go -- and Facebook joins SixDegrees, Friendster and their pals on the scrapheap of net.history.

$$$$

The Future of Internet Immune Systems

(Originally published on InformationWeek's Internet Evolution, November 19, 2007)

Bunhill Cemetery is just down the road from my flat in London. It’s a handsome old boneyard, a former plague pit (“Bone hill” -- as in, there are so many bones under there that the ground is actually kind of humped up into a hill). There are plenty of luminaries buried there -- John “Pilgrim’s Progress” Bunyan, William Blake, Daniel Defoe, and assorted Cromwells. But my favorite tomb is that of Thomas Bayes, the 18th-century statistician for whom Bayesian filtering is named.

Bayesian filtering is plenty useful. Here’s a simple example of how you might use a Bayesian filter. First, get a giant load of non-spam emails and feed them into a Bayesian program that counts how many times each word in their vocabulary appears, producing a statistical breakdown of the word-frequency in good emails.

Then, point the filter at a giant load of spam (if you’re having a hard time getting a hold of one, I have plenty to spare), and count the words in it. Now, for each new message that arrives in your inbox, have the filter count the relative word-frequencies and make a statistical prediction about whether the new message is spam or not (there are plenty of wrinkles in this formula, but this is the general idea).

The beauty of this approach is that you needn’t dream up “The Big Exhaustive List of Words and Phrases That Indicate a Message Is/Is Not Spam.” The filter naively calculates a statistical fingerprint for spam and not-spam, and checks the new messages against them.

This approach -- and similar ones -- are evolving into an immune system for the Internet, and like all immune systems, a little bit goes a long way, and too much makes you break out in hives.

ISPs are loading up their network centers with intrusion detection systems and tripwires that are supposed to stop attacks before they happen. For example, there’s the filter at the hotel I once stayed at in Jacksonville, Fla. Five minutes after I logged in, the network locked me out again. After an hour on the phone with tech support, it transpired that the network had noticed that the videogame I was playing systematically polled the other hosts on the network to check if they were running servers that I could join and play on. The network decided that this was a malicious port-scan and that it had better kick me off before I did anything naughty.

It only took five minutes for the software to lock me out, but it took well over an hour to find someone in tech support who understood what had happened and could reset the router so that I could get back online.

And right there is an example of the autoimmune disorder. Our network defenses are automated, instantaneous, and sweeping. But our fallback and oversight systems are slow, understaffed, and unresponsive. It takes a millionth of a second for the Transportation Security Administration’s body-cavity-search roulette wheel to decide that you’re a potential terrorist and stick you on a no-fly list, but getting un-Tuttle-Buttled is a nightmarish, months-long procedure that makes Orwell look like an optimist.

The tripwire that locks you out was fired-and-forgotten two years ago by an anonymous sysadmin with root access on the whole network. The outsourced help-desk schlub who unlocks your account can’t even spell "tripwire." The same goes for the algorithm that cut off your credit card because you got on an airplane to a different part of the world and then had the audacity to spend your money. (I’ve resigned myself to spending $50 on long-distance calls with Citibank every time I cross a border if I want to use my debit card while abroad.)

This problem exists in macro- and microcosm across the whole of our technologically mediated society. The “spamigation bots” run by the Business Software Alliance and the Music and Film Industry Association of America (MAFIAA) entertainment groups send out tens of thousands of automated copyright takedown notices to ISPs at a cost of pennies, with little or no human oversight. The people who get erroneously fingered as pirates (as a Recording Industry Association of America (RIAA) spokesperson charmingly puts it, “When you go fishing with a dragnet, sometimes you catch a dolphin.”) spend days or weeks convincing their ISPs that they had the right to post their videos, music, and text-files.

We need an immune system. There are plenty of bad guys out there, and technology gives them force-multipliers (like the hackers who run 250,000-PC botnets). Still, there’s a terrible asymmetry in a world where defensive takedowns are automatic, but correcting mistaken takedowns is done by hand.

$$$$

All Complex Ecosystems Have Parasites

(Paper delivered at the O'Reilly Emerging Technology Conference, San Diego, California, 16 March 2005)

AOL hates spam. AOL could eliminate nearly 100 percent of its subscribers' spam with one easy change: it could simply shut off its internet gateway. Then, as of yore, the only email an AOL subscriber could receive would come from another AOL subscriber. If an AOL subscriber sent a spam to another AOL subscriber and AOL found out about it, they could terminate the spammer's account. Spam costs AOL millions, and represents a substantial disincentive for AOL customers to remain with the service, and yet AOL chooses to permit virtually anyone who can connect to the Internet, anywhere in the world, to send email to its customers, with any software at all.

Email is a sloppy, complicated ecosystem. It has organisms of sufficient diversity and sheer number as to beggar the imagination: thousands of SMTP agents, millions of mail-servers, hundreds of millions of users. That richness and diversity lets all kinds of innovative stuff happen: if you go to nytimes.com and "send a story to a friend," the NYT can convincingly spoof your return address on the email it sends to your friend, so that it appears that the email originated on your computer. Also: a spammer can harvest your email and use it as a fake return address on the spam he sends to your friend. Sysadmins have server processes that send them mail to secret pager-addresses when something goes wrong, and GPLed mailing-list software gets used by spammers and people running high-volume mailing lists alike.

You could stop spam by simplifying email: centralize functions like identity verification, limit the number of authorized mail agents and refuse service to unauthorized agents, even set up tollbooths where small sums of money are collected for every email, ensuring that sending ten million messages was too expensive to contemplate without a damned high expectation of return on investment. If you did all these things, you'd solve spam.

By breaking email.

Small server processes that mail a logfile to five sysadmins every hour just in case would be prohibitively expensive. Convincing the soviet that your bulk-mailer was only useful to legit mailing lists and not spammers could take months, and there's no guarantee that it would get their stamp of approval at all. With verified identity, the NYTimes couldn't impersonate you when it forwarded stories on your behalf -- and Chinese dissidents couldn't send out their samizdata via disposable gmail accounts.

An email system that can be controlled is an email system without complexity. Complex ecosystems are influenced, not controlled.

The Hollywood studios are conniving to create a global network of regulatory mandates over entertainment devices. Here they call it the Broadcast Flag; in Europe, Asia, Australia and Latinamerica it's called DVB Copy Protection Content Management. These systems purport to solve the problem of indiscriminate redistribution of broadcast programming via the Internet, but their answer to the problem, such as it is, is to require that everyone who wants to build a device that touches video has to first get permission.

If you want to make a TV, a screen, a video-card, a high-speed bus, an analog-to-digital converter, a tuner card, a DVD burner -- any tool that you hope to be lawful for use in connection with digital TV signals -- you'll have to go on bended knee to get permission to deploy it. You'll have to convince FCC bureaucrats or a panel of Hollywood companies and their sellout IT and consumer electronics toadies that the thing you're going to bring to market will not disrupt their business models.

That's how DVD works today: if you want to make a DVD player, you need to ask permission from a shadowy organization called the DVD-CCA. They don't give permission if you plan on adding new features -- that's why they're suing Kaleidascape for building a DVD jukebox that can play back your movies from a hard-drive archive instead of the original discs.

CD has a rich ecosystem, filled with parasites -- entrepreneurial organisms that move to fill every available niche. If you spent a thousand bucks on CDs ten years ago, the ecosystem for CDs would reward you handsomely. In the intervening decade, parasites who have found an