Ethan Zuckerman has been pondering generativity and aggregation, prompted by Jonathan Zittrain's paper, The Generative Internet and its implications. Many of these themes have been stewing in my head for quite some time so I thought I'd finally join in the conversation with some of my armchair punditry. As seems to be my custom, a one paragraph comment somehow turned into this note. Sadly I don't quite have a toli code to contribute to the fun, and I've already narrated a couple of gospels recently. Thus I'll switch tack and change the frame. This time I give you a theory: the Low End Theory.
It's interesting that the lawyers, from Lawrence Lessig on, are weighing in on these network things and it's about time... Generativity huh? Zittrain argues a legal case built on abundant economic evidence (and one hopes the linguists would also weigh in). It's a nice restatement of the End-to-End principle in terms palatable to lobbyists. And the argument is much like Lawrence Solum and Minn Chung's paper The Layers Principle: Internet Architecture and the Law from a few years ago.
Reed, Saltzer and Clark described the End-to-End principle quite simply in terms of systems design. The heart of the matter is this oft-overlooked sentence
Functions placed at low levels of a system may be redundant or of little value when compared with the cost of providing them at that low level.They knew when they were writing that this notion had wider applicability than the telecommunication networks that were their initial focus hence they labeled their work "end-to-end arguments in system design". The costs that are borne by the "system" are a generativity tax if we use Zittrain's terminology.
In economic terms, if you read the system as a market, restating this principle turns it into a matter of preserving options value. Black and Scholes have a lot to say on this front.
I see this same notion everywhere in the software engineering that I practice. The most successful system in distributed computing has been the web which, by design (pdf) and to a fault, falls back on minimalist protocols and data formats to handle coordination costs and the human factor.
But there are also tensions at work in the design of any system, and hidden assumptions or vested interests at work.
There will always be a difference between the value of a system to its users and its value to its operators and this is perfectly expressed in networks. At issue in this discussion is who gets to see their utility maximized. As Martin Geddes puts it
If there’s one lesson of the Stupid Network, it’s that there’s a massive increase in consumer surplus. The value to users diverges from that to owners. You can’t measure user value by looking at industry revenue.
A priori you have no idea what options are possible at the edge of your market. This means that the design of the core of your system is a political decision. Like an options trader you are gambling on outcomes in the marketplace and attempting to manage risk. Your design decision is anything but neutral despite what the so-called "network neutrality" proponents say (even if I do agree with their endpoint market outcome).
We know that price discrimination is an optimal pricing strategy from many standpoints and, in market utility terms, it makes sense to optimize system design to enable some modicum of price discrimination. The evidence throughout history, however, is that participants in a market hate price discrimination and favour uniform or predictable pricing. For example, flat rate pricing made the fortunes of AOL, and block pricing is remaking the cell-phone industry. By analogy, Paris Metro pricing is not widespread in transportation systems and on the internet because of these explicit user preferences.
Part of the current challenges the US airline companies face is that we aren't sympathetic to them. One major reason for our disdain is the pricing games that are played with plane tickets. We value fairness even though we are creatures of a rough jungle and there's likely an anthropological basis for our sense of indignation at price discrimination. Perhaps our past ancestors were optimistic planners by nature. However the presence of price discrimination in a market, coupled with flexibility of pricing, and the liquidity of relative transparency allows for the existence of middlemen and the possibility of arbitrage and disintermediation. On the whole, these are good market outcomes but, like many "options", they can't be characterized up front.
Where there are networks, we will find power laws that apply. Similarly where there are markets, we will have sharp elbows and monopolistic temptations as we scramble for the loot. There are echoes of F. Scott-Fitzgerald's notion that "the rich are very different from you and me". These factors are found in every part of the technology industry and, when combined, they play out in the Great Game of consolidation and lobbying. The big network operators will happily oblige in their strategies since lots of advantages accrue to the early movers and great powers and they can leverage consumer inertia. In the internet we have many intermediaries at the network level; firewalls, middleboxes and the domain name system are potential, and actual, Great Powers (see the Verisign Tax ® we all pay and the "offers you can't refuse" that content delivery networks like Akamai make).
Moving up a level, intermediaries are acknowledged up front in the web architecture as design constraints. I have cast the web's embrace of visibility to intermediaries as Caesar's Tax Collector Principle and I think it is fitting, taxes are good, on the whole, and especially estate taxes - they keep the trains running and the streets clean at least in my neck of the woods.
If we take Donald Rumsfeld's taxonomy of knowledge as our starting point for our analysis of networks, we find the following.
Legislators and vested interests tend to worry about "unknown unknowns" and, like Rumsfeld, will focus on threat models and such. This is a matter of governance and regulation; among the core competencies of most government are the management of information and regulation. Excessive focus on these leads almost inevitably to data mining, intimidation, the co-option of a complaisant although ostensibly independent press, spying by the NSA , movie-plot security and the like, your basic Global Wars on Everyone.
The "unknown knowns" are our unconscious biases that frame of the discussion of networks. The metaphors that different sides use are instructive and perhaps even my casting the conversation in terms of "sides" betrays my viewing this arena as a game of sorts (hopefully not Mortal Kombat or something). Others may view the networks scene as a Clash of Civilizations which would lead to apocalyptic wars and the End of History if one follows Manifest Destiny.
The "known knowns" in networks have been visibly demonstrated in the economic activity and verve that is taking place on the internet. Even if the four horsemen of the internet ™ were oversold, we can still point to the ongoing transformation wrought by the Four Horsemen of the Web. As an engineer, I tend to think in terms of protocol, hence my nomination of The HUHXtable Quartet (HTTP, URI, HTML, XML) towards that designation. Your mileage may vary and various prognosticators seem to be weighing in with speculations about the identity of these horsemen. With corporations being legal humans, they have assembled an intriguing cast of characters to liven up the debate.
These same historical forces were at work in the history of communication and transportation networks (pdf) and, although we celebrate the Wright Brothers and Henry Ford, who is really celebrating Samuel Homfray, Richard Trevithick, and the various others who played a role in the history of railroads or even, more recently, the extraordinary innovation brought about by the lowly shipping container?
The sweet spots in our analysis are the "known unknowns". These are the things that make venture capitalists salivate, and monopolists turn to paranoia and worse. It's that seductive notion of the startup in the garage that can change the world, of the eBay or Craigslist economy, of the manufactured serendipity and furious and creative energy that has been on display in the past decade or so on the web. This is where that Long Tail notion comes into play, to shamelessly mine another meme. There is the sense that all you need is an idea, and good connections. The fully leveraged network can be a great leveler and promoter of the innovative forces in a system.
Sidenote: This bit about manufacturing serendipity on the road to riches meshes well with the American Dream ™ and it is worth commenting on a little. Paul Graham, in his hermetically-sealed world, apparently believes that a start-up culture is a particularism of the United States (or condensation as he puts it). What a peculiar notion. While he's sleeping soundly at night, after a lullaby of unshakeable Manifest Destiny, I'm sure that some Teutonic engineering will suddenly emerge (and he misreads the German economy so completely that he's a front runner for the huhudious awards 2006), or would it be an easterly wind that blows in from Korea or thereabouts that will give him fits early in the morning. I would have thought that the lesson was that the unknowns by definition are unknowable and, if you go ahead, following the Rumsfeld example, and discount even the knowns, you end up traveling in a lightly-armored Humvee on the road from Baghdad airport with a convoy of outsourced mercenaries. Good luck on that front.
But back to my topic... I'll quote Martin Geddes again since he is, as ever, eminently quotable
the end-to-end principle is really an appeal to preserve option value in a world of rapid technology change and innovation. By resisting this force, you’re either betting you can suppress rival distribution channels for competing innovation, or you can yourself be a lead innovator.Thus Zittrain's pitch about generativity versus responsibility boils down to a tension between architectures of control and architectures of participation (using a much broader sense of participation than the current buzzword).
Stated another way, and since I traffic in coinages, indulge me if you will with Koranteng's first postulate of networks:
The End-to-End Principle in networks is another incarnation of the Low End Theory.
Recall if you will the core tenets of the Low End Theory which follows the Rule of Four as any good code should. Such is the mantra that schoolgirls the world over are reciting as they study Technology Adoption and System Design 101, and it is worth dwelling on:
As should be clear, each tenet of the Low End Theory encourages externalities to accrue in the system in support of preserving options value. Combined they harness the collective energies of participants and harness innovation. We see these building blocks at work in hardware, in distributed computing and in the various Great Games that are taking place in the technology world. In this respect, I've suggested previously that REST, the web style, was the Low End Theory of Distributed Computing and have also written about The Low End Theory in Hardware. I don't see any reason why networks shouldn't be able to join in the fun, hence my current synthesis. Indeed layer stripping restates the end-to-end notion of overlay systems, intelligence at the ends, and the daily reality of leaky abstractions.
I like that Ethan is shrewdly focusing on the question of aggregation - that is also a restatement of the examples of eBay and Amazon. A good marketplace will surely allow for middlemen and aggregators, that's the bit about favouring participation. Aggregation however is only one of the styles that are likely to prevail in the market. Astute participants can focus on different areas and make hay.
As an example, it pays to identify the disruptions that underlie the evolution of the system, if you can hitch a ride at the right point, you can build mansions in Redmond. Similarly the human factor is a double-edged sword. On the one hand, there is the incoherence of the Tower of Babel which is the bit about coordination costs. On the other hand there are network effects to be gained in communication and group forming that argue for emphasizing community features in the system. From hunter-gatherers on, anything that enabled collaboration has given selective advantages in our evolutionary systems. We want to encourage the sharing of knowledge and information but encounter there are costs. The ongoing dilemma we face about how to build systems recognize the people along with the processes.
Operational efficiency is something that companies like Wal-Mart and Dell are good at, and operational efficiency at the margins is all the rage in the staid banking sector. As the Willie Sutton quipped about robbing banks, it's "Cause that’s where the money is". Thus there are other ways to prevail in this marketplace. Rather than optimizing your processes around innovation you can optimize around operations, billing, search, storage, creating markets, or lubricating whatever friction there is in the system. To take an obvious example, the economics of peering is an interesting problem to tackle. Briefly stated it's Animal Farm all over again: some peers are more equal than others. Or call it real estate: location, location, location.
In networks also we are only beginning to realize the social implications of mobility, of intermittent connectivity, of periodic synchronization and of the fluidity of our communal relations. Where there is the realm of the social, there is the realm of conversation and its corollary, the market. To raise the tenor of this conversation, I'll repeat the insight of those 20th century philosophers, the Pet Shop Boys, and their master treatise:
There's lots of opportunities
You've got the brawn
I've got the brains
Let's make lot of money
Thus there is room in the network game for the innovation of Juniper and Skype and all the companies weighing in on the disruption of the internet protocols, and the operational effectiveness of say equipment manufacturers like Nokia and Samsung, and the more astute network operators. Everyone has their own exemplars on this last front: phone, cable, cell, fiber, copper, fixed wireless, wi-fi, ethernet... The TCP/IP suite that is the core of the internet has succeeded and scaled because the abstractions used embraced transparency and existing systems. The incredible ascendancy of ethernet over three decades (and of late wi-fi) is a testament to the value of simplicity and uniformity in network system design, and the concomitant scale and leverage that the low end mass market provides. Like the personal computer, these technologies are canonical disruptions and those who embrace their Fung Wah Bus aesthetic are inexorably moving up-market.
Some have suggested that good starting points for determining strategy are tomes like The Gorilla Game or The Innovator's Dilemma, I believe the jury is still out and the MBA types can expound on their merit. I would point to Jim Gray's Distributed Computing Economics as my favourite starting point in determining to the best strategy. In any case, the game is on. In the interim, we are simply picking and choosing which of the hard problems we want to address. Still game theory remains theory; in real life games we have that nagging human factor.
Engineers are currently hobbled in our advocacy of the internet because we don't have good instruments for determining metrics for things like resilience and adaptability that could inform the decisions of policy makers, lawyers, economists, and those who really matter: Mr. and Mrs. Big Money. Politicians everywhere like statistics and large numbers in what passes for their policy debates, especially in this current silly season of nostalgia. We do have large numbers in the internet, the users, but unfortunately we only have waffly options pricing to throw around as statistics. I would hazard that muttering "options value pricing for networks" doesn't quite cut it when compared to the red meat of the corn and sugar lobbies or the pork of the military industrial complex.
One needs to dangle some glitter when we talk to congressmen, or votes or something. Thus I'm an alchemist searching for Black Gold. They told me it was a goldfield; it turns out that it's an abandoned coal mine in Pennsylvania, a dud concession. Still I've heard there are good things Out West on the frontier, that area called the internet, that web style I need to adopt. Thus the Low End Theory is a work in progress and one could easily get taken to the cleaners by the loaded dice of the other players.
Everybody wants their cut and, if you can tilt the playing field at whatever level you reside at in the network ecosystem, it makes sense to do so, especially if you are judged every quarter by the baying hounds of Wall Street. If you take the market view, the lobbying capabilities of large corporations, a long history of regulation in this sector, and governments' existential need to pass laws or exercise control or compliance of some sort (whether in the name of security, expedience or getting things done) are going to weigh heavily.
We engineers have been lucky to have built enough of the internet and web infrastructure before the lawyers, lobbyists and governments realized what was there. The underlying architecture has proven sound enough to have scaled several orders of magnitude and looks set for several more as the next few billions of humanity come on board. The fact that the internet (after three decades) and the web (in its second decade) are now treated as infrastructure says it all, and this is a Good Thing ®.
The language is also important, and I continue to wince at George Gilder's revolutionary talk although he is now (slightly) chastened. My reading is that we need to emphasize a sedate, gray-suited discourse to keep the bankers and traders happy and the loud "content" industry at bay. Infrastructure should be boring despite the breathless prose we see in the business rags. Technology was made prematurely sexy in the dot com era; the implications of technology are what are important, not the technology itself. The markets for cement and most other commodities don't raise eyebrows and nor should networks. The logical lesson of the end-to-end principle is that communication networks are about connectivity, everything else is gravy.
I'll suggest then that normalcy is what engineers should aim for in this discussion. The Generative Internet is a good contribution to the debate as it is rooted in an understanding of the engineering insight of the end-to-end principle and the way platforms are developed and evolve. The dissection of the personal computer industry is instructive too as an argument by analogy in the technology world. The exploitation of Moore's Law and the black gold of silicon, the sweet spot of the mass-market where Intel and now AMD have been able to make hay, the platform effects that Microsoft has been able to leverage in the mass market etc., all these continue to drive that Great Game.
The weakest part of Zittrain's paper is where he declares that the end-to-end argument needs to be superseded and that "end-to-end does not fully capture the overall project of maintaining generativity". The primary reason for this postulate is the rhetorical strawman he constructs of end-to-end neutrality. Those advocating what they call "network neutrality" are simply being cute. They are implying, for quite pragmatic and rhetorical reasons (read effectiveness of lobbying), that there are no costs to neutrality; that neutrality is value neutral. It's a nice trick in framing a debate but it is a trick nevertheless.
Odlyzko has noted that spending on communication services (and especially the connectivity component) is huge, dwarfing many other segments of the economy (and before Rumsfeld's folly it was even trending towards the amount spent on national defense in the United States). Those vested interests and a $300 billion dollar sector will buy lots of slick rhetoric and astute framing. Zittrain carries on to suggest that the inevitable endpoint of the end-to-end argument is a network teeming with, on the one hand, viruses and spam and, on the other, silos and walled gardens. Having raised the spectre of these undoubted bogeymen that obviously need regulation, we then require new paradigms and frameworks.
This is either misguided or flawed, depending on where you stand. The end-to-end argument, as restated in the low-end theory, stands up well to these charges. It is an argument about lowering coordination costs. Nowhere is it acknowledged in the original paper that there are no costs to be borne, or that there is any such thing as end-to-end neutrality. To take a concrete example of engineering expediency at work consider the implementation of congestion control in the TCP/IP suite of protocols. This is characterized by some purists as a layering hack. The upshot of current practice is that we have repeated injunctions for other network applications to be "TCP friendly" in order to preserve the commons and the kind of congestion collapse that was observed as the internet began to see increased usage in the 1980s. We can and do embed functions inside of the network systems that we develop and sometimes we even cross layers if necessary, this is simply pragmatism at work.
Thus the very premise of the end-to-end argument is that this is a matter of tradeoffs and decisions about who bears costs. Engineering decisions are never neutral, the low end theory is political and is competing with different styles in the marketplace. Its emphasis on favouring participation over control simply aims to tilt the marketplace in a direction that encourages externalities to accrue to the end rather than the core. Similarly layer stripping as a design principle in the core is about reducing complexity. In other words, it suggests a strategy for those who run the network about how they reduce their operating costs and accrue their value in the marketplace.
Lastly I'll note that everybody has to deal with gremlins and parasites and, as Cory Doctorow has noted, "all complex ecosystems have parasites"; they are transaction costs in every marketplace. Things always fall off the back of a lorry, the banking sector tolerates a certain level of fraud etc. Further, these transactions costs are acknowledged upfront in the end-to-end argument. Also, the empirical evidence throughout human history is that silos and walled gardens (from the Berlin Wall to CompuServe) are unsustainable in the long run and that it is shrewd to bet on participation over control (and I hope Burma and North Korea don't give the lie to my optimistic outlook). Still this prognostication is only a small part of Zittrain's remit and perhaps detracts from his wider argument.
I'll acknowledge here that linguistically the generative internet is a good coinage, and perhaps it even works better than End-to-End when pitched to the average congressman. This suggests to me that the enduring value of Zittrain's analysis is likely to be in the branding of the debate. Still as the lawyers and economists gear up and build up their legal frameworks and paraphernalia of economic models to throw at us about the design of networks, we engineers should confront them with prosaic notions of building communities and enabling conversations and marketplaces. I hope the Low End Theory can similarly enrich our vocabulary in this light.
We do have secret weapons in this debate... My grandmother is a surprisingly fierce and effective lobbyist when she puts her mind to it, on topics that sometimes mystify me. Her desire to interact with her progeny and to get on the web to converse with us should not be underestimated. The evidence is clear that she'll even tolerate any amount of spam and the ever-present gremlins and parasites that prevail - in moderation of course. More generally there is the desire to reconnect with old friends (and perhaps old flames), the socialization impulse that lurks in all of us. This is why I try at every opportunity to advocate pragmatism and that we strive to build Sexy Mom Factor Software. Where soccer moms are the demographic the politicians go after, in networks we need to encourage the Grandma Lobby like those cunning financial folks who went after Scottish Widows.
To conclude, participation is winning out over control for the moment, but it is the eternal struggle and I am hoping that the current ascendancy on the internet is not a temporary respite. As with democracy in the Great Game of Politics, eternal vigilance must be our watchword. And to paraphrase he of blood, sweat and tears
The story is the same as it ever was in the Great Game of Networks and I argue here that the Low End Theory is King, or President, if like me you are a republican (with the lowercase r). I've decided that I like the freedom I've found living within the constraints of a series hence I'll cast this note as part 1 of The Great Game of Networks Series which itself is an offshoot of The Great Game of Technology Series which I hereby retroactively announce. The latter series started last year with some musings on The Great Game of Chips.
The Low End Theory soundtrack once again comes courtesy of A Tribe Called Quest's 1991 album.End-to-End systems, or Stupid Networks if you like, are the worst form of networks except for all those others that have been tried.
Postscript
Soundtrack
This time we'll hum along to Jazz (We've Got) and nod our head's to Ron Carter's bass:
Next: Disruptions in Networks. Ergo, what's there to leverage?Stern firm and young with a laid-back tongue
The aim is to succeed and achieve at the age of 21...[The internet hit the prime time around the age of 21 with the arrival of the web]
So push it, along, trails, we blaze
Don't deserve the gong, don't deserve the praise...[innovation and manufactured serendipity continues]
A brand new twist with the homie-alistic
So low-key that ya probably missed it...[low end should be low key infrastructure]
Competition, dem Phifer come sideway
But competition, dem must come straightway[violators need not apply, we can see you coming]
I take off my hat to other crews that intend to rock
But the Low End Theory's here, it's time to wreck shopWe've got the Jazz (x4)
So peace to that crew, and peace to this crew
Bring on the tour, we'll see you at a theatre nearest you