Beyond the scaling problem

Via Ed Brill

URL: From the mailbag: attack of the 49 GB database, redux (cached)

A customer who wishes to be anonymous writes (paraphrased)...
We have sales reps spread throughout the continent.  They have Thinkpads with Notes and they replicate several notes databases via phone or high speed VPN.  The bulk of these databases are a sales force automation program from back in the 4.6 days.  This system is made up of several databases. Some of the databases are fed sales information via Percussion Notrix from our AS/400's.  The 2 largest databases have 600,000 documents (each).  There are several others with varying levels of documents.  I use readers fields so each Sales rep can only see their data, their managers can only see their region etc...   The system works very well for us.  We are not a very large company but are well respected in our industry.

Here's my problem.  We were bought out, and the new company has  "heard" about our sales system and want a demo in a couple of weeks for possible expansion of the system, by up to 2000 users.  I doubt they know much if anything about Notes, so I would anticipate that one of the questions asked will be about scalability.  While our system does work well, we do have issues when dealing with these large databases.  I don't have a good feel for how Notes would do with volumes 10 times greater thenwhat we work with.

Anyway,  I guess I'm just looking for any studies on LARGE databases.
These are nice problems to have in my opinion... trial by fire as it were... Asking the question "will it scale?" though is somewhat beside the point even if you are in the trenches and that is the question that bosses who are breathing down your neck are posing.

The fact of the matter is that very few complex systems can deal with scaling 3 or 4 orders of magnitude and certainly few have been able to handle hypergrowth. Indeed it is only say the Internet (TCP/IP), the web in general, and some of those other applications we know and love (Google, eBay, Amazon) that have been successes and I'd note that all of these have had teething problems (for recent examples see Technorati) despite being well engineered.

I see from the comments that Notes/Domino is being placed in the same company and from my experience of it that sounds reasonable (of course I work for the company that produces it so take my comments with a grain of salt). Like some of those other systems the essential insight is that the system is built on a set of simple primitives at the core of the abstraction (in Notes it is Forms + views +  fields + the nsf file format + replication). I would hope that 15 years of paying attention to the complex systems built upon this foundation should be bearing the fruits of a postive answer to the scaling question.

This is what I've written about Notes/Domino just recently On XForms, XPath, CSS, Brevity, Syntax And More

I see a bright future in which that much maligned Forms "programming model" that is at the core of the Lotus Notes platform could be brought to the web platform leveraging the native primitives of the Web style (hypermedia, uris, linking etc). XForms is singularly well suited to do this. For those unfamiliar with Notes/Domino, my handwaving elevator pitch is that it is a platform essentially founded on the fundamental insight that a huge class of applications can be built based on just a few compositional building blocks: Forms, Views a standard file format, the note in Notes terms. The brouhahas made about messaging, security, directory services, and all that paraphernalia that marketing people throw about when they pitch the platform to you are all syntactic sugar around the core competency of Forms and Views and the client and server processes that can manage them. A whole cottage industry of business partners are doing very fine thank you building custom and evolvable applications for businesses, small and large, everywhere. The fact that email can be construed as a forms application is just a side benefit and detracts from the real focus of the platform. This is much misunderstood by people whose only encounter with Notes is as a Mail client. It's really just a forms and view app for people and processes. Incidentally this same platform is most likely what is funding my current work and much of the IBM Software Group, even as resources are spent on other "sanctioned" and more "strategic" approaches. C'est la vie.
But, the more important question is a productivity one. It's just like GE suddenly discovering that their little pilot deployment of Lotus QuickPlace has blossomed so quickly that their company is running with more than 15,000 QuickPlaces that essentially contain all the wisdom, folklore and institutional memory rather than the unusable collaboration applications that their IT department had put their faith in. They don't want to stop the quick flow of information and sharing that is making their company so productive but at the same time they want things like "manageability", "archiving" and "security" (perhaps for regulatory reasons) or perhaps they are just worried about the number of servers and infrastructure that is needed. I happen to think that Domino handles most of these questions rather well.

But one wonders if  someone couldn't build a quite effective counter-argument about how it is not "sanctioned" technology complete with misleading statistics and handwaving about how it isn't "enterprise-ready".

When companies merge or are taken over it is exactly these kinds of successful applications that sometimes get lost in the mix. IBM is large enough that I have confidence that a sales and support team will work day and night to help the transition. I wonder about some of the banks or smaller companies that are growing.

What I hope isn't obscured in this example and others is the view of those self-same salespeople using the Domino app who are getting on with it, who might complain every now and then that the server is going down too often or who are looking for enhancements since, like all software applications, it is imperfect.  I hope whatever transpires will keep their needs in mind.


Originally posted on: Fri, Apr 29 2005 9:20 AM