I Want a New Platform
One that won’t go away (under load), one that doesn’t keep me up all night (worrying about scaling), one that won’t make me sleep all day (when I should be adding features). If you have tried to build an Internet site or application recently that needs to work for thousands or tens of thousands of concurrent users you may share this desire.
Why is this still so hard? Why do we find ourselves worrying about locating experts in the dark art of database performance tuning? Why are we spending time haggling with Rackspace over the price for another set of servers or racking our own servers at Equinix? Why are we writing our own user management (from scratch)?
Because we are using tools that simply were not made for the job. Relational databases have gotten faster and better, but their fundamental construct of data stored in rows and neatly parceled out across tables (which frequently need to be joined) does not really match up with either rapid development using some notion of objects or with scaling horizontally using commodity hardware. Web servers started out by simply providing static content and were then forced into running applications with the result of not doing either particularly well. Yes, people have been writing new ones to address that, but even those are fundamentally designed to work on a single machine. The second you go beyond one machine you need separate load balancers, reverse proxies, caches and all sorts of other paraphernalia just to make stuff work together. Worse yet, the web server, the application and the database are connected to each other via thin straws that were bolted on after the fact with important information either not passed at all or only painfully (e.g., information about which user is requesting a particular set of data).
What about the growing cast of frameworks, like Ruby on Rails, CakePHP, Django, etc? Don’t they provide a solution to these issues? Not really. These frameworks are for the most part about enabling rapid development. They do great on the add features rapidly front but often make matters worse on the scaling front. The reason is that they layer more stuff on the very tools that are wrong for the job to begin with.
What then is needed? A platform that is created from the ground up to enable modern web sites and applications. What would such a platform look like? It would be hosted and (nearly) infinitely scaleable. It would provide object storage that’s as simple as saying “here’s an object, store it” (you get back a handle, ideally, if you want a human readable, search engine optimized one). Later on, retrieval should be something like – “here’s a handle, give me back the object” (with full user level access control baked in). Stored stuff should be easily indexed so that one could say “give me back all the handles for objects that match this pattern” (and to which the user has access). The same should work for media: “Here’s a picture, store it for me with this metadata” and “Find all the pictures for me tagged x.”
Aside from storage there are other useful services the new platform should come with, since essentially every modern web site / application needs them, such user authentication, authorization and access control. Flexible processing of pretty URLs. Easy creation and maintenance of page templates. Ability to send emails and process bounces. Handling of RSS feeds (inbound and outbound). Support for mobile access and possibly even voice capabilities.
Code would run inside the platform (this is what Marc Andreessen calls a Level 3 platform) but it would not be cut off from the outside world. It would have full access to other services that live on the net via web services. So, for instance, there would be no need for the platform to have its own payment service.
Let me try to head off some potential objections.
Can such a platform be built at all? Won’t there be problems with distributed transactions (or some other technical objection)? Nobody said it would be easy (besides, if it were it wouldn’t be half the fun or represent a big opportunity). But there is no fundamental impossibility here, especially in a hosted environment and keeping in mind that the goal is not to create a platform on which anything can scale automagically, just a large set of web sites and applications that are currently hand crafted. For instance, the object storage magic does not have to work for arbitrarily complex objects. My guess is that the vast majority of sites and applications can live with (fairly) shallow objects based on just a few underlying types.
Isn’t this just a fancy way of saying you want a hosted object-oriented database and didn’t those fail miserably as software products? Just because stuff failed in the past doesn’t mean it won’t work now, especially when circumstances change (the current ugly picture in the credit markets is a reminder that the opposite is equally true).
Won’t folks be completely locked into this platform? We are not really worried about being locked into MySQL, because we feel that if push came to shove, if we really had to, we could switch to say Postgres or DB2 or Oracle. This of course is a lot harder to do in practice because in the quest for performance we often resort to using custom SQL extensions that are specific to our current DB. But it does point to standards as an important way to mitigate lock in and anybody providing a platform like this would do well by working with existing standards where available and promoting new ones where not (e.g. an object storage standard). A more dramatic way to mitigate lock in might be to offer an open source “community” version of the platform and compete based on hosting the most reliable version.
Hasn’t Amazon has already built this? Isn’t this S3 and EC2? S3 and EC2 and the upcoming database service are steps in the right direction. But they are not nearly as easy to program as the “new” platform (in particular EC2 is really just high speed provisioning of traditional servers). Amazon could, however, use their pieces to offer something like the new platform.
I am sure there are other objections (look forward to the comments), but those aside, where might this new platform come from? Marc Andreessen lists companies who are working on it now, including his own Ning, Salesforce and Amazon. I would not be surprised if folks at google were also thinking about offering something. Based on their recent announcements, Salesforce seems furthest along in offering a new platform but they have made many idiosyncratic choices which result in high lock in (not to mention that they run an applications business which might compete). Despite the big guns already working on this, I believe there is still an opportunity for someone new to build the new platform. It’s the kind of disruptive change that requires throwing overboard a lot of existing assumptions and conventional wisdom and trying something radically different. It will also require maniacal focus and independence from the needs and influences of existing operations. Sounds like a startup to me.
UPDATE (December 16, 2007): Amazon has taken the wraps off SimpleDB and it looks promising. It's schema free and appears very reasonably priced. If the performance is good this will bring Amazon a big step closer.
September 19, 2007 12:26 PM, By Albert Wenger
Tags: innovation platform scalability startup
Comments (39)
I want to have your high-class problems. It all sounds good, but I'm also partial to the theory that you shouldn't worry about scaling until you need to worry about scaling. 99% of projects will never see sufficient scale to worry about this, so chances are if your startup (or project) is taking a lot of time to think about this -- it's wasted. You could be building features or delivering faster.
But of course, most of my projects appeal to approximately 1-2 users, so I could be wrong.
Posted by Joe , September 19, 2007 01:44 PM
Albert, very nice article. These are things I think about all the time. Though I'm still working on getting that massive audience of users. Anyway, some of my favorite technologies are coming out of Amazon. If you listen to some of the engineers who have worked at Amazon, they give hints about the kinds of techniques they use. My new favorite is not one of their web services. I learned of it from Werner Vogels' blog. It's called Carbonado. http://www.allthingsdistributed.com/2006/10/carbonado.html
I've started to work with it and I love the simplicity and power. I like to describe it to others as Hibernate for BerkeleyDB. Which is an embarrassingly inaccurate description, since it does not really require BDB and it's not as complicated as Hibernate. I'm also working on massive scalability in another project that uses a pretty interesting solution.
If you have time, perhaps we can talk about this more.
Posted by Hugh , September 19, 2007 02:04 PM
Interesting post Albert and something I've also been thinking about. I'd break your into two topics: application level features (AAA security, RSS hooks, mail handling, pretty URLs, etc.) and high-performance data access.
There are many frameworks that can help with the former. Rails is just one such where many of the elements are either in the architecture or are easily pluggable.
This leaves the latter concern. It seems like any site that I've worked on that needs to scale big comes across this concern. And I see a lot of re-inventing the wheel in solving this problem (e.g denormalized object caches of relational data). The problem definitely occurs when we want the data to be accessed in a flexible manner (i.e SQL queries).
I don't know what the answer is but I do agree that we need a flexible, queryable object layer between the database and the application. It is a tough and very recurring problem. However, I'm convinced that there is a solution to this. I also feel the hosting of this solution should be an optional add-on. My bet's on Amazon getting a real crack at this first by adding to the EC2, S3 stack.
Posted by Vivek , September 19, 2007 02:58 PM
Have you checked the CouchDB?
http://couchdb.org
Posted by Berislav Lopac , September 19, 2007 06:22 PM
2nd on CouchDb.
Also- check out Hadoop. Open Source MapReduce funded by Yahoo, led by Doug Cutting.
Posted by matt m , September 19, 2007 11:24 PM
Not sure if this is really a technology problem, or a labor/education problem. It's hard to hire people with deep knowledge in these areas because studying engineering has been on the decline for years. College students want to study finance, econ, and government - who cares about engineering, math, or science, to say nothing of the liberal arts like sociology or literature.
Some things are just hard. We as a race have created some very complex and difficult problems, and it's going to take a lot of brainpower to solve them. Would that this set of problems really mattered to our survivual as a species.
You're haggling with Rackspace because businesses are focused on the short-term bottom line instead of long-term relationships. You're writing user management from scratch because Open Source methods are great for solving deterministic problems like kernal performance and not evolved enough to apply to human-system interaction cases. And it's hard to find expert database gurus because the work is difficult to learn, geeks are marginalized in primary and secondary education, and when people look at their career options they decide they'd rather learn how to lobby the government (i.e talking and going out to lunch and parties) than learn how to solve intellectually interesting set theory problems (i.e. sit quietly and think).
Posted by Michael J. , September 20, 2007 08:45 AM
I really couldn't agree more with you, Albert. The more projects I've worked on in the past few months, the more I've been frustrated with how long it takes to get it from ground level to the first floor, for the reasons you outlined above. If there were a platform by which you start at floor 1 (writing business logic), and didn't have to think about any of those components below that, development time for the life of the project would be drastically shorter and more productive.
In a lot of ways it's the next layer on top of the stack, I guess. Imagine writing a website using machine language. It got so much easier when we stacked on C and interpreters and frameworks. Now we're at the point where 80% of the stuff that people need to develop on the web looks exactly alike from the feet to the waist. I'd love to be able to start at that point and work my way up.
I've got an hour a week to spare (if I don't sleep), let's see what I can do... :)
Posted by Philip Cotty , September 20, 2007 09:30 AM
If you (or someone else) builds it, I will be one of your first customers.
Posted by Dave Winer , September 20, 2007 10:00 AM
I think in the beginning there will be specialties. Some companies will provide login services, others search, others deal with RSS feeds, etc. Individually each scalable and easily added together to create your own sites and applications and provided as services. You already see this happening. I wish I could go to EC2 to get my servers and have them automatically instantiate when the load increases without me having to do anything special (look at rightscale.com to see how someone is starting to provide this as a specialized service) but Amazon doesn't support it yet and even if they did would they have the resources to support the next thing? I think not. I think multiple platforms will exist that together will compromise an uber-platform and the competition will be to see who will provide the specialized services. The winners will get the spoils. I think believing that one company will provide it all sticks with the walled garden approach and is doomed for failure.
Doug K.
Posted by Doug Kersten , September 20, 2007 10:34 AM
I don’t think this is a business I’d like to be in. The sales effort to get into the kind of people who need this kind of thing would be massive, and at those places you’d be competing not only with enormous juggernauts like Oracle, etc., and not only a huge range of free open source software, but also with internal systems that they’d developed themselves and many would want to protect.
On top of this, it would be really hard to build.
I’ve had a few experiences with “rebuilding from the ground up” to make a better system, and there are limited contexts in which it can be profitable.
Posted by Rpezman24 , September 20, 2007 10:46 AM
What about ZOPE? They have an object store, which backends onto relational databases if you like. It does rollback even.
They run python internally, in a way that makes it a native thing to do. They also do rapid feature development, with full user-access...
Posted by Dave Aitel , September 20, 2007 10:48 AM
The big problem that you wind up having is trying to find the balance between scalability/security/robustness/flexibility/ease of use. Companies like Ning or Coghead are trying to create platforms that are easy to develop on and deploy, but may be limiting in the functionality that is actually available to the developer. EC2 and S3 give the developer/operator a lot more freedom in what they can do, but there is a much more significant technical hurdle to understanding how to effectively build.
It is a daunting challenge to develop a technology that fits in the middle. For example, look at the problems programming language designers have had in designing an easy to use language that has the power to solve advanced problems. In other words, how do you create a language accessible to VisualBasic programmers but powerful enough for C++/Lisp hackers. C# has probably done the best job of this, but as the language matures and continues to add features, it gets more and more complex for the beginner to get started.
Trying to build a platform that is simple to develop in, has good supported features like authentication and mail built in (but still extensible so that I can handle when I have my own custom requirements on how people should log in) plus has an infrastructure that will scale from when I first launch to friends and family all the way to when I am getting millions of hits an hour is probably even harder than building an accessible but powerful programming language.
Somewhat related to this topic is an interesting article about scaling at Amazon.com is available from http://highscalability.com/amazon-architecture. One interesting point is
"Build your own infrastructure for performance, reliability, and cost control reasons. By building it yourself you never have to say you went down because it was company X's fault. Your software may not be more reliable than others, but you can fix, debug, and deployment much quicker than when working with a 3rd party."
Seems to go against relying on Amazon for S3 and EC2 as a long-term strategy :)
Posted by Rob Di Marco , September 20, 2007 10:54 AM
Fantastic post. I couldn't agree more:
http://smoothspan.wordpress.com/2007/09/20/we-are-using-tools-that-simply-were-not-made-for-the-job/
The platform you describe is not in sight at the moment. Salesforce doesn't have it either:
http://smoothspan.wordpress.com/2007/09/19/the-ugly-truth-about-the-force-platform-salesforces-is-no-better-than-saps-or-oracles/
Posted by Bob Warfield , September 20, 2007 11:14 AM
Amazon's S3/EC2/SQS is definitely laying the foundation for something like this - once they come out with a persistent/structured data storage solution, it wouldn't be a quantum leap in thought to add automated clustering, performance tuning, backup/fail over support etc. to the mix.
However, I don't think Amazon (or whoever brings this "killer app" to market) should focus on being a software platform provider. If I've learned anything after owning, operating and investing in technology companies over the years, it's that specialization of labor is extremely important. Separating the infrastructure from the software will allow Amazon (or whoever) to focus on one thing and allow room for framework, programming language and OS providers to innovate and compete on a stable infrastructure platform - think Windows (and possible Facebook).
-"Here's my ten cents, my two cents is free..."
Posted by Wayne Mulligan , September 20, 2007 03:37 PM
The core of this issue is that we're in a **PARALLEL COMPUTING ENVIRONMENT**
That's really the issue. You're planning on using multiple CPUs to do the work, and the architecture bottlenecks at your ability to get the centralized components of the architecture to behave.
Gee... didn't the supercomputer guys have this problem about 30 years ago?
A single system, serving hundreds or tens of thousands of concurrent users is going to have these problems. The more interactive it becomes, the more **LIKE SOFTWARE** your web site is, the worse the problem will be.
You think you have problems? Check out:
http://disastr.org
and imagine this problem in the context of sixty million people hitting your system in a three hour window to try and figure out how to save themselves from a disaster.
Can it be done? Sure. But the STORM botnet is a your model, not your current web applications framework.
Start thinking of swarms of disposable entities serving your web sites. It's not about taking a single webserver app and scaling it no no no. It's about writing parallel supercomputer cluster software that happens to be user-interactive rather than number crunching.
Nodes that carry data. Migration over the network to spare compute resources. Local session data that's spooled to disk replicated on the fly rather than databases.
SQL *on the node* if you need it (think SQLite) rather than an Oracle install.
It's not easy: people in the supercomputer world have been worrying about this since the 1980s, maybe earlier. But that's what you're up against here: supercomputer sized computations with irregular datasets and complex interdependencies.
If you haven't thought about this stuff, you need to go back to the Transputer papers and the Occam programming language. Occam is really the best single source for clear thinking about parallel computation as a fundamental part of software design that I am aware of, and the Transputer is an idea who's time is not yet come - first we need parallel programmers, *then* parallel computers.
Once you've got your head around that - and things like the "Crazy Postman" algorithm and why they worked in a transputer environment - then you're ready to get down to writing code for these massive new generation applications.
Posted by Vinay Gupta , September 20, 2007 06:07 PM
Try building one yourself and you will know the answer to "Why is this so hard?". This is the age of specialization. Anything short of a laser focus on a narrow target is a recipe for disaster. That's why. I understand that for you as an investor it's a lottery with a big enough reward, but for the team you would back, the risk ain't worth it.
Posted by prakster , September 20, 2007 06:43 PM
I just emailed Fred about something like this yesterday! We are having trouble scaling our network on EC3. Although Amazon has set up a cool, unique computing in the clouds, it still takes set up and a lot of code implemented to get going. When our traffic is increasing, we just keep adding database servers and webservers when there could likely be a bottleneck somewhere.
Posted by Daniel Carroll , September 20, 2007 07:32 PM
Sounds like you want Erlang.
Posted by David Mathers , September 20, 2007 07:36 PM
Existing limits are frustrating, yes.
There are aspects of this distributed operating system model across the existing environment that could be applied, whether it is Core Data storage or Erlang processing, but you first have a huge effort building this and then you have to compete in a market that often expects Linux or Windows pricing.
This project scale involves a substantial staff of operating system engineers, networking, security, and related specialties and several years elapsed work before you even have a product beta on targeted platforms available, and yet more years before you're established in the market. Even if you can reasonably start the project with an existing kernel. And how much application and API compatibility is expected and feasible, and how much you'll need to break.
Back of the envelope... Figure thirty million dollars a year for the burn rate (figure $150K times 100 folks, plus iron, plus products, plus overhead, plus marketing, plus lawyers; all probably too conservative), and with no chance of a return on this investment for multiple years — and realize you're well up in the enterprise market here as this product configuration scales upwards, and enterprise buyers can tend to be skittish and superstitious and conservative lot. In this end of the market, concepts like grids can appear scary. Sales cycles can be a year or more. And at the low end, you're competing with MySQL and Linux solutions. And with the folks that don't think grids are scary.
It would be an absolute blast to work on this project; to eliminate the bottlenecks of existing operating systems and databases. Google or Amazon scale. Given few organizations and yet fewer individuals have disposable income and a horizon of this scale, figuring out how to fund this and to then sell this product and its related services at a profit, and how to get communities and businesses invested in the platform, now that's the real challenge here.
Sign me up.
Posted by Mr. Hoffman , September 20, 2007 09:36 PM
Existing limits are frustrating, yes.
There are aspects of this distributed operating system model across the existing environment that could be applied, whether it is Core Data storage or Erlang processing, but you first have a huge effort building this and then you have to compete in a market that often expects Linux or Windows pricing.
This project scale involves a substantial staff of operating system engineers, networking, security, and related specialties and several years elapsed work before you even have a product beta on targeted platforms available, and yet more years before you're established in the market. Even if you can reasonably start the project with an existing kernel. And how much application and API compatibility is expected and feasible, and how much you'll need to break.
Back of the envelope... Figure thirty million dollars a year for the burn rate (figure $150K times 100 folks, plus iron, plus products, plus overhead, plus marketing, plus lawyers; all probably too conservative), and with no chance of a return on this investment for multiple years — and realize you're well up in the enterprise market here as this product configuration scales upwards, and enterprise buyers can tend to be skittish and superstitious and conservative lot. In this end of the market, concepts like grids can appear scary. Sales cycles can be a year or more. And at the low end, you're competing with MySQL and Linux solutions. And with the folks that don't think grids are scary.
It would be an absolute blast to work on this project; to eliminate the bottlenecks of existing operating systems and databases. Google or Amazon scale. Given few organizations and yet fewer individuals have disposable income and a horizon of this scale, figuring out how to fund this and to then sell this product and its related services at a profit, and how to get communities and businesses invested in the platform, now that's the real challenge here.
Sign me up.
Posted by Mr. Hoffman , September 20, 2007 09:40 PM
Albert, the problem is there is no one-size fits all solution.
You can already host your app on "Grids" like EngineYard, mediatemple, LayeredTech's gridlayer, etc.
Some apps will be heavily database-centric (or object-centric, if they were to be implemented in an object-centric data storage way).
Other apps will be heavily app-centric, requiring lots & lots of CPU cycles to handle the business/flow logic of the site.
Still other apps will simply be filesystem/bandwidth intensive. (think YouTube)
There are ways to solve each of these scaling challenges already (some easier than others), but no one-size fits all situation.
DB intensive - you're stuck, roll your own MySQL scaling setup =)
App intensive - GridLayer system
Disk Drive / Bandwidth intensive - Amazon S3, Akamai and/or whoever YouTube used =)
(also remember getting coders to do anything is like herding cats. they'd much rather do whatever their comfortable with doing now, and figure out the scaling thing (which is not nearly as fun) at some later date if they have to. =)
Posted by Shanti Braford , September 21, 2007 12:06 AM
Let me make an unabashed plug here: try zeroCode at www.zeroCode.com. Its scalable, quick to build, all Java inside, but no code-level knowledge needed.
And it has all the usual problems of coders telling us everything from "But where's the code?" to "If there's no code, it can't be good enough for me" and "It sounds too good to be true".
Like someone paraphrased to us when he saw it: "Technology, sufficiently advanced, is indistinguishable from magic".
Posted by Kishore Mandyam , September 21, 2007 04:21 AM
Kabira Technologies - Model-driven architecture, 1 billion transactions/day capacity
ITA Software Inc. has a very hi-performance, high-flexibility environment for airline reservations, at 1/10th cost of IBM mainframes; uses Linux and Lisp!
Appistry Inc., St. Louis - "enterprise app fabric", founded as Tsunami research
Paremus Ltd., London - "enterprise service fabric"
GigaSpaces Technologies
Posted by John Verity , September 21, 2007 01:58 PM
Very interesting idea
Though I don't think it's as hard as some others are suggesting.
Maybe I'm slipping into 'old method' structure, but, the way I see it you could solve it with:
- distributed web server
- concurrent web language
- distributed object based db
- framework with required functionality
I love projects that redesign faulty, evolved systems. If anyone goes ahead with it, sign me up!
David - david at kelso dot id dot au
Posted by David , September 21, 2007 02:42 PM
There is such a platform. It's called the desktop.
Posted by Steve Dekorte , September 21, 2007 06:40 PM
It's all so simple! We need to start throwing graphs around!
Just go back to basics, make a GPL version of AllegroGraph and create a Purely Functional RDF based version of Lisp. A rough example of this is the Neno/Fhat language.
Then get used to Notation3-graphs being communicated between client and server. Focus on advanced synchronization of local and remote graphs. Finally start working this layer through to the front-end where each user has it's own instance of this RDF based virtual machine running, keeping local versions of remote data up to date, keeping a record of all personal information, maintaining a local store of all information posted to the internet while backing this all up in real-time.
By adopting purely functional data structures we can start versioning pretty much every bit of information simply by not garbage collecting it (yes, that's an oversimplification;), which means that:
1) we get a far a richer sense of history in our apps enabling us to do things like scroll back in time to see information develop
2) gain the ability to do SVN like updates on remote data.
The user-interface layer can be reduced to a 'suggested specification' for the interface that each user will be able to easily tinker with or the source-code for running the user-interface can be transferred once and dynamically kept up to date with the remote version.
Of course if such a system gets created we can stop including libraries with our software and let the client automatically download just the right version like maven does for Java and aptitude does for Linux.
We can even automatically create a profile detailing how often different parts of a library are used and let the user pre-load those parts that are always needed, let them load the parts needed often in the background and let them download the rarely needed parts when they are actually being called.
Of course in this scenario, YouTube is suddenly little more than an RDF-store for .FLV files with comment, tags and advertising-metadata and a single, fairly simple user-interface.
Posted by Michiel Trimpe , September 21, 2007 08:45 PM
I'm a firm believer that for multiplatform distribution to really take off, we need a streamlined developer environment that has many of the features you describe here, Albert. Towards that end, I too want to jump onto the Amazon bandwagon here, as the minute they announced S3 / EC2 we were banging away with them and developing some pretty wild use cases.
But I'm surprised no one has mentioned the emerging RIA solutions (rich internet apps). Microsoft and Adobe are both innovating with their powerful new platforms. The battle between WPF (esp paired with Silverlight/Expressions Blend) vs. Adobe's Flex/Apollo platform has driven tremendous innovation in the connected application area, and web apps in general. If you haven't seen them already, check out Taffiti (a silverlight-based search engine) and Kuler (a flex-based social media network).
Would love to hear other's thoughts on this topic.
Posted by Megan , September 23, 2007 01:40 PM
Albert, a lot of what you mentioned already exists. In one of the comments above, Kishore gave a list of companies, including GigaSpaces, which is where I work.
You should check it out, including our new Amazon EC2 offering, available here: http://developer.amazonwebservices.com/connect/entry.jspa?externalID=927&categoryID=101
Posted by Geva Perry , September 24, 2007 01:29 AM
The new platform is here. I just installed it about two days ago. It is about 10x faster than anything on the market. Made from the ground up for the small packets that ajax, web service layers, and SaaS eat for breakfast.
Posted by Gavin , September 28, 2007 06:24 AM
I used (and won a competition by using) Zimki, a hosted Javascript environment with a nice object store, session management, XML and JSON calls, and various other nice features.
Unfortunately internal politics with its parent mean it'll close in a few months, which to me feels like a shame, because it's the closest I've seen to Andreessen's "level 3" service, and the future promised it would be closer still. Digging around blog posts on the site and from developers and management involved in it shows the idea of federation - taking a site and having it fail over to new hardware - was certainly part of their agenda.
I keep meaning to try out Google's Web Mashup Editor to see if it comes close to allowing me to do the same things. It may well do, but they're never going to have the openness Zimki dreamt of.
Posted by Paul Mison , October 1, 2007 03:44 PM
We complain about SQL, but SQL is like COBOL — really old. There's been tons of advances from COBOL and FORTRAN to languages like Ruby and Java and Erlang and Haskell, and nothing similar on the database side. We had the object-oriented programming “revolution” to change how people think about code. Where's the change in thinking about data? I think a lot of these problems will become easier when we think about the problem differently.
Posted by Amit Patel , October 2, 2007 02:18 AM
If you build a conceptually new platform, developers have to learn the conceptually new platform. The more they have to learn, the less likely the problem is to succeed. Wasn't it the case, when they developed C++? It *had* to be C compatible, otherwise nobody would use it.
I afraid that the new platforms will have a hard life unless they manage to somehow look as the old platforms.. and in this case they are not that new anymore. Just as with C++ - plenty of new powerful ideas and you still are allowed to break a program with a wrong pointer.
Posted by Artem Marchenko , October 4, 2007 05:05 AM
Albert, cool post! -- I wanted to address this first comment by TronV:
Ning isn't working on anything near what you're describing. Ning is just the fancy 2.0 way to do YahooGroups, period. Nothing wrong with that but elevating such service to the kind of thing you're describing just doesn't make sense.. And Marc's post, although containing a few valid points, breathes way too much of a Ning's sales pitch.
I love your idea though, I just don't think it has anything to do with what Ning is doing. Amazon on the other hand, it's somewhat closer to that platform you/we dream of.
This is a common misconception. Every one of the functions you describe are available today on Ning. Not only they are available, they are the fundamental building block of the social network application.
I posted some more on this over at my blog: http://blog.diegodoval.com/2007/10/the_ning_platform_yes_it_is_th.html for those interested.
Posted by Diego Doval , October 4, 2007 07:21 PM
Apologies for the broken formatting -- paragraphs 2 and 3 in my comment were meant to be quoted. :)
Posted by Diego Doval , October 4, 2007 07:24 PM
I think we're building it. Watch at http://code.google.com/p/x2o/
Posted by David Baker , October 4, 2007 11:05 PM
I think we're building it. Watch at http://code.google.com/p/x2o/
Posted by David Baker , October 4, 2007 11:06 PM
The Web *as a whole* works on a huge scale. I would suggest the main problem with typical application/site hosting setups is as you say, a mismatch between traditional data techniques and the Web environment - massively distributed, *not* perfect but with a protocol that covers local errors (404s, 500s...).
A big step forward would be to start thinking in terms of the Web's data model - URIs are keys, links are relationships. If these characteristics can be evolved to work beyond documents, with data in general, then the underlying problem can be solved. This is the direction of RDF and Semantic Web technologies.
Still, this doesn't necessarily solve the local issues - there it probably will make sense long term to opt for a more distributed programming model. But if it's the Web you're scaling for, then the first thing to do is sort out the impedance mismatch between your data and the Web.
I suggest you check out the Talis Platform (I work for Talis because I believe they have the right approach to this kind of problem).
Posted by Danny , October 7, 2007 03:32 PM
In my opinion it will not be possible to find a platform that many are seeking without improvement in the way we express information/knowledge in the platform.
Regarding scalability of platforms, there are two type that arise.
1. Volume of data/processing scaling ( e.g. Need more servers, performance tuning etc)
2. Complexity scaling. (e.g. It is easy to do simple and little more complex things in Excel, but beyond certain complexity level it fails exponentially)
The complexity scaling issue is much more difficult to overcome.
This is where the separate threads of Web 3.0/Semantics/Ontologies/Knowledge representation and discusssions on platforms (level 1..3) should be combined. I believe that new platform with all the desired features can only exist by combination of these threads.
It seems that there is still quite a lot of discussion in elucidating expectations from Web 3.0. For me at the center is the word knowledge and semantic - but in a different context. The problem is the following: How come there is a need for armies of people to implement systems such as enterprise management software. It occurs by business expressing their requirement/knowledge and than it goes through layers of management and translation to eventually programing it (expressing knowledge to computer). We express our knowledge to computers in such low level language/manner that it needs this human translation machine to feed the computer. If one studies the various other proposals at expressing knowledge which are associated with Web 3.0 they are way to technical and low level as well, so in my opinion they will not solve the problem. So, there needs to be a way of expressing oneself/communicating with computer/web etc. that does not have to be translated to: for loops arrays, object etc, but is rather on the more abstract level and closer to the way humans think. So, for me, Web 3.0 is a new way of communicating with Web that involves explicitly specified semantic/knowledge that is machine processable. This will result in us being able to share knowledge rather than texts. On line desktop etc are more manners of delivery (on site/on line).
Our company Orfeo has been working on this form of expression for last 9 years and we are about to go live with an enterprise management/process exchange offerings based on deep semantic knowledge expression. It's domain will be ThoughtExpress.Com.
One will be able to express knowledge and processes directly in web developmen/test/production environment, full cloud computing structure will stand behind structure and thus it will be platform level 3.
Pawel Lubczonok
Posted by pawel lubczonok , October 16, 2007 07:14 PM



Ning isn't working on anything near what you're describing. Ning is just the fancy 2.0 way to do YahooGroups, period. Nothing wrong with that but elevating such service to the kind of thing you're describing just doesn't make sense.. And Marc's post, although containing a few valid points, breathes way too much of a Ning's sales pitch.
I love your idea though, I just don't think it has anything to do with what Ning is doing. Amazon on the other hand, it's somewhat closer to that platform you/we dream of.
Posted by TronV , September 19, 2007 01:23 PM