About This Case

Closed

15 Oct 2007, 11:59PM PT

Bonus Detail

  • Top 3 Qualifying Insights Earn $300 Bonus

Posted

17 Jul 2007, 12:00AM PT

Industries

  • Advertising / Marketing / Sales
  • Consumer Services / Retail Industry
  • Enterprise Software & Services
  • Hardware
  • IT / IT Security
  • Internet / Online Services / Consumer Software
  • Legal / Intellectual Property
  • Logistics / Supply Chain
  • Media / Entertainment
  • Start-Ups / Small Businesses / Franchises
  • Telecom / Broadband / Wireless

The Emergence of DIY Application Development Platforms

 

Closed: 15 Oct 2007, 11:59PM PT

Earn up to $300 for Insights on this case.

Mashups have been an integral aspect of web 2.0, since the concept started to emerge, but they've been somewhat limited. For one thing, mashups have typically required some technical knowledge, which cuts off most users. Also, many of them are formulaic, e.g. Craiglist + Flickr or Google Maps plus a list of Starbucks locations. The upshot is that for the typical user, there's been very little ability to develop mashups of one's own liking.

But now a number of companies, both startups and established internet vendors, are trying to expand this area. Yahoo has its Pipes service, and Microsoft has released a similar service, dubbed PopFly. Google is also testing its Google Mashup Editor. Then there are a number of startups focusing on this space specifically, such as Teqlo and Coghead. Former Microsoft exec Charles Simonyi is also working on something in this area, although so far it's been pretty quiet.

Looking down the road, how is this space likely to play out? What can we realistically expect a non-technical amateur to be capable of? What needs to happen, from a technological perspective, before non-technical users are able to create robust software? Or are non-technical users consigned to being only capable of simple, pre-designed mashups? At what point does the emergence of these DIY applications start to have an impact in both the enterprise and on traditional software vendors. If this is going to have an impact on software vendors, what can they do to facilitate and profit from this trend? Don't limit your responses to these questions. Support your predictions with quantitative data, if applicable.

18 Insights

 



First of all, Simonyi's Intentional Programming and similar efforts, such as Dmitriev's Language-Oriented Programming (http://www.onboard.jetbrains.com/is1/articles/04/10/lop/) are _highly_ unlikely to be accessible to non-professionals in their early years. These are both attempts to create manipulable meta-views of the internal constructs of a software program; while the ultimate goal is to raise the productivity of developers, how these meta-views relate to programmer intention will almost certainly be non-obvious to those without a development background. Even within the professional development community, it is not clear that the mental models that Simonyi and Dmitriev are claiming as fundamental are perceived as fundamental by the average developer. Both Simonyi and Dmitriev are brilliant, but brilliant people don't always work the same way as the rest of us.

 As to the main question, the great benefit of mashups is that they represent the re-emergence of the bridge between power users and programmers. By providing a glimpse of the creative power inherent in programming, mashups may very well become the way in which young people "get into" programming. It's hard to overstate the long-term influence of one's first exposure to the power of programming -- not just the mental processes that mashups promote (as discussed below) but fondness for the platform itself (you never forget your first love). So mashup vendors are creating, not lock-in necessarily, but fondness and that, of  course, is valuable to them in the long run as well.

 As to the structure of mashups, the thing to realize is that they are primarily facilitators of what is known as the "Pipes & Filters" architectural pattern (ref. "Pattern-Oriented Software Architecture," Buschmann et al.). This is not a limited architecture! The UNIX shell (one of the primary older bridges between power-user and programmer) was a P&F architecture. MS' new PowerShell is, as well. If you consider the types of things accomplishable in UNIX shell, the potential of the similarly-structured "mashups" is very high (N.B.: to date, most mashups are combining data. The real power of P&F comes from filtering and what functional programmers call Reduction. Google "Map, Fold, Reduce" for more.). 

What distinguishes the architecture of mashups from older P&F systems is that the data sources are externally supplied and controlled. In this sense, mashups fit into the Service-Oriented Architecture models that are currently in favor in the professional development arena. One benefit of SOAs is that they tend to allocate programming talent efficiently -- those developers primarily consuming and consolidating data may be less sophisticated, while those providing the data sources don't have to spend time building HTML pages. That fits very well with the mashup model and suggests that mashups could be used in quite sophisticated applications (especially with the cooperation of data producers, such as has already been seen with the mapping services. But think about those involved in complex, but end-user-driven, supply chains such as travel, car customization, hobbies, etc.)

 Of course, the problem with SOAs is the reliance of the system on an external source! If, for whatever reason, the data source is modified or terminated, the system collapses. This is certainly even more true of mashups, where the supplier may not even be aware of the use of their system as a data source. 

The phrase "create robust software" was used in the question and this is the biggest weakness of beginning programmers (never mind whether they're using mashups or not, never mind whether they're paid money or not). The first-part of a professional programmer's career could well be defined as that period when they learn that their job is not solving "the optimal scenario," it's writing software that can survive the f***-ups: the missing data, the ill-formatted text, the users who say the system's broken when the solution is clearly on the third sub-screen of the "Advanced Options" dialog. 

Additionally, robust error-handling is actually a weakness of the P&F architecture, as it's difficult to coordinate the re-creation of state in a P&F architecture subsequent to a failure.

--

In the longer run and to the question of software vendors, I think the first thing to realize is that mashups are still nascent, but there's very likely to be early-mover advantages to those providing data sources, as they have a chance to be incorporated into "the roots" of the mashup community emerging within a platform. 

If you look at UNIX shell as the conceptual ancestor, that would lead toward thinking that mashups will evolve towards tasks that are fairly linear, but have almost arbitrary complexity in the transformation of a "source" into a "sink." They aren't very good at creating reactive or proactive systems -- mashups are unlikely to be the source of "autonomous agent" type applications. So that provides an opportunity for a vendor: provide the autonomous agent from which mashups draw their data; although to date mashups have focused on consuming almost-complete data, in the long run they will migrate towards direct use of foundational data (e.g., today a mashup will consume a map or scrape a search engine result, in the future, they might be more interested in direct access to geographical data or a prebuilt index). 

--

The big question with mashups is whether any of the platforms are capable of supporting the complete transition from power-user to mashup-developer to programmer.  Today (appropriately) the concern has been on the transition from power-user to mashup-developer (ease-of-use, quick results, etc.). Ultimately, though, the long-term success of a mashup platform will be very dependent on whether it can "scale towards" more complex programming and provide transitions for the talented mashup developer to become a data-source provider. The mashup platform that best provides that sort of capability and starts demonstrating moving people over that bridge is going to be very compelling.

 

Larry O'Brien

Contributing Editor: SD Times 

Founding editor: Software Development Magazine, Game Developer Magazine

The same predictions were made for 4G languages in the late 80's and for spreadsheet macros in the early 80's. And for agents in the mid-80's. However, once people realized that going beyond the very simple operations took more planning than unstructured mixing operations up, and that to do something really advanced you actually had to program (even Visual Basic requires actual programming), they soon got tired.

 The issue is that combining any information source to any other is equivalent to programming, and programming is not really about writing the program code, it is about structuring the operations. This is why HTML appealed to programmers, and why the web took off for real once graphical editors became available, by the way. A non-technical user can certainly progress beyond the simple, pre-packaged mashups, but at the cost of becoming a technical user.

 Another issue is the availability of data sources. While there is plenty of statistics on country level, as well as maps (you could argue that Gapminder, http://www.gapminder.org/, is nothing but a mashup of country statistics), there is not much data available to the public on levels which interest them in their daily lives. Of course, we have yet to see anyone mapping business intelligence onto Google Maps, but this is probably for another reason: The alternatives are good enough. After all, how often do you need to map your competitors to location and include some other statistic? If you are in corporate planning, yes maybe often, but then somebody can be hired to help. If you are a private individual, the breakdown of - say - restaurants with healthy menues is not something you do too often, and to do it requires planning and forward thinking. But you (or your husband) are hungry now.

Of course, a company could make its internal data available. And this means that the software vendors who would benefit from enabiling "corporate mashups" are the database tool vendors, and companies who make statistics tools (like SPSS and Spotfire. Or Adobe). Copyright (or, in Europe, database rights) is another issue, though. You do not want to give out your large data collection for free. And that means there has to be an incentive for the datatabase vendor to let you use the data.

To the "hobbyist" or the information provider, mashups are just too much work, unless your hobby is programming. So that is not a market, and the tools aimed at it are doomed. That said, I do believe they will have a stay of execution, and even what might be considered a flowering when IPTV becomes real. Overlaying information on top of IPTV streams is precisely the thing that would make people as excited as they were over mashups a year or two ago. It will probably also make for another round of tools vendors. But if they are to be successful, they will have to make their tools better for planning and modelling than tools today. When you approach operations on data sets, the programming-like qualities become even more important, as I said. Pipes could be worked out to address this, if it could include some of the modelling capabilities from a moderately good UML tool. But that means raising the bar for the amateur, who wanted to throw something together in a few hours. Still, there will be people willing to pay the price of learning technology to do something neat, if the see a value for themselves. So far, the use cases for mashups have not really demonstrated much more value over what you can already do with the web and a fairly simple programming language (like ruby on rails).

This, however, raises another issue: Do you want to tell Google what you are doing? If you want to make mashups that seriously describe your competitors in a comprehensive way, you do not want to use Google Maps.  So you need assurance that the data is going to be private. If large companies are to start using mashup tools, the main benefactors will be the Thompsons and VNUs of the world, who provide private database access and guarantee it (or Google might change their policy, but that sounds unlikely. Not that I do believe they disclose data, but if they don't tell, you never know).

So in my view, there is probably a niche for mashups in the corporate market, although the hurdles are somewhat higher than you would think - perfect for a startup, however. Venture capital, anyone?

Hope this helps

//Johan

I am going to play devil's advocate on this one...

Granted these DIY vendors and applications are great for creating quick applications. But there will always be conceptual knowledge that systems designers have that amateurs do not. The idea of 'computer' professionals being knowledge workers is right on because regardless of the tools at hand you still need the conceptual capacity to put the pieces together. 

For example you could create the tool that ends all tools to fix cars. But, I still wouldn't know how to fix my car or let alone build one from scratch.

Moving on to the enterprise level... Being a software consultant I have many times had to upgrade or "port" systems. Working with small businesses for the most part I have learned that this, well, sucks. Since Microsoft Access was introduced years ago I've seen many, many, many amateurs think they can build their own systems with this easy to use tool. Well, 99% of the time these systems work temporarily and are hardly scalable - which is when I come in to turn the hodge-podge network of Access databases into a real, functioning application.

I see releases like PopFly, Coghead, etc to simply be Web 2.0 versions of Microsoft Access. Granted people will be able to create semi-functional applications on the fly. But, they will never be built right by amateurs. That's where the knowledge worker is an essential ingredient in custom software development.

The suggested notion of "pre-formatted" mashups would be about the only feasible solution but this in essence has the same flaws because you can't generalize most business process' into standardized templates.

What this next generation of applications does do is give knowledge workers a new tool eliminating many mundane tasks of software development. They can be used as platforms for real applications built by real developers. I have built several application using Yahoo! Pipes as a platform for gathering data. 

Software vendors will likely see no impact. Regardless of how intuitive these systems are built it again comes down to conceptual knowledge that your average small business person does not have. Small businesses, afterall, will be the target market for systems like this because these DIY systems will never adhere to common conventions, ie security, that are standard requirements at the enterprise level. Thus, software vendors will not see an impact.

As far as "when will amateurs be able to create robust software?". I don't see this ever happening ~ at least not in my lifetime.

Mashup editors are a really interesting space right now. They are pretty popular among techies and those who follow tech news closely, but they really have not broken into the mainstream just yet. I don't think that there is any assurance that creating your own mashup will ever be a mainstream practice. Although, I think that improvements can definitely be made to make this seemingly daunting task more accessible to non-technical users.

As it stands right now, all three of the major mashup editors (Popfly, Pipes, and Google Mashup) require some level of technical knowledge in order to use them effectively. You can't even touch Google's editor without a pretty serious knowledge of web programming. The most accessible of the three is Microsoft's Popfly. A non-technical user can probably create a basic mashup using Popfly. The reason that Popfly is the most accessible, of course, is because it's not completely necessary to deal with writing actual code. The biggest obstacle in getting non-technical users to use these editors is eliminating the need for them to write code. Unfortunately most of the potential for creating great mashups lies in the user being able to modify the code to do what they want it to do. This is why I think that even if these mashup editors can simplify enough so that the average user can create a mashup, they will always be very simplistic, formulaic and limited in comparison to what a technical user with knowledge of code can create.

I think that we are already starting to see the effect of DIY applications on traditional software vendors. For example, both Microsoft and Apple have built in a platform for user created applications in their operating systems. Windows Vista includes Sidebar, an application that lets you put Gadgets on your desktop which were created by users. Apple's Dashboard application also offers similar DIY creations in the form of widgets. While the less technical users may not be able to create widgets specifically for themselves, there are enough people out there who have the skills to create them that they already have a wide selection to choose from. Microsoft is trying to help out their Gadget developers by allowing any mashup created in Popfly to be easily converted into a Gadget. This scenario creates benefits for both developers and the software vendor. For the developers, it makes their job much easier because Microsoft has given them the means to easily create and publish gadgets/mashups for their platform. This benefits the software vendor because developers will be more eager to develop for their platform since it is easier to get their application up and running than it is with their competitor's platform. I feel that this same strategy can be applied throughout the software industry. Supporting enthusiastic developers is key because you may have some killer developers working on your product, but they can be supported by a community of highly motivated developers if they are given the means and attention. Giving support like this to the developers who are making these applications for your platform is key in gaining their support in turn.

I also recently wrote an article for my blog on mashup editors. I think that it will definitely help to illustrate the key points that I think the three big players are hitting right now and need to hit in the future.

http://blogs.cnet.com/8301-13515_1-9747138-26.html?tag=head

 

SOA is a good start for creating robust software, but there is still a way to go before the "loose-coupling" involved is a) completely standard and b) completely secure. There is good progress being made here however, and there is a big shift towards SOA environments in business and the wider community. Once there is a stable platform for object-based, graphical mashups to be made, similar to the functionality available in Yahoo! Pipes, but with easier building of the visible front-end (as with Visual Basic for example), we will see a lot more of this type of thing being produced. However, saying that these will be non-technical users is still a leap of faith. For these applications to be widely used on the web, or even on a small intranet, there needs to be a change in the way they are presented. Web server setup is by no means "non-technical".

Of course, ISPs will start offering hosting for such things, and online services to "create your own" will inevtiably appear. Web-driven SOA with a graphical front end seems to be a reasonably good answer to all of your practical questions.

The implications of such a type of service are no more threatening to traditional software vendors than applications such as Facebook in my opinion. Of course when these turn into Salesforce type applications, then those who are slow to move in the marketplace (Sage) lose out, but the software vendors are the first ones who should be aware of these types of innovation.

The first immediate change I think we will start seeing is people piggy backing off each other's online applications to fill holes in functionality and improve products. Like testing the water before acquisitions become necessary, which is a powerful tool for any online company. Initially however, the impact will be low, minor improvements in performance, functionality, etc.

From WebMashup.com: "Gartner said, even limited impact is worth it, as users will appreciate incremental improvements in the usability of applications. High levels of impact and business value can only be achieved when the development process encompasses innovations in usability and reliance on complementary server-side processing (as is done in Google Maps)."

Expect to see issues around licensing of these applications, and problems where companies are restricting the access and/or usage for other applications. It will, like everything internet related create entirely new business models.

Therefore the effect on enterprise is a bit of an unknown at this stage, it depends on what people create I suppose, and if I knew that, I would be out doing it.

Some guesses I would take are an increase in networking tools, mixed up with advertising and sales applications, sticking together people plus sales/advertising = $$$, so that would make sense. Ebay and MySpace, that would be interesting, LinkedIn and SalesForce, imagine the money in that. All done with GoogleMaps of course, so you know where your victim, er, client, is speaking from.

icon
Rob Newby
Fri Jul 27 3:16pm
Well, I just started using Popfly after reading this, and it's everything I described here as being "the future". It appears that MS has beaten me to it. It's still quite shaky in Beta, memory hungry and crashes a lot, not many features yet, but it should prove to be quite powerful.
Oh, and they host it for you, you can build your own site, then embed the mashups you create in a drag and drop OO style. It's still a little bit technical, but this is definitely the way forwards.

Fundamentally, what needs to happen before non-technical users are able to create robust software is that the software is going to have to become a lot smarter.  

Here's an example: back in the 1990s, I was involved in several projects at a large manufacturer seeking to create a real-time feedback network for its geographically-dispersed manufacturing operations - the goal was to be able to be able to take the system's status all the way down to the individual machine level.  The primary challenge lay in being able to make so many different operating systems "talk" to each other.

The answer then is the same as now - you can program the individual systems to incorporate metatags - or some kind of XML - that describes the data, but is designed to do so in a way that the underlying information can be filtered, combined or otherwise manipulated by other coding to produce the desired information product.   But, you don't get that result if you don't develop the ability of the various components to be able to communicate the necessary information.  Without that capability, you can only produce a pre-designed mashup.

The key for a software vendor is to sample what people are trying to do, or that they would do if they could do, identify the potential customers, solicit a development contract and then make the communication possible by making the underlying systems better able to communicate with each other to accomplish the desired task.  Then, if and when the development contract allows it, license it.

 

The mashup space is currently dominated by technical developers who are creating mashups from multiple data sources and then publishing the resultant content onto websites or into widgets.  This is then consumed by the average consumer.

 The reality is that products such as Teqlo & Coghead are only ever going to be consumed by the top 2% of Web users.  Sites such as Netvibes & iGoogle will eventually be able to allow users to consume widgets that are created, but realistically, the average web user will never create a widget, no matter how easy it will become - although they will consume widgets created by 3rd parties.

Non-technical users will be able to configure widgets easily, and publish them onto webpages, and solutions such as Synthasite (www.synthasite.com) will allow them to consume 3rd party widgets quite easily.  

 Traditional software vendors are having to deal with the eventualities that the Internet bring, as detailed in this blog post (http://www.vinnylingham.com/2007/02/top-20-reasons-why-web-apps-are-superior-to -desktop-apps.html).

Enterprise Software vendors are going to have to focus on more agile, customer centric platforms with Web Services, that allows the client to access the data within the systems, and re-use for other previously unimaginable reasons.

I believe that we will see the various software markets becoming more fragmented, as the focus around different customer segments becomes key and the economics of providing software adapts to the changing market.  Mashup tools may never go mainstream, but publishing platforms that allow them to configure widgets, and publish them, either onto personal or public web pages, should become the mainstream route of consumption.

 

 

 

To expect too much of the average Joe is to wind up disappointed. Let's not forget that 80% of users out there keep all their default settings and have a Windows Desktop full of automatically installed "shortcuts". And I'm talking about the Silicon Valley crowd who do business presentations, and reveal the fact that they have an "install AOL" and an "Adobe Acrobat Reader" shortcut on their desktops. If they don't even delete the useless shortcuts on their desktop, can we expect them to create a mashup?

No. The bulk of advanced users, when they cotton on to the notion of Web 2.0, will be at the limit of their ability when they use mashup tools that have been provided to them, such as widgets for Windows, add-in apps to Facebook, and crap that can be added to a MySpace page. Advanced Mashup development tools won't decrease the complexity of designing a mashup, even if they decrease the difficulty in developing it. They also won't automatically increase the creativity of the average user. People will still have a flashing 12 on the VCR, and default settings in their browsers.

But where these tools might be used is by SMBs that are building websites, or groups, clubs, and organisations that previously were limited to "Brochureware" websites or Web 1.0 one-server-based sites. These groups often have stronger motivation, one or more techies, or the services of a webmaster at their disposal. It is in these cases that a set of tools for creating a mashup will be used. And they can be used to great effect.

Software vendors, such as Adobe, MSFT, etc. should include the ability to create mashups in their premier design suites, offering a smattering of widgets themselves (hosted on their own servers) and a raft of partnerships with other data feeds and services. With such tools, a SMB like a real-estate office, that previously had a Web 1.0 brochure website could enhance that site with news tickers, filtered by real-estate theme and also overlay their home listings on a map with satellite imagery. This would revitalize the site, and make sure it always had fresh content. But it's not just "Fresh content": the site could also offer some utility, like tying into zillow.com data when a user hovers the mouse over a listing on the map. (In fact, listing examples of possible mashups is a never-ending effort, as there are simply billions of combinations). But the trick is for the software vendors to integrate these tools into their suites, because without that reduction in complexity, Web 2.0 won't grow beyond a few well-funded geeks, or resourced companies.

This is a lot like the web in 1995. One had to know how to write HTML to make a page. Few did. By 1996, there were tools that allowed the average Joe to make a page, and the web grew. Then WYSIWYG tools emerged and the web took off. Mashups are the same. We are in the HTML coding phase, and we need to get at least to the tools stage.

To profit from this trend, one could set up a developer (or a clearinghouse) for mashup components. Newsfeeds, of course, already exist. But what could a new firm offer that would be a good mashup component. Hey, I don't know the answer to this, but a look at the add in applications on Facebook would be a good primer. Best case is for a developer to invent / aggregate / acquire a suite of apps and build a name as a web 2.0 one-stop-shop.

The general pattern you see with mashup-style content is that a small number of more-savvy users create content and share it with a much larger number of less-savvy users. Creating mash-ups is always going to be complicated, and so only a small minority of users will go to the trouble of learning how to use use those tools.

So success in the mash-up space should be focused on making it as easy as possible for more-savvy users to create mashed-up content or services and share them with less-savvy users. I think there are two principles that are crucial:

* First, the mash-up functionality should be an extension of a service users would be using anyway. Google Maps is a good example of this: everyone understands what a map is, and Google has managed to create a variety of useful applications by layering functionality atop the basic map interface. General purpose "mash-up software" like Pipes isn't likely to appeal to a broad audience (even the most savvy 1-5 percent of users) because users just won't know where to get started. It's much easier to start with a map and say "I wish I could add feature X to this map" than to start with a completely blank canvas and come up with a mash-up application from scratch. And once someone has created a mash-up, it's going to be much easier to share the mash-up with other users if the other users are already using the site on which the mash-up was constructed.

* Second, the application should make it as easy as possible for more-savvy users to share their creations with less-savvy users. This might happen at a couple of different levels. At one level, the site should provide APIs so that programmers (who will be a tiny proportion of your user base) can add features to your platform and make them available to a broader class of users who are technically savvy but aren't programmers. Facebook's API is a good example of this: adding an application to your Facebook account is just a matter of clicking a button. On another level, and more importantly, the platform should give those power users a lot of different ways to share their creations with ordinary, non-technical users. YouTube's embedding model is a good example here: only a small fraction of users upload videos to YouTube, but YouTube makes it extremely easy for those users to make the videos available to a much broader class of users who aren't YouTube users at all.

So the way software vendors can profit is by putting their application at the center of this creation-and-sharing process. The goal shouldn't be to get your typical user to create new mash-ups. The goal should be to help the most technically-savvy 1-5 percent of your users create mash-ups, and then to make it as easy as possible for them to share the results with the other 95 percent of your users. The latter won't know they're using "mash-ups," they'll just know that your platform has a constant stream of cool new features.

Incidentally, the question "What needs to happen, from a technological perspective, before non-technical users are able to create robust software?" is not a new one. Lots of people have tried to solve that problem, and they've inevitably been disappointed. Visual Basic, for example, is a pretty user-friendly programming environment. Before that, Hypercard was an elegant quasi-programming language. Yet only a tiny fraction of users used those tools and others like them. The reason isn't that they weren't "user friendly" enough. The reason is simply that programming is an inherently complicated activity. Making a programming language powerful enough to do something useful inevitably means making it complicated enough to require a substantial upfront investment in learning to use those tools.

The vast majority of users have no clue what a mash-up is, and it will never occur to them that they ought to find out. The only way you'll get a mass audience of users to use, much less create, mash-ups is if they don't know that's what they're doing. They should be able to regard it as simply adding new functionality to a website they're already familiar with. A general purpose "mash-up platform" is going to be limited to the Slashdot crowd.

 

You can tell when something is supposed to be the next big thing when all the big boys of the industry jump on the beta bandwagon. Yahoo pushed out Pipes, Google has its Mashup Editor and Microsoft has PopFly. What I think they don't understand is that the general public are not programmers by far. They don't understand the basic concepts of programming and thus will not be able to create these mashups with the existing tools. Pipes and PopFly I have seen hands on. I have played around with Pipes Beta and have yet to see the Google Mashup editor. Mashups have become all the rage in online communities and generate lots of web interest. Is it just the new buzz word right now? Is it up there with Web 2.0?

The existing tools are not easy to use and for the inexperienced user they will sit staring at the screen until they say... This just isn’t for me, and move on to something a little easier on the brain.I consider my self a pretty technical person and I could not get Pipes working correctly. I had data and feeds all setup, I was sure I was going to wind up with some pretty cool results. Then I was stuck and it took me a little while to see what was going on. I realized I needed to add it to a feed reader - so that is what i did. Ah hah! I had data! Kind of but not the results I wanted and gave up. This was about two weeks ago and I have not thought about the service again since then. None of these services will succeed if the people they are designed for can't use them.

If services are designed to allow people to "Mashup" data like Google maps with data pulled from the big boys like craigslist or eBay easily and fullfilling a purpose and a need. Then we would be getting somewhere. Combine that with a simple click, click, click interface that anyone could use and you will have a winner. The reason that Pipes, PopFly nor Google Mashup Editor hasn't made a huge splash in the blog-o-sphere is due to its inability to be utilized by the people it was made for. Looking to easily grab data from multiple sources and apply it to whatever they want to do. I went back after reading this question and investigated my MASHUP options again and realized Yahoo's Pipes was the only one I was able to really play with. Again Google and Microsoft have both of them limited to Beta Invites only. So I started playing with Pipes again. I still was unable to create what I set out to create - it was not predictable. I still had not sought out help and instead decided to try some pre configured pipes. Hey guess what those worked. I tried some more. Success! So I drew one conclusion from this and that is for any company to succeed with a Mashup site or service will either need a super simple method for users to create them or a lot of pre created mashups that people can request. And maybe as the service evolves it will be easier to use.

WYSIWYG Editors have been around for a long time now for web projects, visual basic and even CAD like software. No amateurs are creating professional sites, software or designs. And it will be the same thing for this Mashup service. Until a breakthrough in easing the technical knowledge necessary to create a mashup it will not take off. Either introduce people to existing mashups to generate some more excitement and then once you have their attention teach them how to use your service. Until then people are going to be steering clear.   Remember the normal user does not want to create software they want to use it. And they are willing to pay for that. If the process of creating your mashup was fun and breezy it might be a different story. But sadly it is not. 

If the future changes the tools then mashups will live on and thrive if not they will become another programming language the general public doesn’t know but wants to get someone to do it for them so they can put it on their "blog". For a software vendor to capitalize and make money for this product still in its infancy they need to become synonymous with MASHUP - like i think band-aid and not plastic adhesive strip. If you Google something or go Mapquest it - then you need to become the mashup. It needs to be fun and easy to use and have a Digg like network to share, promote or bury in top 10 lists or what have you. Allow users to save their mashups and compare and discuss them with others. Premium users could store their mashups with extra space for their music, pictures, documents and whatever other data is important to them. Then they can use their stuff to mashup with existing content and really personalize the experience. How about MyMashup, MashMeUp or something along that rode. Be the hot new thing and grab people using DIGG for their only news source. Now have your users key words go out and aggregate news with those words and further customizing with that data you have saved stock lists for financial news. Sports scores for that list of teams on your fantasy sports team list in notepad. If you turn along those lines your possibilities could be endless.

In conclusion the mashup project can go in several different ways and depending on which it might flop or it could be the next big thing. Instead of people getting rss feeds from their news sources they are getting these new mashed up feeds customized to their likening with just the data they want to see. Filters get rid of stuff they don’t want. This data from these feeds could feed into tickers in the home or office and right on to the refrigerator. Feeds for grocery lists mashed up with a Google map of the stores that have the stuff in stock. Done correctly this could be the next BIG thing.Get it into the home in cheap wifi devices that display tickers get people used to it and the business market will be yours very quickly. Enterprises are comprised of real people if those peoples shopping lists and household finances are all of a sudden in better order than their enterprise... I think they will realize the obvious move. 

 

My opinion on that is that those "pipes" or PopFly services mostly serve as demonstrator for the available APIs provided by the service developers.

We are moving towards a "social" internet, were not only people socialize but also services. You can get some localization data from A and put them on B and display that on you web page, that's a mashup, web services socializing through their APIs.

Can we expect a big toolbox where a non-geek would be able to create about anything ? Honestly I don't believe that.

APIs are living objects, changing everyday, evolving... Ideas usually emerge before the API function enabling its easy development. Trying to make a super-service-mashing-tool is simply impossible, as there is bound to be a lot of limitations and thus frustrations for users that simply won't be able to create exactly the thing they imagined. Moreover, a non technical user can change a RSS feed in a well designed widget, but cannot design the widget itself, but can givee good feedback on its usability...

thus we have some kind of value chain :

API Provider\

API Provider ------   Developers ------- End users

API Provider ///

But DIY applications are really happening, because more and more people are learning to write basic AJAX, and AJAX has established itself as the way to create RIA (Rich Internet Applications). Thus full libraries of those DIY appications are being created and called "Widgets. My favorite example is the awesome boom of Netvibes  with a growing community providing real applications on their widget library.

Hopefully this widget thing will emerge as a standard (UWA ? ) and thus be able to run on any platform as either RIA or RDA (Rich Desktop Application). And when this is done it will change the shape of your PC. You won't run anymore those general applications that you use at 10% of their capabilities, either online or offline those mashups and simple vertical apps will bring you the service you want in a light and ATAWAD form... and you'll just have to go online on your favorite widget library and pick it up !

I was previously the CEO at Teqlo so I do know a little about the space. :)

The issues are as follows:

- Who is the targeted user? Most people won't build mashups and developers generally want to have rich IDE tools, therefore the prosumer audience is somewhere in the middle (think of the Excel power user). What functional areas and vertical markets do you focus on? Sales and marketing in service-based industries would be my target.

- API capabilities are limited. There are hundreds of APIs but most of them are crap. There are performance issues with APIs as well as most companies don't build their business around offering APIs so they don't invest in delivering them.

- UI, specifically the lack thereof. APIs don't always have a UI so someone has to build it, which goes well beyond the capabilities of the average user.

In spite of these challenges it makes a lot of sense to deliver mashup capability inside of the enterprise. The secret to success, IMO, is taking feed based data sources (RSS and APP) and delivering them in such a way as to enable integration with pre-packaged application components. I have written many blog posts on this subject, too many to count, and continue to hope for a future that includes user generated mashup capabilities.

The Masses as Developers: Can Everyone Write Software?

 

Mashups have indeed come to characterize some of the main value propositions for Web 2.0 as a whole. As web-based applications which combine data from more than one source into an integrated experience, mashups represent the latent potential of Web 2.0 as well as some of its glaring shortcomings. Even with the takeoff of acceptance by end users, there are numerous barriers to entry for the development of mashup applications. These barriers characterize the difficulties facing the prospect of everyday users participating in the development of more complex applications.

A strategic approach must be formalized towards identifying the various realities associated with developing these types of applications, in addition to stimulating broad-scale participation in doing so. These Do-It-Yourself (DIY) platforms embody the future of wide-scale application development, which is both an opportunity and threat for established web platforms as well as traditional software vendors.

The current market for do-it-yourself application development platforms (all currently mash-up oriented) is best described as barely nascent. A proper analysis of available options is crucial to ascertaining an understanding of what needs to be addressed moving forward. In doing so, it is less important to concentrate on the generic usability-related aspects (design, outlay, user interface composition, etc.) of these platforms and more critical to evaluate how seamlessly they produce quality web-based applications. Granted, they must be easy-to-use and intuitive but the main focus of value determination should be based on how easily robust, functional applications are created. Therefore, such platforms should transparently enable the following qualities to be purported to applications:

Distinctiveness
Can users infuse individual vision into the development process? Or are they limited by the boundaries of the given platform? Are everyday-users given the equivalent of control that knowledge of a programming language would afford, without having to learn a language?

Cohesive User Experience
The nature of combining multiple sources into a single entity is prone to create undue perplexity. Developing useful mashups without technical skill requires not only an eye for how the user will perceive and interact with it, but also the tools to integrate the sources in a manner that promotes a consistent user experience.

Diversity of Utility
Is it possible to produce mashups with a high level of utility? Services should never dumb-down the ability to construct usefully complex output. Concomitantly, they should not prevent the creation of simpler, less intricate ones. Optimally, a healthy range of complexity should be feasible regardless of user preference.

Support for Content
Obviously if a mashup consists basically of different data sources, content is king. So it does matter how well content can be manipulated. Are there innovative methods made available to integrate the content outside of the traditional drag-and-drop? Content support/handling offers opportunity to assert a variety of differentiation unlike any other area.

 

An Analysis of the Trend
The shift to the Web 2.0 phase has already begun, with DIY applications set to play out as a shift therein. Despite all the hype there is a sizeable set of practical benefits to be realized from related technologies, applications, and concepts. The user interface advancements contributed by Web 2.0 technologies have fundamentally affected the expectations of users of software as it relates to web-based software as well as software in general. The transition by the Internet from flat, static web sites into a more dynamic, participative medium has opened the door for application developers to write complex yet useable software without investing an inordinate amount of effort. In turn, this has made the concept of further lowering the barriers to application development a reality, hence the movement towards DIY creation. Just as HTML opened the doors to a wave of new web developers, it is feasible that DIY application platforms will do the same for web software that is logarithmically more multifaceted.

The development platforms which emerge as successful will be those that address the difficult prospect of providing transparent control over the creation of mashups. In this way it is almost compulsory that editing is done from a What-You-See-Is-What-You-Get (WYSIWYG) perspective. A point not lost by the current designers of available services. However, what is missing is a strong task definition which provides users with an intuitive understanding of what to use to accomplish something. Traditional programmers already understand what it means to compile, debug, and profile an application and when provided with an integrated development environment that pulls these together, can go about manipulating each. At this stage in DIY application development there isn’t a defined set of terminology in terms of its tasks. Therefore, ordinary users have no base of understanding when it comes to reliably picking up various nuances. This must be addressed both at an individual service as well as at an industry level.

Bungee Labs (http://www.bungeelabs.com/) with its Bungee Connect service provides a working example of both a development and deployment platform for the web using the software-as-a-service model, where everything happens in the Web browser. Clearly, the applications meant to be developed on Bungee Connect are degrees more sophisticated than the typical DIY application will most likely be in the short-term, yet the company’s model is very much applicable. For starters, they have done well to provide a straight-forward conceptual understanding of the development process and environment. Bungee Labs emphasizes a process that can be summed up in a Start -> Develop -> Deploy -> Share characterization. DIY application development, while remaining significantly less intensive and critical would benefit from something similar. A set of steps that go into pulling together a dynamic application must be spelled out in terms which communicate an essence of simplicity (not to be confused with effortlessness) and transparency, even if the application created is not simple.

Incentives to Grow
Another critical aspect to the pickup of DIY applications will be how well platforms providers work at creatively introducing the concept of incentives into the picture. After the initial point of engagement where novelty of use begins to wear thin, what makes an end-user come back and continue to exhibit the desire to interact with the service? In the same manner that open source software communities are faced with the reality of encouraging participation and contribution within mostly volunteer driven groups, DIY development will be powered by a large mass of individuals who aren’t being paid to do so. It will matter very little that development platforms for non-trivial applications exist if there is no real motivation for using them.

Therefore, certain incentives must be introduced as a realistic acknowledgment of human nature. People blog to express themselves on topics of their choosing they use social networking applications to connect with others, etc. The logic behind incentives should be steered by the answer to, why a person would want to continue to develop what would amount to, complex applications for use by themselves/others? Incentives don’t have to be monetary compensation (which might actually be the least effective variety), but they must serve as a buttress for the primary motivation(s) behind participation and uptake.

The actual growth curve of DIY application development platforms will be determined by how well it wins bottom-up support from its target market. Similar social networking applications adoption and validation will come from the user community first and foremost. Then industry recognition will follow. Encouraging participation should focus on creating an explosive relevancy, which is the ultimate source of “buzz” and of social capital. Accordingly, the technology provided to create applications must provide simple tie-ins to social networks. It should be possible to create an application and to customize its interaction with services like Facebook, Blogger, Digg, LinkedIn, StumbleUpon, and Twitter. Application creators shouldn’t need any technical knowledge of integrating with the aforementioned, only they should be afforded the ability to do so within their applications.

Crowds are most receptive when they are invited to gauge quality for themselves or if someone with relevant credentials, i.e. an influencer, alerts them. Blog entries, open beta and other forms of non-traditional approaches garner the best results. The goal is to establish and encourage a brand of open, bi-directional dialogue that promotes relevancy across interconnected crowds. That means doing whatever it takes to endorse feedback of all variety. In this case even unstructured, opinionated feedback is good as it provides a pulse for the end-user community in the absence of being able to interact with them on a consistent basis.

The Road Ahead
To succeed in enabling more to participate in developing non-trivial, web-based applications requires an understanding of the challenges of effectively easing the task of developing any type of application. In the early days of traditional application development very few productivity tools were available to aid the process. Today there is a plethora of platforms and toolsets which capably do the job, one such example is the open source Eclipse Platform. Governed by the Eclipse Foundation which posits the following as its purpose of “…building an open development platform comprised of extensible frameworks, tools and runtimes for building, deploying and managing software across the lifecycle.” Likewise, DIY development platforms must encapsulate and provide support for the entire application creation process.

In order to find success doing this, it will be necessary to enable the collaborative aspect of this process. It should be possible for multiple individuals to participate in the application creation process. A one-to-one mapping of author to application is in no way reflective of the reality of development as a mostly collective approach. A single individual is very rarely capable of producing as output, the type of applications that gain significant traction even amongst smaller groups of users. The best will be considered true platforms and not just services as they will cover and support the entire experience.

As seen with other Web 2.0 enabled applications, the concept of perpetual beta works well by enabling early adopters to provide valuable, early stage feedback. Interaction between users and do-it-yourself application authors is critical as a method of gauging efficiency and empowering continuous improvements to be added. Therefore, more complex DIY application development platforms must seek to integrate the perpetual beta concept into offerings. Successful development is best categorized by spurts of activities over a long haul, as a better reflection of the time constrained reality of its participants. The option to roll out different versions that include specified changes, or automatic change detection / versioning handling would be well-received. Without the notion of versioning, application development can become stale, as the idea of incremental progress is lost in a sea of modifications.

As web applications have demonstrated an increasing ability to imitate client-side software in terms of functionality as well as look-and-feel, the web browser has become the de-facto delivery medium. In doing so, a larger mass of potential users can be reached. The prospect of enabling those potential users to participate in fashioning applications is quite powerful. Currently a host of web-based Application Programmer Interfaces (API) like Representational State Transfer (REST), Really Simple Syndication (RSS) and JavaScript Object Notation (JSON) has emerged as links within this sea of unstructured information. They help connect the content which attracts and connects people. Enterprises must realize this and adjust their business practices and policies accordingly. Walled-off, closed door approaches will not scale and will leave companies with them at the behest of more agile competitors.

Why the Enterprise Should Care
In the same breath, enterprises have quite a bit to gain from the concept of DIY platforms. Since people are the fuel for business processes and have a first-hand understanding of related tasks, responsibilities and requirements. Empowering them to create applications which encapsulate collective understanding of the work-to-be performed and sharing them with co-workers boosts productivity exponentially. Moreover, these applications could augment the heavy-duty Business Process Management (BPM) suites available, filling in the various practice-area specific shortcomings among other things. Employees would be capable of producing not only work-output but also contributing tools that enhance its quality and consistency.

Adobe Systems made its task-based “kuler” service publicly available (http://kuler.adobe.com/). It showcases the capabilities of the company’s Adobe Integrated Runtime (AIR). Adobe, in its attempt to transition into a first rate tooling/platform vendor, has identified that the current generation of development techniques for software (both enterprise and consumer oriented) will be required to imitate a more end-user friendly composition. Badly designed, difficult to use software with steep learning curves do not make the cut. Users have already begun to expect software which affords them several degrees of control more than what has traditionally been provided. Interactive web sites really aren’t all that new, only that the level and grade of interaction between user and software has skyrocketed in volume and complexity. As a result, key elements of the development processes and delivery of enterprise software are being affected by the paradigm shift. Expectations for software have rapidly changed and those in the business of selling software must adapt or suffer the consequences.

The demand to create personalized applications for use by others is set to take off as the newer forms of Web 2.0 style functionality are being exhibited on wider scale basis. Yahoo! is in the process of testing a significantly different and advanced version of its web mail client (Yahoo! Mail beta) that features an entirely new interaction style than the current version. A key principle behind the paradigm shift is ease-of-use driven design and development. Increasingly popular categories of software will continue to enable this at a basic level (Social networking – Ning, the services mentioned earlier). The net result is a tremendous amount of room for growth in the area of applications which are fundamentally user-centric.

A Paradigm Shift
The path to aligning with (and ultimately profiting from) the movement of empowered users as application creators entails gaining an understanding of the trend itself. Software vendors should prepare for an extended cycle of disruption that transfers power into the hands of communities. In order to leverage this disruption, community outreach will have to be effectively integrated into development strategies. The community surrounding a product is set to become critical to the success of a product.

Therefore, focus should move from being solely product-oriented to one which is more centered on the application-ecosystem and prizes building community knowledge and dynamic forms of documentation. Examples include companion forums, wiki’s, message boards, etc. Software vendors should seek to encourage the growth of self-sustaining user communities that draw-in other users and generate social capital. Users should cease to be viewed as commodities to be “captured” and more as valuable, active participants within an ecosystem.

A driving theme behind DIY platforms is the concept of dynamic application assembly controlled by unique user needs. The option of pulling together content and data from disparate locations across the web into singular applications is indeed appealing. This should be noted by software vendors who are in a position to model the different elements of Web 2.0 interaction styles within their application development strategies. Outside of the familiar MVC pattern, there are other interaction styles like pipes (see: Yahoo Pipes) which are viable alternatives and should be evaluated.

Particularly, these patterns feature higher levels of loose coupling between the user interface, core services and content, which improves both usability and architectural flexibility. It will be invariably helpful for software vendors to track the types and uses of applications enabled by DIY platforms seeing how they represent living, breathing case studies of user wants and needs. The output from these platforms won’t be perfect, by any means, but they will serve as assets to be analyzed for respective strengths and weaknesses.

Moving Forward
Facilitating the comprehensive understanding of the character of DIY applications through careful analysis will set the stage for developing profitable strategies. The boundaries of this analysis should be fashioned around the trends that represent what will potentially emerge as a major paradigm shift. Currently, DIY application platforms are essentially embryonic but their ongoing evolution has serious implication on the range of currently accepted models for the design, implementation and deployment of software. Both traditional software development vendors and the enterprise stand to benefit from taking notice of this reality.

icon
Alex Fletcher
Tue Oct 9 4:45pm
I mistakenly submitted a version of my work containing two typo's:

"Despite all they hype there is a sizeable set of practical benefits to be realized from related technologies, applications, and concepts." The 'they' should be replaced with 'the.'

"They goal is to establish and encourage a brand of open, bi-directional dialogue that promotes relevancy across interconnected crowds." The 'they' should also be replaced with 'the.'


Thank you,

Techdirt Insight DIY Applications

 

The desire to move from complex and specialized to simple is a trend that has followed a technological course since day one. As a process is invented, it is further simplified until human interaction is almost not necessary. From printing presses to automobile assembly lines, this process has carried out for at least since the dawn of the manufacturing age. We have seen this same trend hold true in technology, from spaghetti code on the early mainframes, and binary programming, through Pascal, COBAL, C, C++, C#, java, PHP, and HTML. All iterations of technology move towards simplicity and libraries of commonly used routines.

 

There is no reason to believe that the long-term view of the mashup process, currently only truly available to the technologically perceptive programmer will end up being componentized like Popfly and Pipes. Popfly has the fortune of being an early drag and drop system that takes various preprogrammed routines and makes a mashup that can be used on a web page. As these technologies get more refined, the lowest common denominator of non-programmer who wants a widget on their web page can build their own. At some point, the user will have to point to a data source, say what they want to do with the data source, and click “OK” to have a widget that does a thing.

 

A prime example of this is an RSS reader, or mapping of dynamic technology like Twitter Maps, you can easily drag and drop to create a twitter map using Microsoft Virtual Earth using popfly in under an hour as a non-programmer. Add a different dynamic data stream like a P2P application, and you now have a P2P mapping system using popfly.

 

The winner of this space is going to be the one that simplifies widget making to the point as HTML pages are made now. No coding involved unless the person really wants to do HTML coding. With templates, precut designs, blogging software, it is no longer needed to understand let alone code HTML. The great boom in content making has been in part driven by the ability to write in a word processor and upload using your favorite CMS type system.

 

The winner of this space is going to be the Wordpress or Moveable Type of widget making. Simply say what you want to do, where you want to get data from, and where it will be posted, then click on upload. Traditional vendors have not gotten to this type of interface yet, but the first one who does stands a very real chance of seeing their systems become as popular as Wordpress or Moveable Type. The DIY applications are already having an influence on the corporate work environment. When a user has their own page or group page, one of the first things that goes up is an RSS reader to collect headlines and data from trusted sources from within and without the company. Many CMS systems come with this ability already, and people working in Share point, Drupal and other CMS systems are using it.

 

The way to profit from this trend is to be the first one to be as simple as Word Press and other blogging or CMS systems. To facilitate and profit from this trend, development should be towards a package that plugs into any CMS or Blogging system, and provides levels of portability so that it can be write once, use anywhere.

 

While mashups started as tools for building DIY apps, their usage will increase in enterprise. For mashups to exist, you need web services (either REST or SOAP based). Here are a few things we will see happen.

  • Providing web services as an interface to applications will become the norm (Look at the number of APIs on Programmable Web (http://www.programmableweb.com) 
  • Increasingly platforms will make webservices as the default interface (Facebook, SalesForce)
  • WebServices exchanges (like StrikeIron) will grow
  • Making mashups will become as easy as writing EXCEL macros (QEDWiki from IBM provides drag/drop building of mashups)
  • Combining,Filtering, transforming web data will be made easy (Yahoo Pipes)
  • More companies will jump in to fill the gap in security, identity and other aspects

Mashups are the tip of the ice berg. They create a new way to build multi-os, multi-client applications easily. It is a bottom up way of ushering in Web Services. We still have a long way to go. We especially need:

  • A mashup directory (machine readable - like UDDI)
  • A uniform way to use identity (and security) - openid seems to be a candidate
  • A simple way to abstract mashups or a declarative language to describe mashups ( for example, a mashup markup language)
  • Techniques for scalability, abstract interfaces to common services (like mapping)

Here are a few  blogs I wrote after my first mashupcamp:

http://dorai.wordpress.com/2006/07/13/mashups-and-wikis/

http://dorai. wordpress.com/2006/07/10/mashups-and-microformats/

http://dorai.wordpress.com/200 6/07/09/mashup-markup-language/

http://dorai.wordpress.com/2006/02/18/mashups-sho uld-be-as-easy-as-excel-macros/

As a non-programmer I have experimented with several of these tools and none work as claimed. These are written by programmers and they simply can't get out of their mindset. The best example I can think of of an application platform that truly doesn't require a programming background is Ning, the do-it-yourself social network creator. Its designers really thought through what a non-programmer needs: drag and drop, selection of features from menus, simple ability to arrange functionality, etc. To be successful, these applications should be simple enough to not require any instructions.

The challenge is that the application tools need to be intelligent enough to self-organize, that is, if I as a user do something that defies programming logic, the tool should fix the problem behind the scenes. I suspect this is a much bigger challenge that what they are currently tackling. The problem is that programmers understand logic and non-programmers don't.

I have written software business requirements docs. These define what the software should accomplish and lay out a suggested user interface based on what a user needs. It is preferable for simple applications to have a non-programmer write these specs for the reasons covered above. I think the designers of these tools need to bring in regular, non-technical users and observe their use of the tool. It's my guess that they'll be shocked at what they see.

What needs to happen, from a technological perspective, before non-technical users are able to create robust software?


Here's five things which will improve mashup development.

A Local Pipe

One of the available pipes should always be information uploaded from a user's hard drive. It's the information they know best, and probably the information they want most. Local pipes would create million of possible outcomes by creating millions of possible starting points.

If that's the only thing a "non-technical" user accomplishes, the end results can still be spectacular with a sophisticated set of "pre-fabricated" pipes online.


Universal Interoperability

Anyone who's played with mashups always finds one big problem: "This pipe can't talk to that pipe." But beyond that, there's another obvious pipe compatibility problem. Can your Yahoo Pipes talk to your Google and Microsoft Pipes? (Think of this as "These pipes can't talk to those pipes.") This immediately suggests two possible outcomes for all the competing pipe environments...

An open standard comes along and kills them all.
An open standard comes along and makes them all more useful.
Pipes are only useful if they're interoperable -- and the more flexible pipes will be exponentially more popular. Competing pipe development platforms will ultimately be judged by how much data they can (and can't) make available.


More Pipes

Data mashups obviously exist to exploit the "network" effect, since every new pipe makes the existing pipes more useful. Having cool data excites developers -- even data that's been traditionally closed off from the development community. And its availability may ultimately excite users -- even "non-technical" users, who will then create a strong demand for site-specific pipe-based applications.

I'm sure MySpace's 100 million accounts would be a tempting target for pipe developers. Savvy sites might even implement a developer outreach program to ensure they're not left on the sidelines.


Data Tags.

Data tagging will make it much easier for pipes to be interoperable. Any mashup environment will be able to accept any properly tagged and formatted data coming into its pipes. And when pages on the web become tagged and pipe-ready, the web may evolve entirely new sets of capabilities.

It's inevitable that all the data we create will someday be tagged. Emerging standards will eventually create a tipping point after which tagging will become universal -- and this will help the spread of mashups. (Imagine every digital camera in the world adding a timestamp and a geocode to every picture, by default...)


Education

There's an old salesman's maxim. "They want it; they just don't know it yet." Pipes are revolutionary, and will draw millions of users once their potential is realized. But today, pipe development could benefit from some corporate sponsorship, and maybe even some pipe evangelists.


Looking down the road, how is this space likely to play out?

I think we'll hit a milestone when someone develops the first killer app using a simple mashup of pipes.

This will generate press coverage and attention, which will spur new waves of development and the release of more pipe-compatible data and applications. It's a virtuous circle, since the availability of more pipes attracts more developers and more successful applications -- which will encourage the availability of still more pipes, attract more developers, and lead to more applications.

Ultimately web services will also be appraised for their usefulness as "piping commodities," and there will be a rush to create pipe-compatibile web applications. For prominent applications, some data-holders might even negotiate fees for making their data available on high-speed connections, since an immensely popular application could otherwise put an (unmonetized) load on system resources. (For example, the New York Times site handles 18,500,000 pageviews a day -- but they're able to monetize that through advertising.) Another possible outcome is special firewalled data pipes available only to paying subscribers. (For example, the archived content of the Wall Street Journal.)


mashups have typically required some technical knowledge, which cuts off most users.
Here's an interesting anecdote. The woman who created the Hamsterdance didn't even know how to write HTML. GeoCities offered her a template which asked what image she wanted on her web page -- and she just kept hitting the "insert" key until she'd added the five images of dancing hamsters 300 times. She then added a sound file using the same template, and put it on the web. It attracted close to 100 million visitors, and at that time became one of the 20 most popular sites on the entire internet.

The pool of developers increases with how easy the interface is -- and how intriguing it is, and how much buzz there is about the potential results. But the interfaces I've seen have a drag-and-drop interface, which is relatively easy to work with, so the divide between "technical" and "non technical" users may not be as sharp as we think.

 

Also, many of them are formulaic, e.g. Craiglist + Flickr or Google Maps plus a list of Starbucks locations.
Don't underestimate the value of the formulaic. The power of mashups is that they can be exponentially more useful. Imagine searching for apartments on Craig's List, but limiting the results to a specific geographic region identified on Google Maps, and then pulling up pictures of the neighborhoods on Flickr or the local businesses with Amazon's A-9 service. Though it's formulaic, it's also truly revolutionary, a new kind of "pre-qualified" information which simply has never been available before.

Every permutation will be tried -- and many embraced, but at a level that's hard to detect. Think of it as a "long tail" of usefulness. To somebody, somewhere, one formulaic mashup may be exactly what they need. So even though it never rises to widespread utility -- it never needs to.


At what point does the emergence of these DIY applications start to have an impact in both the enterprise and on traditional software vendors.

Don't forget that it could be a good impact. Enterprises are suddenly able to get exactly the information they need -- customized for their market and product -- by mashing it together using freely available web tools.

I don't think it's an either/or choice between "mashups" and "commercially-designed software." After all, there may be commercially-designed software which uses data from pipes. When any pre-existing web service becomes pipe fodder, the web really will complete its transition into a platform for developers.

The real distinction might be between software which uses piped data, and software which doesn't, as users may get used to the infinite customization that pipes allow. Someday Microsoft Word could pipe web-based data streams into a document, while Excel accesses a user's favorite graph-generating mashup on the web to format the numbers in its spreadsheets...

It's a piped world after all...

How is this space likely to play out?

Mashups have already become a key component of web development and this trend will continue and expand. I predict that mashups in some form will be an essential component of many successful web projects and will be used at some level by at least half of all major web projects within 2 years.

However, Gartner's 2006 "Hype Cycle" report warning is well advised:

... because they combine data and logic from multiple sources, they’re vulnerable to failures in any one of those sources..

Also, as data and content ownership lines continue to blur, mashups create a potential economic vulnerability.

 

What can we realistically expect a non-technical amateur to be capable of?

Quite a bit. Video gaming has spawned a generation of extremely gadget and tech savvy onliners and the trend towards high sophistication of this online and gaming group will continue. However motivation will remain primarily focused on gaming, entertainment, and video enhancing applications rather than traditional website mashups.

What needs to happen, from a technological perspective, before non-technical users are able to create robust software?

"Robust software" is generally not the province of non technical people and that is unlikely to change, especially as technologically robust open source projects absorb much of the budding amateur and "newbie" technical talent.

However for robust applications that enhance websites the DIY platforms are improving fast and will soon make mashup configuration the lowest barrier to entry. Yahoo Mash, their beta social network, has done a great job of making mashups simple enough they can be done in seconds at a user profile. More important is simply the fact that most onliners are not interested in development.

Are non-technical users consigned to being only capable of simple, pre-designed mashups?

Not necessarily. As DIY's become modular and user communities develop we'll see some good stuff on a limited scale, but it's not realistic to think that a homemaker from Peoria will be *motivated* to do more than basic profile enhancements or website work. The motivation is not there now and this won't change.

 

At what point does the emergence of these DIY applications start to have an impact in both the enterprise and on traditional software vendors.

Last year and even before. The threat to enterprise applications grows daily and is increasingly rooted in the fact that mashups, CMS, and online document tools such as Google's suite of "Office-like" applications offer users completely viable alternatives to legacy software. Few users need elaborate applications anymore as work flows increasingly through online and email applications. Enterprise applictions still benefit greatly from the human aversion to switching and the fact that office workers are trained in those particular aps. As office workers increasingly become comfortable with online applications for their personal use (e.g. Google spreadsheets), they'll be easy to wean off of the enterprise applications which are often clunky and more problematic than their online equivalents.

If this is going to have an impact on software vendors, what can they do to facilitate and profit from this trend?

This is a *very* tall order because Google and Yahoo and others are "giving away" superb applications in an effort to consolidate traffic and loyalty. It's working spectacularly well for Google. But here are some ideas for profiting from the "free software" revolution:

* Create inexpensive applications with potentially huge user bases. e.g. Facebook or Myspace or Google Gadgets or user interfaces. Monetize thse through advertising or very small user fees.

* Create utilities that are sellable to big players as great standalone applications they can give away, and sell these to the big players. This is potentially the big payday for a great application. Only big players can scale applications quickly, and they are too busy to create everything they'd like to see. Flickr was a simple, excellent application that Yahoo purchased and scaled rapidly up to huge success.


Summary note:

John Musser’s Programmable Web is a superb mashup resource. John’s listed thousands of mashups and APIs and categorized them in helpful ways.

Mashups are reshaping the internet in very dynamic ways and will continue to do this for some time.

Users are increasingly inclined to want an internet that is free or very cheap, very open, very rich with content, and has few restrictions on the use or mashing up of any and all content. To that end they will put up with advertising if it remains targeted and unobtrusive and won't necessarily stop using services even when ads cross the line, as with full screen interstitials and popups.

The early internet has kind of “swung back” and again is characterized by information sharing rather than the “closed walls” that came about when big money started to flow into the system. This poses a challenge for new companies based on mashed up content because ownership of the content that results from a mashup is not always easy to define. At Mashup Camp 2 I spoke with Venture Capitalist Peter Rip who at that time felt that mashups *in and of themselves* were not the key value proposition, rather how the mashup might enhance the prospects of an existing company. Companies counting on facilitating mashups are even harder to analyze, though I would suggest that they should build a community around them ASAP (e.g. NING).

Charging for things like API development for small firms might work for a few years, but IMHO all but the most complex software services are going to be available for free within 5 years, perhaps sooner. Rightly or wrongly many are staking everything on ad revenue rather than sales of applications, services, or software. This will shape the web for the next decade in ways that are hostile to shrink-wrapped software providers.