Software Factories – Refactoring An Industry

(I’m republishing an old article I had written about 10 years ago that seems to have disappeared from the web. This was originally published in 2006 on The fact that the issues described below have not really changed in 8 years is, frankly, sad.)

Software development has always been costly and time-consuming process. Specialized requirements and lack of skilled resources are just two of the difficulties facing companies today. Pressure to deliver software, on time and within budget, have pushed developers to look for a way to increase value delivered, while decreasing development time.

For many years now, efficient reuse of existing assets, either through object-oriented programming, component-based development or patterns-based architecture, has been one of the core objectives for the IT industry as a whole. Software reuse is seen as a means to combat many of the problems facing development teams. However, for many years and several different technology paradigms, this level of reuse has eluded the industry as a whole.

The book Software Factories: Assembling applications with Patterns, Models, Frameworks, and Tools aims to change this by modifying the definition of reuse as used by the IT industry and bring a more manufacturing approach to software reuse.

Economies of Scale vs. Economies of Scope

The difference between these two definitions are subtle and is something that has been proven outside of the software industry, in more established industries like manufacturing and fabrication. While both promise to reduce time and cost and improve product quality, they are actually quite different.

Economies of scale occurs when the initial development of a design and subsequent construction of a prototype result in multiple copies of that prototype are created. This is similar to the fabrication industry when a custom part is created and that part is used to produce hundreds or thousands of similar parts. In the manufacturing industry, it’s similar to a machine being built that can produce screws in bulk. In this case, ruse occurs in the later stage of production, namely construction.

Economies of scope occurs when multiple similar, but unique designs are produced in groups, as opposed to individually. Authors Jack Greenfield uses the example of a car manufacturing plant that uses existing designs, such as chassis, body and interiors, to produce several lines of distinct cars. Each product line is complete with custom features, but all share that same underlying design. Here the reuse occurs earlier in production, namely design and prototyping.

Greenfield point out that for years, the IT industry has been trying to apply economies of scale for achieving software reuse, when in fact, the IT industry should have been trying to apply economies of scope to achieve reuse.

Chronic Problems of Software Development

The authors of Software Factories identity some major problems with how the IT industry is attempting to achieve reuse. Unfortunately, simply altering how software reuse is applied within an industry or an organization won’t make systematic reuse a reality. Greenfield identify four chronic problems with software development, each of which impede a teams’ ability to gain valuable insight and reuse from existing domain knowledge and experience. These problems are: •

  • Monolithic Development
  • Copious Granularity
  • Process Immaturity
  • One-off Developer

Monolithic Development

Monolithic development means the creation of software in such a way as to make it difficult or even impossible, to utilize the resulting artifacts outside of a narrowly defined scope. Software development teams within large organizations are certainly familiar with this style of development. Several projects, several teams, all building isolated applications, without concern for what anyone else may be doing, or what, if any, domain knowledge they are embedding within the application. This results in large, inflexible applications that are of little use to anyone outside the original, intended audience. Even two projects within a single organization can be faced with a mini-integration effort in order for one application to take advantage of another’s functionality. Why does this keep occurring within organizations? Industry leaders have championed design by assembly for years, and yet very few applications take advantage of existing components, even within their own organization.

Copious Granularity

Typical software development for business applications consists of a strikingly similar set of features and functionality. For example, many business applications read data from a database, present it to the user, allow the user to modify the data in some way, and then allow the user to persist the change back to the database. Even though this is an over simplification of most applications, the basics still remain the same across projects. It begs the question of why developers use such fine-grained tools as standard programming languages like C# and VB .NET for representing such basic patterns? Part of the reason is the immaturity of modeling tools and languages. While a language such as UML is suitable for documenting software architecture, it is inadequate for allowing implementation to be derived from such models. UML lacks the extensibility required for generating large amount of code, and also lacks the breadth required to represent all aspects of software, including databases and user interfaces.

Process Immaturity

  • Controlling complexity at the expense of change — Many “traditional” processes would fall under this heading, including RUP and Waterfall
  • Controlling change at the expense of complexity — Most “agile” methodologies would fall under this heading, including Scrum and XP Before a process can be tuned for software reuse, it must mature because automation can only automate well-defined processes.

One-off Development

Software development projects within a company are usually so focused on the bottom line and delivery time, that overall architecture is placed on the back burner as “lofty” or “academic”. Very little regard is taken to analyze and evaluate existing assets, and very little time is dedicated to ensuring that new assets produced are reusable within other contexts. This results in many development efforts within a single company that create various amounts code and valuable assets for use within the company’s domain. Projects rarely perform post-mortems where reusable components are identified, documented, and packaged in such a way as to become reusable by other projects. These four problems, as well as the industrys’ misunderstanding of how to apply reuse with the current economic model, paint a pretty bleak picture as to the future of software development as an industry, especially compared to other more mature industries, such as manufacturing. Luckily, this is where the Software Factory comes into play. The next section contains a brief overview of what is a Software Factory is and identifies some of its main components.

Software Factories: The Solution?

When creating software within a specific vertical, knowledge about that vertical is frequently embedded in the software being developed. Unfortunately, this knowledge remains hidden inside the software; unable to be reused outside of the original scope of the software that contains it. This means reuse of this knowledge is next to impossible. This is where Software Factories look to step in with methods and procedures to harvest that knowledge, turning that knowledge into reusable production assets for a company. A Software Factory is defined by Greenfield, et al., as a software product line that configures extensible tools, processes, and content using a software factory template based on a software factory schema to automate the development and maintenance of variants of an archetypical product by adapting, assembling and configuring framework-based components. In layman’s terms, a Software Factory are about collecting existing, specialized knowledge about a certain domain and applications built within that domain. You can then use that knowledge to create a blueprint for other applications of a similar type. That schema can then be tweaked and configured to produce semi-functional to functional applications. In short, Software Factories means application generation from valuable, and reusable production assets that exist within an organization. Before delving into what makes components comprise a Software Factory, let’s first take a look at a look at the current state of the IT industry as compared to other industries.

Moving from Craftsmanship to Industrialization

All too often, highly skilled application developers and architects have to use their time for low-level, implementation level tasks. Usually, junior developers are not able to complete such tasks because of lack of appropriate domain knowledge, requiring the senior developer to mentor the junior developer. This fosters not only knowledge transfer, but also an introduction to the complexities of the current development environment. Since developers are always involved at some stage of development, very little time is spent in making development more efficient, especially low-level implementation details. This method of development resembles early goods based industries, where single workers create custom solutions, tailored to each individual requirement. Think early tailors or shoe cobblers.

This craftsmanship approach to software development does not scale very well. The lack of quality senior developers creates a mentoring environment, where specialized knowledge must be transferred, similar to an apprenticeship. Since there is such a hands-on approach required, each part of the project need to be created, most of the time, by hand. This often leads to higher quality, but also leads to performance and efficient issues. Migrating from a craftsmanship-based industry to a more industrial-based industry has been the path of progression for many more mature industries. If this is the end result for so many other industries, why is software development still based on small groups of highly specialized craftsmen?

Most people within the IT industry will agree that a form of standardization and modularization is the key to enabling the kind of reuse required for efficient industrialization of software development. What they don’t agree on is the means to which this standardization and modularization is achieved. The Software Factory aims to address this effort by prescribing a process by which software can be modularized into reusable assets.

Components of Software Factories

There are 3 main components of Software Factories:

  • Models and Patterns
  • Domain Specific Languages
  • Software Product Lines

Models and Patterns

The authors of Software Factories define Model-Driven Development as:

using models to capture high level information, usually expressed informally, and to automate its implementation, either by compiling models to produce executables, or by using them to facilitate the manual development of executables.

The importance of models comes from their ability to keep a consistent representation of concepts within a project. For example, while it’s easy to draw and manipulate a person object in a model, it’s much more difficult to manipulate the same person object at a lower level, because the person object could be represented by class files, tables, and columns in a database.

Representing and manipulating core abstractions and concepts within a software system is only half the battle, though. The other half comes from being able to effectively use those models to generate the underlying implementation details represented by the model

Domain Specific Languages

Domain Specific Languages, or DSLs, have long been a way to provide an abstraction atop existing concepts within a specialized domain. Examples of a DSL include SQL and HTML. Both provide specialized languages for manipulating concepts within their respective domain — tables, rows, and columns in the case of SQL and elements, tables, and forms in the case of HTML. For years DSLs like these remained the only cost effective DSLs because of their wide spread use and generalized concerns. No matter what your specific project entails, if you use a database, you can use SQL and if you use web pages, you can HTML. Creating DSLs for other, more vertical concerns has always been cost prohibitive because of the lacks of tools.

However, several companies have recently announced tools or plug-ins for the creation of DSLs. Once these DSLs are created, they can be utilized within a company to work with components at a much high level, with a much higher level of software reuse.

Software Product Lines

Software Product Lines are entire subsets of components that can be configured, assembled, and packaged to provide a fairly complete product. The resulting product should only require customization from a developer for the highly specialized aspects of the project. Perhaps the largest component of a Software Factory, Software Product Lines not only provide the greatest value, but also require the greatest investment. Software must be carefully partitioned into distinct and reusable components and those components must readily fit together in a coherent manner. Configuration is the key to Software Product Lines, as projects must be able to pick and choose which components they want to utilize, and then generate an application off of that configuration.

Implementing a Software Factory While all of the promises of Software Factories sound appealing, many companies have tried to provide the tools and components, only to fail under the load of inflexible tools or proprietary formats.

All of this is about to change. Big-name companies like Microsoft and Sun are getting ready to release many of the components necessary for building and assembling a Software Factory within an organization. With the release of Visual Studio 2005, Microsoft will unveil several add-ins and plug-ins that enable the creation of not only Domain Specific Languages, but also the integration of those languages with the IDE itself. This will allow developers to manipulate and use the language from within the Visual Studio.NET IDE.

Not to be outdone, Sun Microsystems is working on its own implementation of Software Factory technology, simply named Project Ace. Although, very little details of “Project Ace’ are available, developers shouldn’t expect Sun to let Microsoft provide .NET tools, without answering with a comparable set of tools for Java.

What about now?

While all this conjecture sounds wonderful for the future, many developers will be asking themselves what they can do now to utilize Software Factory techniques in their organizations today. Well, the good news is that a lot of functionality already exists with Visual Studio.NET. Products like Enterprise Templates, Project Item Templates, and Nant allow for the creation of standard artifacts that a team or organization can utilize today.


When I’ve been out selling my A Part-Time CTO services, I’ve sometimes run up against a company saying they have a CIO already, so why do they need a CTO? It’s a good question. I’ve been reading a lot about the role of the CIO and it’s constantly evolving responsibilities. A good primer on the subject is a white paper from PWC title “The situational CIO”.

After talking with CEOs and other CTOs & CIOs, I think I’ve come to the conclusion that you’re never really have both a CIO and a CTO. Your company is going to fall into one of two categories:

  • Technology is integral to your company. You have a CTO that manages the entire IT organization, including software development and infrastructure. Your CTO reports directly to the CEO and is a peer with the other CxO executives.
  • Technology is newer to your company and/or not as integral. In those cases, you probably already have a CIO who overseas several areas. There is a Director or VP of Technology that reports directly to the CIO

This is only my conclusion after a dozen or so conversations over a month, so I could be wrong. However, I think understanding where technology in general and a CTO in particular fits into an organization structure, compared to much older and well defined areas, is very valuable. It allows for technology to be given it’s proper seat at the table, so to speak.

Sr. vs. Jr. Engineers

When putting together a team of developers or engineers, there is a grand belief that most fall into one of two buckets: Senior or Junior. That designation never really worked for me because it’s so fuzzy. Unfortunately, it’s here to stay so I’ll be using it for this post.

I’ll be using “Sr.” and “Jr.” to represent the two buckets, with quotes, on purpose. Again, I hate the labels because it makes one seem inferior to the other. Of course, that’s exactly why some people like them, but I digress.

Which begs the question: “How do you know which bucket someone falls in?”

If you find yourself in the position of interviewing and staffing a team, you’ll find out that there’s an easy answer that most people use: years of experience. In fact, I’ve found that the cut off seems to be 5 years. If you’ve got under 5 years of experience, you’re Junior. Over 5? Senior.

For me, however, I believe that nothing could be further from the truth. When I interview people, the whether they fit for the role depends just as much on their thought process and the kind of experience they have then how much of it. I’ve met people with 10+ years of experience that I would have a hard time hiring for “Jr.” positions. I’ve also met incredibly smart people with 2-4 years of experience that I would put in the right “Sr.” role. If someone never evolves their thinking process, I don’t think that makes them “Sr.”, regardless of their years of experience.

If I could distill how I distinguish people to one point it would be: how the person understands and processes tradeoffs.

Software development is a difficult profession. As an engineer progresses in their career, they begin to understand that the decisions that go into shipping software are not black and white. There is a spectrum of gray. “Sr.” resources understand this and can make decisions in direction, architecture and implementation that reflect that mindset. They can analyze and articulate decisions about what will scale and what won’t. For example:

  • How will file uploading be handled when there is more than one web server?
  • What happens if I leave the team? Will others be able to work with what I’ve built?
  • Implementing X works for our needs now, but will need to be changed when Y becomes a requirement.

“Jr.” resources are usually more concerned with the task at hand and getting it done as quickly as possible, regardless of what position that puts them in the future.

To put it another ( albeit over simplified ) way: “Jr.” resources frame all of their decisions in terms of days. What are they implementing today, tomorrow and Friday. “Sr.” resources frame their decisions in terms of weeks & months. How is what they are implementing today going to look next week, next month and 3 months from now.

And don’t fall into the aggregating superstar myth. You need both types of people to ship software. Don’t just think that “Sr.” > “Jr.” and shoot for filling your team with all “Sr.” people.  Depending on the actual people on your team, it could work out.  However, chances are that you can crank out more software with a 1:3 or 1:4 ratio of “Sr.”:”Jr.” people.

But will it scale?

Whenever I’m in a discussion about platforms or language choices, someone will invariability bring up how some language or platform theoretically performs and use that as a reason to use it / not to use it. This is especially true in startups. ‘Will it scale?’ is always a topic of initial conversations and choices.

As an aside, I’ve seen many startups collapse under the complexity of a product they ‘built to scale’. In reality, they should have been more focused on creating an iterative, responsive design and being able to pivot easier. But I digress.

Before discussing what my answer is in that situation, I think it’s important to point out the difference between performance and scalability, since people usually confuse the two:

Web site performance is how fast a web site responds to a request. User clicks button, requests a page or does some otherwise comparable action, the server handles it and returns a result. A sites performance is how fast that happens, whatever it is. Are there a lot of DB calls? Images to load? I/O ? Is the response gzip’ed?

Web site scalability, on the other hand, is how a sites performance is affected under increased load, either user or data or something else. How is performance altered when there’s 10,000 users making requests? 100,000 users? What about the backend? How do the database tables respond with 1,000,000 rows? 100,000,000? That’s scalability. It’s dangerously hard to plan for scalability, especially since so many sites are a large collection of moving parts. Sometimes you won’t know how things interact with each other until they do.

One of my favorite sayings with regards to scalability is ‘Everything is fast for small n

At a micro level, performance is much more important. Users love fast, responsive pages. At a macro level, scalability is much more important. Users want stable, reliable software. If you have tons of the former, but none of the latter, you’ll be screwed.

Now, back to technology platform and framework choices. My answer to questions of platform scalability is always “With one exception, languages & platforms don’t scale, architectures do.

That exception is functional languages like Erlang that are based message passing communication as opposed to shared memory, like many OOP languages ( C#, Java, etc.. ) This gives them the ability to support concurrent execution from the start rather than rely on a threading model made up of locks, semaphores and mutexes. This doesn’t mean that they’re scalable out of the box, just that they support more scalable constructs from the start.

The design and architecture of your site will have much more to do with the scalability that any specific technology choice you make in the beginning. Your deployment and operations architecture can make or break your site once it’s actually made public. Spend a lot more time planning your caching strategy, connection pooling and proxy / web server / app server setup than arguing over which language can create 10,000 strings the fastest.

Distributed Version Control Systems

Most development and engineering teams have long since seen the benefits of using a version control system (VCS) to maintain and version their source code. For many years the popular free choice was CVS. That was supplanted several years ago by Subversion. Subversion adoption steadily increased throughout the years until it supplanted CVS as the default choice for companies wanting a free source control product.

Note: For the purpose of this post, I’m concentrating on free source control systems. There are several products, from Perforce to TFS to Clearcase that are widely used, but their price tags and complexity usually keep them out of the reach of smaller companies. (TFS does support Subversion, but it’s still several thousand dollars.)

That is, until a new breed of source control systems, called Distributed Version Control Systems (DVCS), emerged. Offering a more decentralized model than traditional VCS, DVCS promised to remedy some of the more clunky aspects of centralized VCS, especially when working with multiple branches and distributed teams. As such, scores of high profile OSS projects are migrating their code to a DVCS, including Python, Ruby on Rails to Linux.

But does that make it right for you, your team and your company?

It’s important to remember that this is a tool choice for your company as a whole, not just your engineering team. As such it’s important to include several factors when making a decision like this, such as:

  • What, if any, are the licensing costs for using the tools client? How about the server?
  • How user friendly is the tool? Does it require 3rd party GUIs or addons to work well for your needs?
  • What’s the deployment model? What types of servers, security and access will you need? What about backups and redundancy?
  • What kind of support do 3rd parties offer in terms of integration with the tool?
  • How mature is it? Can it handle what your team is going to throw at it?
  • Is it supported? How are security fixes rolled out? How easy are they to apply?
  • Who’s going to be using the tool? Just the developers? The engineering team? The entire company?


In the case of a DVCS, the tool choices are all stable and their usage is growing. Developers love them for their support of offline commits and much improved branching algorithms.

Note: Recent versions of Subversion have added the ability to merge branches that comes very close to the functionality offered by a DVCS.

However, rising above pure technical ability, most (maybe all) DVCS have some drawbacks. Three that come to my mind:

First, right now, there’s not a whole lot of integration with 3rd party systems. There is a nice ecosystem of bug trackers, wikis, file comparison tools, system admin tools and IDEs that offer built in support for Subversion. The corresponding ecosystem for DVCS choices is much smaller, though growing every day.

Second, the tools for working with a DVCS are pretty primitive and almost completely command line based. Now, this may not be an issue if developers are the only people working with the DVCS. However, non-developers are working with a VCS with greater frequency as the benefits of versioning artifacts and content become apparent. Trying to make them work with command line tools will not be easy.

Lastly, there is quite a learning curve between tradition VCS like CVS / Subversion and a DVCS. This could result in a gap in productivity. Though, as with any tool, the more experience your team gets with the tool, the better they will be at working with it. However, it’s quite easy to use a DVCS without using some of it’s major benefits ( like ease of branching, etc.. ) Thus without changing your workflow to utilize those benefits, you’ll be missing out on some of the very reasons to use a DVCS.

Note: I’m also ignoring the use case of maintaining mirrored repositories, one DVCS and one Subversion. In my opinion, I can’t see a team gaining enough of a benefit to outweigh the maintenance of two entire server solutions, complete with access control, backups, etc..

While DVCS are increasing in popularity, I’d have a hard time recommending them for a company to use as it’s VCS right now, except in some very specific circumstances. There’s just too much of a ecosystem to throw out by going with a DVCS over Subversion. Subversion is a known entity, it’s fast, it’s stable and it’s widely supported.

I have no doubt this is going to change in the ( possibly very near ) future. One of the nice things about most DVCS solutions is the ease at which you can migrate from a Subversion repository. As such, it’s probably best to wait for the tool support to catch up before taking the plunge.

In-House or Outsource?

One of the main challenges of starting a products based company is how do you build your product? This is especially true of web based companies, whose product consists of mainly a web site and very little more. For many companies, their web site is their lifeblood. Unless you’re fortunate to have one or more technical founders, you’ll quickly be faced with this simple question:

Do we hack out a prototype somehow, find a developer or outsource the development?

All have pros and cons ( only some of which are listed below ):
1. Create the prototype yourself

  • Cheap – this only takes up the time of your and your founders.
  • Total Control Over Implementation – Since you’re building it, you have the kind of fine grained control over everything.
  • Flexibility – You won’t need to commit to something, either cash, equity or involving other people, upfront. You can make decisions as needed.


  • Slow – You’re not technical. Depending on the level of complexity and difficulty, it might not be the most efficient use of your time.
  • Un-knowledgeable – Since it’s just you and your partners, you might not be making the best choices.

2. Find and hire one or more developers

  • Implementation Speed – Developers working full time will be much more efficient for you and your product since they’ll be dedicated to your company.
  • Technical Expertise – Bringing expertise in-house is the best way to foster communication and feedback when building a product.


  • Money – Unless you have cash to pay, the pool of developers willing to work for equity is small.
  • Equity – It’s not unheard of for a developer to look at a web based company, see that he’s asked to build the entire product and ask for a large chunk of equity as compensation.
  • Access – Good developers are tough to find.
  • Communication – Hiring a technical employee early on can sometimes result in a founder developer gap.
  • Need – If you just need a prototype or initial version of your product built, hiring a staff might be premature. While your team is trying to iterate and perform Customer Developer, your tech staff will be idle. And there’s nothing idle developers like to do more than think of new features to implement.
  • Lack of Knowledge – Building out a technical team when you’re not technical is fairly difficult.

3. Outsourcing

  • Capacity – development companies have people waiting to build your product.
  • Known Entity – You only need to guide the development of the product, they take care of finding developers, development process, etc..
  • Delayed Commitment – You don’t need to worry about hiring on people too early.
  • Cost – Outsourcing tends to be cheaper than hiring on developers and building your product internally.


  • Cost – Most development shops work on a cash basis. There are a few that take equity, but a ) they’re rare and b ) you’re making a significant decision about your company pretty early on. If the relationship sours for some reason, severing it could become pretty hairy.
  • Lack Of Communication – Typically there is a very “over the wall” approach to development where you specify requirements upfront and they give you what you specify. The type of day-to-day communication that takes place when developing a product is limited when the development staff is outside of the actual company.
  • Lack of aligned interests – While there is a desire to perform good work, development shops are not necessarily looking at the long-term outlook for your product. They may make certain concessions or decisions that could hinder the flexibility of your application

There is also a hybrid variant of #2 and #3 above that I’ll call a development partner. A development partner is an consulting company that works very closely with you, similar to an internal development staff. They are looking at a long term relationship and are more likely to take a craftsman approach your product development. Thus, they typically take a vested interested in ensuring whatever they build will be able to evolve with the company. Lastly, they are more likely to take reduced rates, be open to a debt-to-equity relationship with you and your company or some other more flexible compensation setup.  

So given all of that, which should you choose? Well, like most decisions, it depends:

  • #1, from my point of view, is out. If you’re not someone who can get up to speed VERY quickly and hack out a rough prototype, it’s simply not worth your time and effort to make the attempt. Unless, of course, it’s your very, very last option. Options #2 & #3 would have to not be possible for you to go this route.
  • #2 is also difficult, though not impossible. It’s only a viable route if you have to cash to hire someone and you can keep them busy enough with relevant work, not just aimlessly spinning their wheels. Without the cash, you’re very close to being DOA. Most of the technically inclined engineers who would have the itch to work at a start for equity are already working on their own idea.
  • #3 is a viable option if the relationship is properly managed and the product itself is fairly well thought out ahead of time. However, it should be known that, especially early in a products history, it will fluctuate. Having a outsourced relationship that can withstand those fluctuations is critical. You should place a premium on outsourced companies where the speed of feedback and communication can be the quickest.

In the end, I think there are two preferred routes: either find a development partner or hire a technical advisor or CTO-type person to manage an outsourced, development shop relationship. For the former, the partner you choose should be flexible enough to work in small bursts for you, as needed. In the case of the latter, this keeps your staff and employee commitments down. The latter option also allows you at least one person that can act as a liaison between the purely business founders and the technology team, whether they’re internal or external. You can also use them to bridge from the development shop into a internal team setup when the need arises.

The Ideal Web Architecture for Responsive Development

For years, whenever I’ve been involved in building a new web based architecture, I’ve always advocated that the engineers follow a simple guiding principle:

Design as though your web site is but one possible interface to your overall system.

This is because eventually most systems will need to be accessed in a variety of different ways:

  • Web Browsers
  • Console Applications ( ETL data loaders, etc.. )
  • Fat client apps ( iPhone, Android, etc… )
  • 3rd party services ( via an API of sorts )

The pattern of an N-Tier architecture has been a best practice for over a decade. However, that pattern has typically espoused a logical separation while maintaining a single physical deployment. There is a implication that the communication between the tiers takes place in-process. What I’m describing is a more of a physical separation of tiers and/or services.

Why would you do this? Well, the primary reason to separate entities like this is to encapsulate them. This insulates them and minimizes the effects changes that one component has on another. This is true for inter-connected components at most levels: the more two things are decoupled, the more one is resistant to changes in the other. This goes double for software products. This separation and encapsulation also allows you to changes things more easily in case you need to pivot your product to match an evolving business model.

“Isn’t this premature optimization?” No.

I’m not advocating building something you don’t need or trying to make things more complex at the start in the hopes that you’ll need it later on. What I am saying is that there are clear patterns and paths that can be taken to ensure that you’re not impeding your products ability to change and evolve as the business needs. There is a clear distinction between trying to solve problems before you have them and maintaining a flexible and evolutionary design.  Even if you never have to support multiple clients, your overall system will still be more flexible and responsive to change than if it were a more monolithic architecture.

For example, most newer web sites utilize AJAX technologies to make calls asynchronously back to a server, retrieve small data payloads, then process/display the results to the user. This updates the page without requiring a full refresh. Those calls are usually web service based, utilizing a URL to as the endpoint. It doesn’t take a large jump to make sure that each call isn’t specific to a particular client. Rather, it should not care who is calling it ( web page, command line app, python script, etc.. ), should be as RESTful as possible and as simple as possible. Taken in this light, suddenly your own calls become an API of sorts for others to possibly use. You end up building a web site utilizing your own API calls. Eating your own dog food.

Another example comes from the widely popular MVC architecture. It’s the architecture that Ruby on Rails, Django, ASP.NET MVC and a host of other frameworks are based on. The two pieces most closely tied to rendering data from your backend are the views and the controllers. However, if you’re building a web site, intimately tying your controllers with the expectation that the views are HTML based limits the would-be applications for the logic contained in your controllers. It’s better to not make such assumptions and rely on the view to render the data as it sees fit. If you’re using one of the above frameworks, your controllers will be URL based. If that’s the case, then suddenly reusing large parts of your backend becomes easier. For example, swapping out your regular web site for one optimized for mobile devices becomes a job of swapping out a set of views only, not rewriting your backend.

One last benefit of basing your product off of an architecture like this is that it makes it easier to scale individual pieces. However, that’s a large topic and will have to be addressed in another post.

Startup Product Development: Simple Now vs. Complex Later

When starting any software project, there’s an age old argument: should we build something simple that solves our current problem or should we use an existing product that’s more complex, but more feature rich, since we know that’s where we’re going to end up in the future?

This is especially true when starting a company because you don’t want to get into a situation where you have something that won’t scale up to handle your impending traffic and users. (Your company is going to be a huge success right? )

One side of the argument says to start simple and solve the problem(s) at hand. That’s not the same as making short sighted, possibly limiting, decisions. It means trusting that your engineering staff can handle any iterative changes required to support your changing needs. Understand any tradeoffs in performance & scalability that you’re making now in exchange for speed & time to market.

The other side of the argument is to target something that is more complex, but that solves more complex problems. The thinking is that when you eventually reach the point where you have the more complex problems, your solution will be waiting there, dormant, ready to be turned on. The proposed benefit is that you won’t end up with something simple that you need to trash once it outgrows your needs. You can grow into your product as needed.

However, in my experience, product development rarely, if ever, happens that way.

I recently ran into this very situation last week while talking to a startup founder. He had a client that needed some simple web sites built. However, the client believed that his needs would grow to require a more full-featured CMS platform. The founder reasoned that they should just start with the CMS, so when the need arises, it will take a minimal amount of effort to support the additional complexity.

I argued the opposite. Making the “just flip a switch” assumption ignores a large swath of other assumptions, any of which could prove to be a giant roadblock:

FYI: these assumptions are not mutually exclusive. You can ( and probably will ) encounter one or more of them.

  • That you’re actually going to need the complexity – Sad to say, but sometimes the need just doesn’t materialize.
  • That you understand all of your futures needs enough to select a product that would solve them.Premature optimization is the root of all evil. Everyone loves to make predictions and be proven right. In reality, it almost never works out that way. You’re not omniscient. If you’re doing Customer Development ( you are, aren’t you? ), you’ll be taking your direction from your eventual market anyway. Why make some of the most critical decisions about your company when you have the least amount of information possible?
  • That the product you picked can actually solve your future needs – If you’re like most companies, an exhaustive product evaluation isn’t in your initial project plan.
  • That, in the time between implementing the simple solution and needing the more complex solution, nothing changes with the product that you’ve chosen that could impede your transition – Products change & evolve. New versions are introduced, sometimes with backwards incompatible changes. Even understanding any version changes and testing your product for any issues is going to be a non-trivial effort.
  • The cost of maintaining and administrating the product you’ve selected is worth the trouble – Maintenance, Backups, database updates, security updates, etc… are all factors to consider when evaluating any product. If you don’t stay on top of updates ( especially security updates ) you can find yourself in a nightmare scenario.
  • That the effort it takes implement your product in the short term allows for an easy transition to the more complex solution in the future – There’s a good chance that you’ll be new to whatever product you choose. There’s an even better chance that, however you initially use it, that your usuage won’t be optimal.

Those are just a few to keep in mind. Any one of those assumptions can cost you time and money. Both are precious resources in a startup.

However, an oft neglected repercussion of building too much too quickly is that the extra functionality can calcify your product and make it very rigid. Releases become more complex, new features take longer to implement and bugs take longer to fix. You can find yourself a prisoner of your product, maintaining functionality and features that no one ( or very few ) people use. It can demoralize a engineering team, making them more and more susceptible to the nuclear option: the big rewrite.

I think the tendency to lean towards a more exhaustive solution upfront comes from a time when the effort require to change software was much higher than it is today. When systems were written in C, C++, Perl or even Java, making changes was a large undertaking. The thought of possibly throwing away chunks of code was nerve racking.  It represented a huge investment in time and money. However, with todays rapid development languages and frameworks like Ruby/Rails & Python/Django, the investment required to create something, both in time and money, is rapidly shrinking.

To me, for any new product development effort to succeed, you should do three things:

  • Start simply and build only what you need, nothing more.
  • Pick a platform / language / framework that allows you to pivot and change direction easily.
  • Hire good engineers that are experienced with different technologies and can change and adapt to different technology needs. This will by far the hardest, but it can be done.