Software Factories – Refactoring An Industry

(I’m republishing an old article I had written about 10 years ago that seems to have disappeared from the web. This was originally published in 2006 on TheServerSide.net: http://www.theserverside.com/discussions/thread.tss?thread_id=32812 The fact that the issues described below have not really changed in 8 years is, frankly, sad.)

Software development has always been costly and time-consuming process. Specialized requirements and lack of skilled resources are just two of the difficulties facing companies today. Pressure to deliver software, on time and within budget, have pushed developers to look for a way to increase value delivered, while decreasing development time.

For many years now, efficient reuse of existing assets, either through object-oriented programming, component-based development or patterns-based architecture, has been one of the core objectives for the IT industry as a whole. Software reuse is seen as a means to combat many of the problems facing development teams. However, for many years and several different technology paradigms, this level of reuse has eluded the industry as a whole.

The book Software Factories: Assembling applications with Patterns, Models, Frameworks, and Tools aims to change this by modifying the definition of reuse as used by the IT industry and bring a more manufacturing approach to software reuse.

Economies of Scale vs. Economies of Scope

The difference between these two definitions are subtle and is something that has been proven outside of the software industry, in more established industries like manufacturing and fabrication. While both promise to reduce time and cost and improve product quality, they are actually quite different.

Economies of scale occurs when the initial development of a design and subsequent construction of a prototype result in multiple copies of that prototype are created. This is similar to the fabrication industry when a custom part is created and that part is used to produce hundreds or thousands of similar parts. In the manufacturing industry, it’s similar to a machine being built that can produce screws in bulk. In this case, ruse occurs in the later stage of production, namely construction.

Economies of scope occurs when multiple similar, but unique designs are produced in groups, as opposed to individually. Authors Jack Greenfield et.al uses the example of a car manufacturing plant that uses existing designs, such as chassis, body and interiors, to produce several lines of distinct cars. Each product line is complete with custom features, but all share that same underlying design. Here the reuse occurs earlier in production, namely design and prototyping.

Greenfield et.al point out that for years, the IT industry has been trying to apply economies of scale for achieving software reuse, when in fact, the IT industry should have been trying to apply economies of scope to achieve reuse.

Chronic Problems of Software Development

The authors of Software Factories identity some major problems with how the IT industry is attempting to achieve reuse. Unfortunately, simply altering how software reuse is applied within an industry or an organization won’t make systematic reuse a reality. Greenfield et.al identify four chronic problems with software development, each of which impede a teams’ ability to gain valuable insight and reuse from existing domain knowledge and experience. These problems are: •

  • Monolithic Development
  • Copious Granularity
  • Process Immaturity
  • One-off Developer

Monolithic Development

Monolithic development means the creation of software in such a way as to make it difficult or even impossible, to utilize the resulting artifacts outside of a narrowly defined scope. Software development teams within large organizations are certainly familiar with this style of development. Several projects, several teams, all building isolated applications, without concern for what anyone else may be doing, or what, if any, domain knowledge they are embedding within the application. This results in large, inflexible applications that are of little use to anyone outside the original, intended audience. Even two projects within a single organization can be faced with a mini-integration effort in order for one application to take advantage of another’s functionality. Why does this keep occurring within organizations? Industry leaders have championed design by assembly for years, and yet very few applications take advantage of existing components, even within their own organization.

Copious Granularity

Typical software development for business applications consists of a strikingly similar set of features and functionality. For example, many business applications read data from a database, present it to the user, allow the user to modify the data in some way, and then allow the user to persist the change back to the database. Even though this is an over simplification of most applications, the basics still remain the same across projects. It begs the question of why developers use such fine-grained tools as standard programming languages like C# and VB .NET for representing such basic patterns? Part of the reason is the immaturity of modeling tools and languages. While a language such as UML is suitable for documenting software architecture, it is inadequate for allowing implementation to be derived from such models. UML lacks the extensibility required for generating large amount of code, and also lacks the breadth required to represent all aspects of software, including databases and user interfaces.

Process Immaturity

  • Controlling complexity at the expense of change — Many “traditional” processes would fall under this heading, including RUP and Waterfall
  • Controlling change at the expense of complexity — Most “agile” methodologies would fall under this heading, including Scrum and XP Before a process can be tuned for software reuse, it must mature because automation can only automate well-defined processes.

One-off Development

Software development projects within a company are usually so focused on the bottom line and delivery time, that overall architecture is placed on the back burner as “lofty” or “academic”. Very little regard is taken to analyze and evaluate existing assets, and very little time is dedicated to ensuring that new assets produced are reusable within other contexts. This results in many development efforts within a single company that create various amounts code and valuable assets for use within the company’s domain. Projects rarely perform post-mortems where reusable components are identified, documented, and packaged in such a way as to become reusable by other projects. These four problems, as well as the industrys’ misunderstanding of how to apply reuse with the current economic model, paint a pretty bleak picture as to the future of software development as an industry, especially compared to other more mature industries, such as manufacturing. Luckily, this is where the Software Factory comes into play. The next section contains a brief overview of what is a Software Factory is and identifies some of its main components.

Software Factories: The Solution?

When creating software within a specific vertical, knowledge about that vertical is frequently embedded in the software being developed. Unfortunately, this knowledge remains hidden inside the software; unable to be reused outside of the original scope of the software that contains it. This means reuse of this knowledge is next to impossible. This is where Software Factories look to step in with methods and procedures to harvest that knowledge, turning that knowledge into reusable production assets for a company. A Software Factory is defined by Greenfield, et al., as a software product line that configures extensible tools, processes, and content using a software factory template based on a software factory schema to automate the development and maintenance of variants of an archetypical product by adapting, assembling and configuring framework-based components. In layman’s terms, a Software Factory are about collecting existing, specialized knowledge about a certain domain and applications built within that domain. You can then use that knowledge to create a blueprint for other applications of a similar type. That schema can then be tweaked and configured to produce semi-functional to functional applications. In short, Software Factories means application generation from valuable, and reusable production assets that exist within an organization. Before delving into what makes components comprise a Software Factory, let’s first take a look at a look at the current state of the IT industry as compared to other industries.

Moving from Craftsmanship to Industrialization

All too often, highly skilled application developers and architects have to use their time for low-level, implementation level tasks. Usually, junior developers are not able to complete such tasks because of lack of appropriate domain knowledge, requiring the senior developer to mentor the junior developer. This fosters not only knowledge transfer, but also an introduction to the complexities of the current development environment. Since developers are always involved at some stage of development, very little time is spent in making development more efficient, especially low-level implementation details. This method of development resembles early goods based industries, where single workers create custom solutions, tailored to each individual requirement. Think early tailors or shoe cobblers.

This craftsmanship approach to software development does not scale very well. The lack of quality senior developers creates a mentoring environment, where specialized knowledge must be transferred, similar to an apprenticeship. Since there is such a hands-on approach required, each part of the project need to be created, most of the time, by hand. This often leads to higher quality, but also leads to performance and efficient issues. Migrating from a craftsmanship-based industry to a more industrial-based industry has been the path of progression for many more mature industries. If this is the end result for so many other industries, why is software development still based on small groups of highly specialized craftsmen?

Most people within the IT industry will agree that a form of standardization and modularization is the key to enabling the kind of reuse required for efficient industrialization of software development. What they don’t agree on is the means to which this standardization and modularization is achieved. The Software Factory aims to address this effort by prescribing a process by which software can be modularized into reusable assets.

Components of Software Factories

There are 3 main components of Software Factories:

  • Models and Patterns
  • Domain Specific Languages
  • Software Product Lines

Models and Patterns

The authors of Software Factories define Model-Driven Development as:

using models to capture high level information, usually expressed informally, and to automate its implementation, either by compiling models to produce executables, or by using them to facilitate the manual development of executables.

The importance of models comes from their ability to keep a consistent representation of concepts within a project. For example, while it’s easy to draw and manipulate a person object in a model, it’s much more difficult to manipulate the same person object at a lower level, because the person object could be represented by class files, tables, and columns in a database.

Representing and manipulating core abstractions and concepts within a software system is only half the battle, though. The other half comes from being able to effectively use those models to generate the underlying implementation details represented by the model

Domain Specific Languages

Domain Specific Languages, or DSLs, have long been a way to provide an abstraction atop existing concepts within a specialized domain. Examples of a DSL include SQL and HTML. Both provide specialized languages for manipulating concepts within their respective domain — tables, rows, and columns in the case of SQL and elements, tables, and forms in the case of HTML. For years DSLs like these remained the only cost effective DSLs because of their wide spread use and generalized concerns. No matter what your specific project entails, if you use a database, you can use SQL and if you use web pages, you can HTML. Creating DSLs for other, more vertical concerns has always been cost prohibitive because of the lacks of tools.

However, several companies have recently announced tools or plug-ins for the creation of DSLs. Once these DSLs are created, they can be utilized within a company to work with components at a much high level, with a much higher level of software reuse.

Software Product Lines

Software Product Lines are entire subsets of components that can be configured, assembled, and packaged to provide a fairly complete product. The resulting product should only require customization from a developer for the highly specialized aspects of the project. Perhaps the largest component of a Software Factory, Software Product Lines not only provide the greatest value, but also require the greatest investment. Software must be carefully partitioned into distinct and reusable components and those components must readily fit together in a coherent manner. Configuration is the key to Software Product Lines, as projects must be able to pick and choose which components they want to utilize, and then generate an application off of that configuration.

Implementing a Software Factory While all of the promises of Software Factories sound appealing, many companies have tried to provide the tools and components, only to fail under the load of inflexible tools or proprietary formats.

All of this is about to change. Big-name companies like Microsoft and Sun are getting ready to release many of the components necessary for building and assembling a Software Factory within an organization. With the release of Visual Studio 2005, Microsoft will unveil several add-ins and plug-ins that enable the creation of not only Domain Specific Languages, but also the integration of those languages with the IDE itself. This will allow developers to manipulate and use the language from within the Visual Studio.NET IDE.

Not to be outdone, Sun Microsystems is working on its own implementation of Software Factory technology, simply named Project Ace. Although, very little details of “Project Ace’ are available, developers shouldn’t expect Sun to let Microsoft provide .NET tools, without answering with a comparable set of tools for Java.

What about now?

While all this conjecture sounds wonderful for the future, many developers will be asking themselves what they can do now to utilize Software Factory techniques in their organizations today. Well, the good news is that a lot of functionality already exists with Visual Studio.NET. Products like Enterprise Templates, Project Item Templates, and Nant allow for the creation of standard artifacts that a team or organization can utilize today.

Silicon Valley Startups: Low-risk R&D

Wired Reporting:

As the engineer and writer Alex Payne put it, these startups represent “the field offices of a large distributed workforce assembled by venture capitalists and their associate institutions,” doing low-overhead, low-risk R&D for five corporate giants. In such a system, the real disillusionment isn’t the discovery that you’re unlikely to become a billionaire; it’s the realization that your feeling of autonomy is a fantasy, and that the vast majority of you have been set up to fail by design.

The most expensive lottery ticket in the world

From Felix Salmon:

Founding a Silicon Valley startup, then, is a deeply irrational thing to do: it’s a decision to throw away a large chunk of your precious youth at a venture which is almost certain to fail. Meanwhile, the Silicon Valley ecosystem as a whole will happily eat you up, consuming your desperate and massively underpaid labor, and converting it into a few obscenely large paychecks for a handful of extraordinarily lucky individuals. On its face, the winners, here, are the people with the big successful exits. But after reading No Exit, a different conclusion presents itself. The real winners are the happy and well-paid engineers, enjoying their lives and their youth while working for great companies like Google. In the world of startups, the only winning move is not to play.

That’s fine for Griffin

More than once, for some reason, people have asked me for advice on a variety of subjects: life, business, technology, etc… When I try and answer them as best as I can, I’m honest for what works for me. Not necessarily what works for everyone, all the time.

Yet, oddly enough, when presented with recommendations for how to approach and solve problems, people tend to rebuff some things as fine for me, but it’s not scalable to everyone in the whole wide world. “What if everyone saved their money? The economy would collapse!”

That’s bullshit

Sure, a ton of people can get smart, save their money, run their own businesses & lose weight. But most people won’t. Your goal in your own life is to do the very best for you and the people you care about. Everyone else can take care of themselves. After all, you shouldn’t expect other people to have your best interests in mind. As the saying goes “no one cares more about your money than you.”

Lack of Engineering Talent

I recently attended a dinner event for a prominent university here where internal university updates are discussed. Filled with lecturers, deans, VPs and other state & local VIPs, it’s a pretty standard gathering of higher education people. At this years dinner, the president of the university was discussing the brand new data center. Proud of the fact that it’s the schools first LEED Platinum certified building, she wanted to call out the person responsible for the initiative. This is how she announced them:

“My resident computer geek”

I talk to a lot of people about the state of the technology industry, especially with regards to job opportunities. Whenever I point to the wealth of technical job openings that remain unfilled, people always ask “why is there such a lack of tech people?”. The reason I give surprised many and gets dismissed. It’s this:

It’s still not cool to be involved with computers.

Despite all the recent fame & success of technology / internet entrepreneurs, people still think of techies as the taped glasses & pocket protector wearing geeks from the movies. It’s very hard to imagine why the “D&D playing virgins, living in their parents basement” historical stereotype persists. Yet, here we are.

“My resident computer geek”

Its difficult to find a faster growing sector of the economy with higher earning potential than computers. Yet despite the existence of demand, the year over year growth of the sector, the massive unemployment rates of certain generations and the high earning potential of a computer based job, the supply is actually going down.

To explain this some people point to the relative newness of technology professions as a leading indicator. The narrative goes that we’ll see younger generations see the demand, flock to it, then start to fill it. This might have been true in the 80’s, 90’s or early 00’s. But we’re squarely in the 3rd decade (at least) of computers underpinning most of our daily lives. If timing was an issue, we would have seen an influx of grads after the mid-80’s, late 90’s or mid 00’s. Yet, enrollment is down in almost every single STEM major around the country. The rise in use of non-US talent is at an all time high as companies go to Central & South America, Europe & Asia for talent.

Note: I know that higher ed is not the only source of talent and training, but it’s a big one.

Something deeper is going on that is steering people away from the sector.

“My resident computer geek”

This wasn’t some football throwing jock, stuffing kids into garbage cans. This was the president of a major university. If anyone should be sensitive throwing around pejorative names, it should be her. The dismissive remarks seem to ignore just how quickly tech savvy people are lapping non-tech savvy people in terms of knowledge, business acumen, social mobility and plain economic power. To dismiss that section of the population is dangerous at best and ignorant at worst.

Among the other members introduced that night were lit professors, authors, pharmacists and CEOs. How many of those people do you think were reduced to a unflattering stereotype? She could have easily used stereotypes such as bookworms, alcoholics, med school dropouts and crooks to describe the other members mentioned above. But she didn’t.

“My resident computer geek”

After her speech, at the end of the dinner, she had a new recruitment video cued up to show everyone. As we sat watching, the video froze and stopped playing. Everyone in the room sat there, with no idea what to do. The person running the laptop could do nothing but click play / pause a few times before the president declared “we’re having technical difficulties”.

If only there was a computer geek around to help.

You’re probably half-assing your startup idea

I finally finished reading Paul Grahams prose on how to get startup ideas. The whole essay is incredible. I did want to call out a few critical nuggets that he touches on that I especially liked:

The verb you want to be using with respect to startup ideas is not “think up” but “notice”

And

When you have an idea for a startup, ask yourself: who wants this right now? Who wants this so much that they’ll use it even when it’s a crappy version one made by a two-person startup they’ve never heard of? If you can’t answer that, the idea is probably bad. [3]

In all honesty, this is enough to make you stop and nod your head ‘Yes!’. After 4 years running two startup groups here in Chicago, I got tired of saying this exact phrase. If you want to start a company, find someone already solving a problem they have, but doing it poorly. That’s really it. Of course, never confuse simple with easy.

Otherwise, you have to convince them that a) they have the problem, b) they should spend money to solve it and c) you’re the one to solve it. That’s a tough thing to do.

How about the incumbents you’re trying to displace?

When startups consume incumbents, they usually start by serving some small but important market that the big players ignore. It’s particularly good if there’s an admixture of disdain in the big players’ attitude, because that often misleads them.

Make no mistake, someone is making money “solving” the problem you’re trying to solve with your startup. This isn’t kindergarden. You’re trying to take money that would otherwise go to them. They’re not gonna like that. You have to move quick, fight dirty & scrappy and use your quickness to your advantage. It’s up to you to figure out exactly how to do that, otherwise, you’re screwed. Don’t whine if you can’t understand why you just can’t catch a break. Nobody is owed a business model.

Lastly, on idea sexiness:

In fact, one strategy I recommend to people who need a new idea is not merely to turn off their schlep and unsexy filters, but to seek out ideas that are unsexy or involve schleps. Don’t try to start Twitter. Those ideas are so rare that you can’t find them by looking for them. Make something unsexy that people will pay you for.

There are so many ideas out there just waiting to be implemented. The problem is that they’re not sexy enough. They’re not going to make the WSJ or NYT. They are micro-opportunties & little buckets of gold just waiting for someone to pickup and run with. Find your flywheel

The “Jobs Creation” Fallacy

Felix Salmon chimes in on the Startup Act 3.0 in his latest post. The focus is on the so called immigrant-entrepreneur visa:

One is the immigrant-entrepreneur visa; the second is the idea of giving green cards to up to 50,000 foreign students who graduate from an American university with an advanced degree in science, technology, engineering, or mathematics — so long as they remain in that field for five consecutive years.

This is a very very very good thing. Our bleed to other countries in technical fields is quickly turning into a hemorrhage. However, he also makes a assumption that trips up most people: the lure of Job Creation.

And of course — by definition — it would create jobs. The Kauffman foundation’s math is solid, here: they conservatively estimate job creation at somewhere between 500,000 and 1.6 million new jobs after ten years, and possibly substantially more. (Those estimates don’t include jobs created by the new firms after they’ve left the program, for instance.)

Now, some of these entrepreneurs will start companies in the classics: education, healthcare, manufacturing, etc.. However, the majority will be tech related startups. And here’s the rub: the required employees of tech startups are not the ones sitting around unemployed. There’s a major shortage in knowledge workers of all kinds, from engineers on down. So simply creating more available jobs is not actually going to help out of work people get work. They already have work.

This visa does nothing to stimulate the creation of supply for these jobs, only demand. In a way, this will actually hurt the economy a bit because you’re adding to the price war going on for technical workers. This drives prices up to levels only large companies can afford, pushing out the smaller, scrappy entrepreneurs. The rich get richer.

If the government really wants to stimulate job creation, in addition to the above visa, they should work on increasing supply. But how? Simple: subsidize the education ( college or otherwise ) of anyone getting a degree / certification in a field that’s among the highest demand for the past 3-5 years. Computer Science, Electrical Engineering, Software Development, etc..

With the sky rocketing costs of tuition, people are facing the choice of education or no education. By offering them alternative situation ( Major in communications and take out loans or Major in Technical Writing and go for free ) you’re also reducing the debt load of an entire generation of teenagers who would otherwise die in debt.

A week with my new favorite watch by @Hodinkee

NOMOS Zurich

This watch is the type of watch that should be worn by a man who travels the world and thinks nothing of it; a man who is at home in Zurich, Hong Kong, Chicago, and Santiago, and knows the best places to eat in each without having to use his iPhone. It was made for the type of person who reads Monocle Magazine not to impress people on the train, but who genuinely cares about stalwarts of sustainable design in an obscure Scandinavian city. This watch is for a man who appreciates that fact that this watch features an in-house manufacture movement with hand-finishing, but doesn’t need everyone around him to know how much he paid for it. The NOMOS is a watch for a man who knew exactly who Nick Horween was before he saw that this watch came on Horween leather.

Banksy distills our upcoming revolution

This profile of Banksy outlines why he might just be remembered as one of the most influential people of the 21st century.

On indie artists:

While he may shelter behind a concealed identity, he advocates a direct connection between an artist and his constituency. “There’s a whole new audience out there, and it’s never been easier to sell [one’s art],” Banksy has maintained. “You don’t have to go to college, drag ’round a portfolio, mail off transparencies to snooty galleries or sleep with someone powerful, all you need now is a few ideas and a broadband connection. This is the first time the essentially bourgeois world of art has belonged to the people. We need to make it count.”

We may very well look back at Banksy as the catalyst for the upcoming indiepocalypse

On capitalism:

I love the way capitalism finds a place—even for its enemies. It’s definitely boom time in the discontent industry. I mean how many cakes does Michael Moore get through?”

Finally, a bit of sarcasim:

“Hollywood,” he once said, “is a town where they honor their heroes by writing their names on the pavement to be walked on by fat people and peed on by dogs. It seemed like a great place to come and be ambitious.”