Software Factories – Refactoring An Industry

(I’m republishing an old article I had written about 10 years ago that seems to have disappeared from the web. This was originally published in 2006 on TheServerSide.net: http://www.theserverside.com/discussions/thread.tss?thread_id=32812 The fact that the issues described below have not really changed in 8 years is, frankly, sad.)

Software development has always been costly and time-consuming process. Specialized requirements and lack of skilled resources are just two of the difficulties facing companies today. Pressure to deliver software, on time and within budget, have pushed developers to look for a way to increase value delivered, while decreasing development time.

For many years now, efficient reuse of existing assets, either through object-oriented programming, component-based development or patterns-based architecture, has been one of the core objectives for the IT industry as a whole. Software reuse is seen as a means to combat many of the problems facing development teams. However, for many years and several different technology paradigms, this level of reuse has eluded the industry as a whole.

The book Software Factories: Assembling applications with Patterns, Models, Frameworks, and Tools aims to change this by modifying the definition of reuse as used by the IT industry and bring a more manufacturing approach to software reuse.

Economies of Scale vs. Economies of Scope

The difference between these two definitions are subtle and is something that has been proven outside of the software industry, in more established industries like manufacturing and fabrication. While both promise to reduce time and cost and improve product quality, they are actually quite different.

Economies of scale occurs when the initial development of a design and subsequent construction of a prototype result in multiple copies of that prototype are created. This is similar to the fabrication industry when a custom part is created and that part is used to produce hundreds or thousands of similar parts. In the manufacturing industry, it’s similar to a machine being built that can produce screws in bulk. In this case, ruse occurs in the later stage of production, namely construction.

Economies of scope occurs when multiple similar, but unique designs are produced in groups, as opposed to individually. Authors Jack Greenfield et.al uses the example of a car manufacturing plant that uses existing designs, such as chassis, body and interiors, to produce several lines of distinct cars. Each product line is complete with custom features, but all share that same underlying design. Here the reuse occurs earlier in production, namely design and prototyping.

Greenfield et.al point out that for years, the IT industry has been trying to apply economies of scale for achieving software reuse, when in fact, the IT industry should have been trying to apply economies of scope to achieve reuse.

Chronic Problems of Software Development

The authors of Software Factories identity some major problems with how the IT industry is attempting to achieve reuse. Unfortunately, simply altering how software reuse is applied within an industry or an organization won’t make systematic reuse a reality. Greenfield et.al identify four chronic problems with software development, each of which impede a teams’ ability to gain valuable insight and reuse from existing domain knowledge and experience. These problems are: •

  • Monolithic Development
  • Copious Granularity
  • Process Immaturity
  • One-off Developer

Monolithic Development

Monolithic development means the creation of software in such a way as to make it difficult or even impossible, to utilize the resulting artifacts outside of a narrowly defined scope. Software development teams within large organizations are certainly familiar with this style of development. Several projects, several teams, all building isolated applications, without concern for what anyone else may be doing, or what, if any, domain knowledge they are embedding within the application. This results in large, inflexible applications that are of little use to anyone outside the original, intended audience. Even two projects within a single organization can be faced with a mini-integration effort in order for one application to take advantage of another’s functionality. Why does this keep occurring within organizations? Industry leaders have championed design by assembly for years, and yet very few applications take advantage of existing components, even within their own organization.

Copious Granularity

Typical software development for business applications consists of a strikingly similar set of features and functionality. For example, many business applications read data from a database, present it to the user, allow the user to modify the data in some way, and then allow the user to persist the change back to the database. Even though this is an over simplification of most applications, the basics still remain the same across projects. It begs the question of why developers use such fine-grained tools as standard programming languages like C# and VB .NET for representing such basic patterns? Part of the reason is the immaturity of modeling tools and languages. While a language such as UML is suitable for documenting software architecture, it is inadequate for allowing implementation to be derived from such models. UML lacks the extensibility required for generating large amount of code, and also lacks the breadth required to represent all aspects of software, including databases and user interfaces.

Process Immaturity

  • Controlling complexity at the expense of change — Many “traditional” processes would fall under this heading, including RUP and Waterfall
  • Controlling change at the expense of complexity — Most “agile” methodologies would fall under this heading, including Scrum and XP Before a process can be tuned for software reuse, it must mature because automation can only automate well-defined processes.

One-off Development

Software development projects within a company are usually so focused on the bottom line and delivery time, that overall architecture is placed on the back burner as “lofty” or “academic”. Very little regard is taken to analyze and evaluate existing assets, and very little time is dedicated to ensuring that new assets produced are reusable within other contexts. This results in many development efforts within a single company that create various amounts code and valuable assets for use within the company’s domain. Projects rarely perform post-mortems where reusable components are identified, documented, and packaged in such a way as to become reusable by other projects. These four problems, as well as the industrys’ misunderstanding of how to apply reuse with the current economic model, paint a pretty bleak picture as to the future of software development as an industry, especially compared to other more mature industries, such as manufacturing. Luckily, this is where the Software Factory comes into play. The next section contains a brief overview of what is a Software Factory is and identifies some of its main components.

Software Factories: The Solution?

When creating software within a specific vertical, knowledge about that vertical is frequently embedded in the software being developed. Unfortunately, this knowledge remains hidden inside the software; unable to be reused outside of the original scope of the software that contains it. This means reuse of this knowledge is next to impossible. This is where Software Factories look to step in with methods and procedures to harvest that knowledge, turning that knowledge into reusable production assets for a company. A Software Factory is defined by Greenfield, et al., as a software product line that configures extensible tools, processes, and content using a software factory template based on a software factory schema to automate the development and maintenance of variants of an archetypical product by adapting, assembling and configuring framework-based components. In layman’s terms, a Software Factory are about collecting existing, specialized knowledge about a certain domain and applications built within that domain. You can then use that knowledge to create a blueprint for other applications of a similar type. That schema can then be tweaked and configured to produce semi-functional to functional applications. In short, Software Factories means application generation from valuable, and reusable production assets that exist within an organization. Before delving into what makes components comprise a Software Factory, let’s first take a look at a look at the current state of the IT industry as compared to other industries.

Moving from Craftsmanship to Industrialization

All too often, highly skilled application developers and architects have to use their time for low-level, implementation level tasks. Usually, junior developers are not able to complete such tasks because of lack of appropriate domain knowledge, requiring the senior developer to mentor the junior developer. This fosters not only knowledge transfer, but also an introduction to the complexities of the current development environment. Since developers are always involved at some stage of development, very little time is spent in making development more efficient, especially low-level implementation details. This method of development resembles early goods based industries, where single workers create custom solutions, tailored to each individual requirement. Think early tailors or shoe cobblers.

This craftsmanship approach to software development does not scale very well. The lack of quality senior developers creates a mentoring environment, where specialized knowledge must be transferred, similar to an apprenticeship. Since there is such a hands-on approach required, each part of the project need to be created, most of the time, by hand. This often leads to higher quality, but also leads to performance and efficient issues. Migrating from a craftsmanship-based industry to a more industrial-based industry has been the path of progression for many more mature industries. If this is the end result for so many other industries, why is software development still based on small groups of highly specialized craftsmen?

Most people within the IT industry will agree that a form of standardization and modularization is the key to enabling the kind of reuse required for efficient industrialization of software development. What they don’t agree on is the means to which this standardization and modularization is achieved. The Software Factory aims to address this effort by prescribing a process by which software can be modularized into reusable assets.

Components of Software Factories

There are 3 main components of Software Factories:

  • Models and Patterns
  • Domain Specific Languages
  • Software Product Lines

Models and Patterns

The authors of Software Factories define Model-Driven Development as:

using models to capture high level information, usually expressed informally, and to automate its implementation, either by compiling models to produce executables, or by using them to facilitate the manual development of executables.

The importance of models comes from their ability to keep a consistent representation of concepts within a project. For example, while it’s easy to draw and manipulate a person object in a model, it’s much more difficult to manipulate the same person object at a lower level, because the person object could be represented by class files, tables, and columns in a database.

Representing and manipulating core abstractions and concepts within a software system is only half the battle, though. The other half comes from being able to effectively use those models to generate the underlying implementation details represented by the model

Domain Specific Languages

Domain Specific Languages, or DSLs, have long been a way to provide an abstraction atop existing concepts within a specialized domain. Examples of a DSL include SQL and HTML. Both provide specialized languages for manipulating concepts within their respective domain — tables, rows, and columns in the case of SQL and elements, tables, and forms in the case of HTML. For years DSLs like these remained the only cost effective DSLs because of their wide spread use and generalized concerns. No matter what your specific project entails, if you use a database, you can use SQL and if you use web pages, you can HTML. Creating DSLs for other, more vertical concerns has always been cost prohibitive because of the lacks of tools.

However, several companies have recently announced tools or plug-ins for the creation of DSLs. Once these DSLs are created, they can be utilized within a company to work with components at a much high level, with a much higher level of software reuse.

Software Product Lines

Software Product Lines are entire subsets of components that can be configured, assembled, and packaged to provide a fairly complete product. The resulting product should only require customization from a developer for the highly specialized aspects of the project. Perhaps the largest component of a Software Factory, Software Product Lines not only provide the greatest value, but also require the greatest investment. Software must be carefully partitioned into distinct and reusable components and those components must readily fit together in a coherent manner. Configuration is the key to Software Product Lines, as projects must be able to pick and choose which components they want to utilize, and then generate an application off of that configuration.

Implementing a Software Factory While all of the promises of Software Factories sound appealing, many companies have tried to provide the tools and components, only to fail under the load of inflexible tools or proprietary formats.

All of this is about to change. Big-name companies like Microsoft and Sun are getting ready to release many of the components necessary for building and assembling a Software Factory within an organization. With the release of Visual Studio 2005, Microsoft will unveil several add-ins and plug-ins that enable the creation of not only Domain Specific Languages, but also the integration of those languages with the IDE itself. This will allow developers to manipulate and use the language from within the Visual Studio.NET IDE.

Not to be outdone, Sun Microsystems is working on its own implementation of Software Factory technology, simply named Project Ace. Although, very little details of “Project Ace’ are available, developers shouldn’t expect Sun to let Microsoft provide .NET tools, without answering with a comparable set of tools for Java.

What about now?

While all this conjecture sounds wonderful for the future, many developers will be asking themselves what they can do now to utilize Software Factory techniques in their organizations today. Well, the good news is that a lot of functionality already exists with Visual Studio.NET. Products like Enterprise Templates, Project Item Templates, and Nant allow for the creation of standard artifacts that a team or organization can utilize today.

That’s fine for Griffin

More than once, for some reason, people have asked me for advice on a variety of subjects: life, business, technology, etc… When I try and answer them as best as I can, I’m honest for what works for me. Not necessarily what works for everyone, all the time.

Yet, oddly enough, when presented with recommendations for how to approach and solve problems, people tend to rebuff some things as fine for me, but it’s not scalable to everyone in the whole wide world. “What if everyone saved their money? The economy would collapse!”

That’s bullshit

Sure, a ton of people can get smart, save their money, run their own businesses & lose weight. But most people won’t. Your goal in your own life is to do the very best for you and the people you care about. Everyone else can take care of themselves. After all, you shouldn’t expect other people to have your best interests in mind. As the saying goes “no one cares more about your money than you.”

Lack of Engineering Talent

I recently attended a dinner event for a prominent university here where internal university updates are discussed. Filled with lecturers, deans, VPs and other state & local VIPs, it’s a pretty standard gathering of higher education people. At this years dinner, the president of the university was discussing the brand new data center. Proud of the fact that it’s the schools first LEED Platinum certified building, she wanted to call out the person responsible for the initiative. This is how she announced them:

“My resident computer geek”

I talk to a lot of people about the state of the technology industry, especially with regards to job opportunities. Whenever I point to the wealth of technical job openings that remain unfilled, people always ask “why is there such a lack of tech people?”. The reason I give surprised many and gets dismissed. It’s this:

It’s still not cool to be involved with computers.

Despite all the recent fame & success of technology / internet entrepreneurs, people still think of techies as the taped glasses & pocket protector wearing geeks from the movies. It’s very hard to imagine why the “D&D playing virgins, living in their parents basement” historical stereotype persists. Yet, here we are.

“My resident computer geek”

Its difficult to find a faster growing sector of the economy with higher earning potential than computers. Yet despite the existence of demand, the year over year growth of the sector, the massive unemployment rates of certain generations and the high earning potential of a computer based job, the supply is actually going down.

To explain this some people point to the relative newness of technology professions as a leading indicator. The narrative goes that we’ll see younger generations see the demand, flock to it, then start to fill it. This might have been true in the 80’s, 90’s or early 00’s. But we’re squarely in the 3rd decade (at least) of computers underpinning most of our daily lives. If timing was an issue, we would have seen an influx of grads after the mid-80’s, late 90’s or mid 00’s. Yet, enrollment is down in almost every single STEM major around the country. The rise in use of non-US talent is at an all time high as companies go to Central & South America, Europe & Asia for talent.

Note: I know that higher ed is not the only source of talent and training, but it’s a big one.

Something deeper is going on that is steering people away from the sector.

“My resident computer geek”

This wasn’t some football throwing jock, stuffing kids into garbage cans. This was the president of a major university. If anyone should be sensitive throwing around pejorative names, it should be her. The dismissive remarks seem to ignore just how quickly tech savvy people are lapping non-tech savvy people in terms of knowledge, business acumen, social mobility and plain economic power. To dismiss that section of the population is dangerous at best and ignorant at worst.

Among the other members introduced that night were lit professors, authors, pharmacists and CEOs. How many of those people do you think were reduced to a unflattering stereotype? She could have easily used stereotypes such as bookworms, alcoholics, med school dropouts and crooks to describe the other members mentioned above. But she didn’t.

“My resident computer geek”

After her speech, at the end of the dinner, she had a new recruitment video cued up to show everyone. As we sat watching, the video froze and stopped playing. Everyone in the room sat there, with no idea what to do. The person running the laptop could do nothing but click play / pause a few times before the president declared “we’re having technical difficulties”.

If only there was a computer geek around to help.

You’re probably half-assing your startup idea

I finally finished reading Paul Grahams prose on how to get startup ideas. The whole essay is incredible. I did want to call out a few critical nuggets that he touches on that I especially liked:

The verb you want to be using with respect to startup ideas is not “think up” but “notice”

And

When you have an idea for a startup, ask yourself: who wants this right now? Who wants this so much that they’ll use it even when it’s a crappy version one made by a two-person startup they’ve never heard of? If you can’t answer that, the idea is probably bad. [3]

In all honesty, this is enough to make you stop and nod your head ‘Yes!’. After 4 years running two startup groups here in Chicago, I got tired of saying this exact phrase. If you want to start a company, find someone already solving a problem they have, but doing it poorly. That’s really it. Of course, never confuse simple with easy.

Otherwise, you have to convince them that a) they have the problem, b) they should spend money to solve it and c) you’re the one to solve it. That’s a tough thing to do.

How about the incumbents you’re trying to displace?

When startups consume incumbents, they usually start by serving some small but important market that the big players ignore. It’s particularly good if there’s an admixture of disdain in the big players’ attitude, because that often misleads them.

Make no mistake, someone is making money “solving” the problem you’re trying to solve with your startup. This isn’t kindergarden. You’re trying to take money that would otherwise go to them. They’re not gonna like that. You have to move quick, fight dirty & scrappy and use your quickness to your advantage. It’s up to you to figure out exactly how to do that, otherwise, you’re screwed. Don’t whine if you can’t understand why you just can’t catch a break. Nobody is owed a business model.

Lastly, on idea sexiness:

In fact, one strategy I recommend to people who need a new idea is not merely to turn off their schlep and unsexy filters, but to seek out ideas that are unsexy or involve schleps. Don’t try to start Twitter. Those ideas are so rare that you can’t find them by looking for them. Make something unsexy that people will pay you for.

There are so many ideas out there just waiting to be implemented. The problem is that they’re not sexy enough. They’re not going to make the WSJ or NYT. They are micro-opportunties & little buckets of gold just waiting for someone to pickup and run with. Find your flywheel

The “Jobs Creation” Fallacy

Felix Salmon chimes in on the Startup Act 3.0 in his latest post. The focus is on the so called immigrant-entrepreneur visa:

One is the immigrant-entrepreneur visa; the second is the idea of giving green cards to up to 50,000 foreign students who graduate from an American university with an advanced degree in science, technology, engineering, or mathematics — so long as they remain in that field for five consecutive years.

This is a very very very good thing. Our bleed to other countries in technical fields is quickly turning into a hemorrhage. However, he also makes a assumption that trips up most people: the lure of Job Creation.

And of course — by definition — it would create jobs. The Kauffman foundation’s math is solid, here: they conservatively estimate job creation at somewhere between 500,000 and 1.6 million new jobs after ten years, and possibly substantially more. (Those estimates don’t include jobs created by the new firms after they’ve left the program, for instance.)

Now, some of these entrepreneurs will start companies in the classics: education, healthcare, manufacturing, etc.. However, the majority will be tech related startups. And here’s the rub: the required employees of tech startups are not the ones sitting around unemployed. There’s a major shortage in knowledge workers of all kinds, from engineers on down. So simply creating more available jobs is not actually going to help out of work people get work. They already have work.

This visa does nothing to stimulate the creation of supply for these jobs, only demand. In a way, this will actually hurt the economy a bit because you’re adding to the price war going on for technical workers. This drives prices up to levels only large companies can afford, pushing out the smaller, scrappy entrepreneurs. The rich get richer.

If the government really wants to stimulate job creation, in addition to the above visa, they should work on increasing supply. But how? Simple: subsidize the education ( college or otherwise ) of anyone getting a degree / certification in a field that’s among the highest demand for the past 3-5 years. Computer Science, Electrical Engineering, Software Development, etc..

With the sky rocketing costs of tuition, people are facing the choice of education or no education. By offering them alternative situation ( Major in communications and take out loans or Major in Technical Writing and go for free ) you’re also reducing the debt load of an entire generation of teenagers who would otherwise die in debt.

Banksy distills our upcoming revolution

This profile of Banksy outlines why he might just be remembered as one of the most influential people of the 21st century.

On indie artists:

While he may shelter behind a concealed identity, he advocates a direct connection between an artist and his constituency. “There’s a whole new audience out there, and it’s never been easier to sell [one’s art],” Banksy has maintained. “You don’t have to go to college, drag ’round a portfolio, mail off transparencies to snooty galleries or sleep with someone powerful, all you need now is a few ideas and a broadband connection. This is the first time the essentially bourgeois world of art has belonged to the people. We need to make it count.”

We may very well look back at Banksy as the catalyst for the upcoming indiepocalypse

On capitalism:

I love the way capitalism finds a place—even for its enemies. It’s definitely boom time in the discontent industry. I mean how many cakes does Michael Moore get through?”

Finally, a bit of sarcasim:

“Hollywood,” he once said, “is a town where they honor their heroes by writing their names on the pavement to be walked on by fat people and peed on by dogs. It seemed like a great place to come and be ambitious.”

Incredible Marketing Device

While walking in NYC this weekend, I noticed the Camper SOHO store had an incredible hook for bringing people in. This is the street view of the store as you’re walking past it:

camper street view soho

No products in plain view. The store view is this:

camper store view soho

How freaking cool is that? You get so intrigued by the display you have to go into the store to take a peek. Really cool execution.

My Mantra

I was reading through the interview Offscreen did with Drew Wilson for their first issue when I stumbled upon this quote:

Everything I’ve ever built was built because I wanted it to exist so I could use it.

Replace “built” with “done” and you have the response I give to everyone when they ask me why I did something.

Seth Kravitz of Technori fame once asked my why I started the Boostrappers Breakfast in Chicago. He assumed, like most people, that I get some leads, work, etc… from it. My response was simply

No, I don’t get anything from this, besides some social capital. I did this because it needed to exist.

A bit overwrought and dramatic, but it’s the truth. It’s easy to work through the kinks and do something when you’re the audience for it. Your standards will also be higher.

20 Questions

I was reading the latest issue of Worth magazine & saw something that caught my eye. Their last page is a Playboy style 20 questions with an entrepreneur. I thought the questions were good, so I decided to do my own for myself. I’ll revisit it every few years and see if my answers change.

Favorite Entrepreneur?
Warren Buffet
Favorite Restaurant
Mercadito
Wine, Beer or Spirits?
Spirits. I’ve been a tequila drinker for about 13 years. Chinaco is my favorite. If not that, then a nice Pinot Noir
Favorite City in the U.S.?

Chicago aside, it has to be NYC.

Globally?
Porto Alegre, Brazil for obvious reasons.
How many days do you spend on the road?
Up until recently, it’s been about 20 for business. For pleasure, I’ve been averaging about 21 days
Private or Commercial
Ha…. commercial of course with as much leg room as I can afford.
How much do you sleep?
6 hours minimum, but I try and get 7 most nights.
What’s on your nightstand?
My kindle. Queued up are a lot of books about management, lean/kanban and other geeky topics
Last music you listened to?
Watchu want from me?” – Homeboy Sandman
First website you check everyday?
Finance, banking & news websites. Varies from day to day
Favorite clothing?
It’s a splurge, but has to be my Barbour jacket from Jack Spade. Fits like a glove and works great. Will last me decades.
What do you wear to work?
Jeans, nice shoes & a polo or sweater. Occasionally a button down, but always untucked & sleeves rolled. I brag that the first suit I owned was for my wedding.
Unjustifiable extravagance?
Has to be my vinyl toy collection. I’m really into some of the underground artists like Huck Gee, Carson Catlin, Task Oner, MAD, and VISEone.
Your Watch?
I used to rotate among 6-7 different ones, but since Nov 11th, I only wear one. I bought a Tissot Seastar 1000 for my wedding and I cherish it. It’s my wedding band.
Your Hero?
My dad Carmen.
Your business philosophy?
Doesn’t really differ from my life philosophy: Work hard & chop that wood. When the odds are in your favor, bet and bet big.
What charities do you support?
Several privately. I don’t really like to draw attention. I will say that my willingness to help out is directly proportional to my contact with the recipient. I don’t like foundations that much.
Investment philosophy?
With a few SWAG (Silver, Wine, Art, Gold) exceptions, diversified index funds. I like to balance between domestic, international & bond based.
Best entrepreneurship tip?
Stay away from ‘lotto’ opportunities where you sacrifice yourself for someone else. Everything you do should equally reward you and not unequally reward someone else.