This is my third draft: Thursday 5^th Sept. 1996 TG. Edit 11sept 3.1

Evolutionary Object Management

Introduction.

The Evolutionary method.

Evolutionary ('Evo' for short) project management is a process based on 'feedback' and 'learning how to reach objectives', and even 'what those objectives realistically should be'. Evo is a first cousin to the famous Plan-Do-Study-Act paradigm from the Statistical Process Control method, espoused by the quality guru's W. Edwards Deming (see 'Out of The Crisis' MIT Press by Deming), Joseph Juran and Walter Shewhart. Made most successful by the Japanese use of it to build their industry..

Evo interests us today primarily because it has exhibited ability to successfully cope with complex large scale systems development, where traditional 'Waterfall' Methods have failed.

As early as the 1970 to 1980 decade, Harlan Mills of IBM Federal Systems Division reported that all FSD projects using the method were delivered on time and under budget, to the highest state of the art quality levels [Gilb88, and IBM Systems Journal No. 4 1980) for many projects, and for years on end. Now the US Department of Defense has learned this lesson, at an estimated cost of $400 (Four hundred) Billion lost on 'Waterfall' management software projects, currently annually losing $20 (Twenty) Billion (50% of current investment levels) according to a private, but centrally placed DoD source of mine. The US Air Force has made it official that Critical Software projects are not allowed to use the Waterfall Model! [Gilb96, 9.0) and the new DoD Software Project Standard (MIL STD 498 (1994) and consequent civil versions replacing it) explicitly allows for evolutionary project management and object-oriented delivery. This standard is available via the same web site as listed at the end of this paper (STSC). Leading Corporations, notably Hewlett Packard, are investing in teaching their employees Evolutionary combined with Object Oriented technology [HP, MAY96, COTTON96]

The Supporting Planguage framework.

I have used primitive intuitive versions of Evo since I designed and implemented my first systems about 1959. Maybe you have used the method intuitively too! What I can offer is a disciplined support method so that you get more out of the Evo method. I have described the method and supporting ideas in public for at least two decades. The most detailed public description is in 'Principles of Software Engineering Management' [[POSEM88] , where the theory of the method from my point of view was expounded, and a considerable literature summary was given. It is from this book, and its happy reception amongst the UK Object Oriented Community, that I have been invited to give my latest views on the subject in this article.

It is important at the outset to understand that there are two basic Evo components:

1. The Evo process itself
2. The supporting methods and languages to direct and control the process.

These are described in the '[POSEM88]' reference in detail. But I can offer the reader a substantial and free upgrade to the method and discipline. Basic information here, and details on web sites in the form of my new book manuscript 'Planguage' [Planguage96].

Part One: The Evolutionary Concepts

Comparison to other project management methods

Evo is based on the assumption that you don't know everything, because you cannot know everything about requirements, design , costs, timing, and risks. Evo is based on the idea that you are going to have to 'feel your way'. You are doing something new with technology, culture and economics. You do not want to take the risk of spending a lot of money and time, and then failing, because of some detail which nobody had insight into. Or which nobody believed was risky.

So, Evo is very different from 'Waterfall' project models, in which things goone way from all requirements to all design to all construction and one big bang delivery. Those models are fine under certain circumstances of 'just another bridge'. We know what we want, we know the technology, we have done it before. We just need another one.

Evo is based on the idea that we think we know 'approximately' what we want. That we know 'approximately' how to get there. That everybody could change their mind during the project. That we are willing to learn safely and at low cost, early, as we still achieve some results, how to satisfy our users and customers. We are willing to learn what things really cost, and how much 'other people' value those qualities and functions. We are willing to do small 2% of budget experiments to see how new technology really works, or how it is accepted by real customers and system users. If an experiment fails, we will declare success, at learning how not to do things. Good science that!

Comparison to other evolutionary methods.

Now, we have many methods available to us which are cautious, some purely individual common sense and intuitive, some formally taught methods with proper names on them. I cannot in this short paper explain them all, but the reader might try [POSEM88] and Dorothy Grahams classic paper [Graham90]. To the software engineering professional the names will be familiar, 'incremental', Spiral', 'phased', 'prototyping', 'Cleanroom', and variations on those cultures. They all have in common that not everything is delivered at once, that there is some partial delivery concept, some potential for learning something before all resources are committed. The Evo method is very close to the IBM-developed 'Cleanroom' form of evolutionary delivery. But it is not identical. In short, I believe that Evo is the most advanced of all these methods. Why? The total reasoning is complex and will be found in the references. But, the short answer is that Evo is completely devoted to 'systematic achievement of multiple quantified critical quality and cost attributes of any system'. Most all of the above-mentioned other methods are totally limited to 'software' as they are taught and practiced. They are also primarily devoted to some mystical implementation of 'functionality' with little or no regard for any resource or quality. They are centred around the notion of 'programming', whereas Evo is centred around the notion of giving the 'customer and user' 'what they need' at a 'price they can afford'. Evo is a 'systems' approach, for even 'software' systems use lots of hardware and peopleware and dataware. 'All' problems are not solved with the 'right algorithm'.

How, briefly does Evo work?

1. REQUIREMENTS: All critical quality (in widest sense, including performance) and cost objectives are stated quantitatively. Most people are incapable of doing this. They cannot articulate quality quantitatively. This is one decisive advantage, Evo includes and teaches this capability.
2. DESIGN: Design is made and estimated, with a safety margin, to be able to reach all requirements levels of quality within all constraints of time, people, money and other limitations (legal, ethical, political etc.). Most people have no tools for estimating the impact of their designs on their objectives. So Evo has another decisive advantage in showing exactly how to do this with 'Impact estimation' tables.
3. PROJECT MANAGEMENT: The delivery of results to customers and users is planned in approximately 2% increments, with the 'most valuable' results earliest, wherever possible. Most people do not know how to decompose a system into 2% delivery stages. Many will assume it is impossible. Others will not be asked to try, or expected by management to achieve such a decomposition. Note that the decomposition is not into 'building components', but into holistic useful system increments, which are useful to real customers and users. This is a teachable discipline (see 'How to Decompose Systems'), and it must be taught to many people, while for others it is intuitive.

2% increments means weekly for a years total project. 2% increments means maximum 2% risk of failure. 2% increments means 98% earlier results, maybe the 20% value the customers most care about is delivered in 4% or 10% of the time and budget. They don't have to wait 100%, and then maybe for a delay of another 214% for the whole thing to finish, just to get the most vital changes or improvements to their lives.

[editing note, you could insert the page from Planguage 9.6 'How to decompose Systems into small evolutionary steps' here as a sidebar. It is included at the end.].

1. CONTINUOUS EVERYTHING: Once the Evo cycles have begun, the entire 'life cycle' is exercised repeatedly until no more goals are to be achieved, or no more resources remain to achieve them.

This means that every cycle, sets quality and cost objectives for the cycle, every cycle adjusts long term quality, cost, and time objectives, every cycle adjusts designs which are not working well as hoped for, every cycle tests systems in the field ( probably under safe trial circumstances), every cycle does some construction, every cycle does some training, and every cycle does some deployment to wider geographic areas.

Why is Evo interesting for Object Oriented Culture?

Object oriented systems are more focused on getting results, than building components. An object oriented culture is delighted to rapidly deploy a reusable component, and get the system working or improved. Evolutionary delivery has the same focus. It is less interested in 'doing programming' and totally focused on getting the results to the users.

Evo needs open-ended systems, meaning systems which are easily adaptable to unforeseen and changing circumstances. It is the nature of Evolutionary delivery that it needs to be a fast 'learner' and a fast 'changer'. Evo needs building technology which enables that, so Object Orientation is the natural partner. Object Orientation is the 'construction technology' best-suited for getting rapid results, in changing unpredictable circumstances. Evo is the management process which can make good use of the OO construction technology. It is as if man met woman, natural partners are attracted. So, I hope the OO technology professionals will be interested in the Evo process, and they related goodies (below) as a way of more intelligently exploiting the capability offered by the OO technology.

Part Two: The Planguage Concepts

A major feature of my version of evolutionary delivery is the fact that it is primarily focused on delivering user improvements, in qualities and costs. The key to this is the ability to quantitatively articulate any interesting qualitative idea. The second key to this is to be able to estimate the quantified multiple impacts expected, at each step of evolution, during planning and design stages. This can then be used to manage the Evo project by measuring reality as it emerges, against the plans, and making correction when reality dictates doing so, in order to reach a reasonable set of goals. I have designed and published a special powerful new language, I call it 'Planguage' which can be used to do the requirements specification, design analysis, and project management of results required.

It is worth noting that this same Planguage might be considered by OO specialists as a potential tool for describing objects, and systems of objects. In particular, I have noted that most object cultures seem to totally lack a means to describe the qualities and costs of the objects. But these are essential elements of making appropriate selection of objects. Maybe we are at such a primitive stage of building object libraries, so we are happy to get objects that do the basic function 'at all'? But there should come a time when we have a variety of objects with identical function and will need to choose amongst them based on multidimensional quality and costs aspects. It is then vital that we can articulate our needs, and compare these to the object descriptions. This is no more than is expected in any market of similar goods.

Requirements Specification

The Planguage for requirements specification is a rich set of concepts, defined parameters and graphical symbols. A simple quality requirement specification looks like this.

Adaptability:

GIST: Ease of reacting to new market needs, compared to competitors.

SCALE: Elapsed days from new product concept until first product in shops.

METER [European Market] At least 5 representative products or product variations from each European County which were put on the shelves during the last calendar year, will be investigated to develop our current average time to market metric.

PAST [Europe, 1996] 396 days average Marketing manager, guess.

PLAN [next year] 200 days average, [year after next] 100 days average.

This planning language can be used to articulate all interesting or critical attributes of an object. The main point is that you are forced to be very explicit and clear, and to define and agree the meaning of concepts. No more 'vastly enhanced adaptability in software development' phrases.

Latest HP Fusion Development Attitude Towards Evo and Requirements.

Here is a quote from an email I received from Todd Cotton (Aug 20th1996) [HP, MALAN95] which points to the importance of this above type of specification from the “Fusion" method point of view.

Design Specification

Design specification needs to be made to satisfy the complete formal set of quality, attribute, cost and constraint requirements. Each design idea needs to be added to all others, and to fit in; to increment all still-lacking qualities, in the direction of the 'formally planned' levels of quality requirements, without exceeding cost requirements, time requirements or other constraints. The major problem is simply that people do not specify a design in enough detail to permit understanding its properties. This lack of detail does not become obvious until people are challenged to estimate impacts of an idea accurately. We insist on such accurate estimation, and quickly discover through both Document Quality Control (see below), and Evolutionary feedback from reality, that people do not know what they are talking about. They do not understand the consequences of their too-vaguely worded designs. The Planguage contains many expressions and concepts which permit us to specify designs far more precisely than we are used to. For example:

Software: ……

Database:………….. “not filled out in detail here"

Logic:

Applications:

Command and Control [NATO, Airborne]

Transport Aircraft [Jet, IF Russia = NATO-Member] Use <Mil Std 149>? - Joint Chiefs. [Propeller, All Full Members] Use {J-STD-016 | ISO 12207} NATO C.

Interfaces: ………………. “not filled out in detail here"

Other Software:……………… “not filled out in detail here"

This is only a very simple sketch showing some aspects of the language. I do not have space here to explain it all. See references such as [Planguage96] for hundreds of pages of detail. You will notice that it resembles a programming language. Many of its 'tricks' are borrowed from those languages, but this is not a logic specification language. It is a requirements and design language.

Design Evaluation.

Planguage contains a third major feature. A language and process, 'Impact Estimation', for evaluating how good any design idea is with respect to its requirements. This is a method unlike that found anywhere else in the literature. It is based on an attempt to calculate in real terms how any design idea will impact any interesting set of qualities and costs. Methods of similar intent, like Quality Function Deployment (QFD), which were widely taught at for example IBM and HP, never use well-defined requirements or designs, and evaluate impact on primitive ' *, **, ***' subjective rating scales. Estimating the real impact, using Impact Estimation, on 20 quantified quality requirements is normal industrial practice in, for example, the telecommunications projects we have used the method on.

Impact Estimation evaluates relationships like

'Design X IMPACTS Goal Y: xx SCALE. “ xx being a result on the defined goal scale of measure."

XY: 6 Hours “ example of the result of Design Idea X, on Goal Y, being estimated at 6 hours.

More importantly we evaluate multiple designs, objects or total architecture impacts on multiple or all goals and costs.

“designs" {D1, D2, D3} IMPACT “ a set of defined designs impact"

“goals" {Quality A, Quality B, Cost C, Cost D}: “ a set of goals for quality and costs

{< 3x4 table answer>}. “giving 12 impact pairs"

“<angle brackets are used to indicate fuzzy statements needing more work on them>".

This Impact Estimation process, forces us to look at side effects, of apparently good design ideas.

In addition, for each relationship {Design Goal} we collect a set of information helping us to evaluate and understand the estimate. This is the evidence for the estimate, the source of the estimate, the uncertainty of the estimate and the credibility rating of the estimate.

These relationships are precisely what we manage in Evolutionary delivery, and the Impact Estimation table is how we do it. For most professionals who I encounter, this is a whole new world of thinking. But they recognize that it is a useful way to get early insights into what they are trying to do and the risks associated with it. It is demanding intellectually, but a lot cheaper than wasted project effort building things that will fail. That is why engineers and other professionals use so much intellectual energy. History has taught them that the alternative, high risk, is unacceptable.

Quality Control

I am a strong believer in rigorous quality control of requirements and design specification. The Planguage format is a strong basis for systematic quality control. Think about quantified goals and Impact Estimation tables compared to the unorganized undefined stream of nice-sounding words you are usually confronted with. I use the methods described in our book “Software Inspection" [GILB93].

Specification of Evolutionary Steps.

Here is a sample of one way Planguage can specify Evo steps precisely, based on earlier design and function specification.

An Evo step can be defined in a variety of ways: for example, as a set of previously defined functions (FX) and/or design ideas (DIA, DIB).

STEP1[BASIS Current Product]: {FX[COUNTRY=USA], DIA, DIB}.

An evolutionary step is a package of functions and design ideas, which when implemented for some real, or trial, system-user give measurable testable results, hopefully positive. The 'results' should be some degree of fulfillment of the current formally stated outstanding requirements.

Here is one way of stating a complete Evo plan, or parts of it.

EVO-PLAN: {STEP1[BEFORE REST], REST:{SM, SV, SX [AFTER SM ENDS:], SZ}.

This means, there is a defined set of steps {STEP1, SM, SV, SX, SZ} which make up the Evo plan. STEP1 is defined in the plan as one to do 'before' 'REST'. The others have not had their sequence determined. But SX must be done 'after' SM 'ends'. See these three parameters in index or glossary.

The 'step trial customer' or 'delivery area' can be specified by a qualifier.

STEP22 [STATE={CA, NV, WA}] REST. “see STEP22 on table below"

STEP1.1 [COUNTRY={USA, CAN, MEX}] {STEP1, SM, FX [STATE={WA, GA, FL}].

Impact Estimation for control of evolution.

The impact estimation table forms the key to project planning and control.

The Glossary.

A 300+ term 'concept centred' glossary supports exact definition of key Planguage concepts. All concepts have an identity number like '*999'.Here is a sample, cut directly from the book. Bold indicates defined terms, italics is the comment, plain is the definition.

EVENT: *062.sign. condition class. A discernible historic event, like "president inaugurated"

evidenceevidence: *063.evidence*063.evidence. written historic facts and their sources, which were our basis for making an estimate. If necessary, an admission of lack of evidence.

evolutionaryevolutionary: *196.evolutionary*196.evolutionary.

A gradual holistic process of conscious improvement . Abbreviation EvoEvoabbreviation for evolutionary.

By 'gradual' we are thinking in terms of steps of 2% of the long term horizon goals. Contrary to popular misconception, evolutionary is the fastest and safest way to deliver project results. Instead of 100% of needs being years late, we usually experience 80% of the needs delivered in 20% of the total project time, by selecting the largest benefit to cost ratio steps for early delivery. It is inherently safer than 'waterfall'waterfall or 'big bang'big bang methods because early delivery steps give us feedback, from which we can correct any errors in our plans. *196.Evo.abbreviation*196.Evo.abbreviation.

Exit Exit . *290.Exit *290.

is a DQC exit process where the Team Leader checks a set of defined DQC exit conditions, especially maximum estimated remaining Major defects per page in the process output document, and either decides to allow exit from the larger DQC mega-process (as opposed to DQC sub-processes), or to take other action.

exit conditionexit condition: *064.exit condition*064.exit condition. a written standardstandard for release of a document to outsiders (it's "customerscustomers" in the sense of user and client).

References

[COLEMAN94] Coleman D et al, Object-Oriented Development, The Fusion Method, Prentice Hall Object-Oriented Series, 1994 (This describes the HP efforts!)

COTTON96: Cotton, Todd, “Evolutionary Fusion: A Customer-Oriented Incremental Life Cycle for Fusion". Hewlett-Packard Journal, August 1996, Vol. 47, No. 4, pages 25-38.

An excellent detailed view of the Evolutionary project management process as taught Corporate-Wide at HP. The author was on a project team at HP about 1990 which Gilb taught early versions of the Planguage method. See another member of that Project in MAY96.

HP Journal subscription free to qualified individuals: Write Distribution Manager, HP Journal, M/S 20BH, 3000 Hanover Street, Palo Alto, CA, USA-94304, or Email: hp_journal@hp_paloalto-gen13.om.hp.com. HP Journal is available on World Wide Web at “http://www.hp.com/hpj/Journal.html". Author Email:Todd_Cotton@HP-PaloAlto-om4.om.hp.com.

[Gilb93] Gilb, T. and Graham, D., Software Inspection, Addison-Wesley, 1993.

Graham90] 'INCREMENTAL DEVELOPMENT AND DELIVERY FOR LARGE SOFTWARE SYSTEMS' by DOROTHY R. GRAHAM,

It has been published in two places, a journal and a conference proceedings:
Information and Software Technology, Vol 31, No. 1, Jan/Feb 1989, pp 7 -20, published by Butterworths.

Sofware Engineering for Large Software Systems, ed. B. A. Kitchenham,

Porceedings of the Sixth Annual CSR (Centre for Software Reliability)

Conference on Large Software Systems, published in book form by

Elsevier, 1990.

Electronic version 36 pages.

Author: Grove Consultants, Grove House, 40 Ryles Park Road, Macclesfield, Cheshire SK11 8AH, UK Tel +44 1625 616279 Fax +44 1625 619979, email Dot@ Grove.Co.UK.

[HP] Todd Cotton,

“Fusion Newsletter" Vol. 4.2 April 1996, Software Technology Lab, HP Labs, 1501 Page Mill Road, Palo Alto CA, USA-94304. Email: TODD_COTTON@HP-PaloAlto-om4.om.hp.com , HP Corporate Engineering, Palo Alto. See Also COTTON96, MAY96, MALAN95

[PLANGUAGE96] Gilb, T. “Planguage" (working title) Manuscript in progress (used on authors courses, available on web site listed below), no concrete plans for publisher and publication at the moment. (August 1996).

[POSEM88] Gilb, T. Principles of Software Engineering Management, Addison-Wesley, 1988

[MALAN95] Malan, Ruth et al, Object-oriented Development at Work, Fusion in the Real World,

Hewlett-Packard Professional Books, Prentice Hall, 1995. This contains a lot more on life cycles and evolution than the other Fusion Book, see esp. Todd Cotton's papers and page 180-186 “Evolutionary Development"., and “Evolutionary Fusion" (6.2) pp 186-201, and other articles by other authors. Todd Cotton's material are directly based on Gilb's writings and personal teaching to HP and Mr. Cotton.

[MALAN95] Malan, Ruth et al, Object-oriented Development at Work, Fusion in the Real World,

Hewlett-Packard Professional Books, Prentice Hall, 1995. This contains a lot more on life cycles and evolution than the other Fusion Book, see esp. Todd Cotton's papers and page 180-186 “Evolutionary Development"., and “Evolutionary Fusion" (6.2) pp 186-201, and other articles by other authors. Todd Cotton's material are directly based on Gilb's writings and personal teaching to HP and Mr. Cotton.

MAY96MAY96: Elaine L. May and Barbara A. Zimmer, “The Evolutionary Development Model for Software",

Hewlett-Packard Journal, August 1996, Vol. 47, No. 4, pages 39-45.

This is an excellent complimentary article to COTTON96COTTON96.

The author was on a project team at HP in 1989 which Gilb taught early versions of the Planguage method. The article is full of practical advice and case studies gleaned from ten major projects in eight HP divisions. It must be strongly recommended to anyone interested in the practical implementation of Evo in a project and especially in an organization for many projects.

HP Journal subscription free to qualified individuals: Write Distribution Manager, HP Journal, M/S 20BH, 3000 Hanover Street, Palo Alto, CA, USA-94304, or Email: hp_journal@hp_paloalto-gen13.om.hp.com. HP Journal is available on World Wide Web at “http://www.hp.com/hpj/Journal.html".

Author Email: Elaine_May@CE.HP.com?? Being found

???[Kruchten96] Dr. Philippe Kruchten, "A Rational Development Process"

Crosstalk, US Dept. of Defence, July 1996 pages 11-16

author Philippe Kruchten Rational,

240-10711 Cambie Rd

Richmond, BC V6X3G5

Canada

Tel. 1 604 231 3706,

Author email: pkruchten@Rational.com.

Sept 10th 1996 he emailed us a paper clarifying some points of the article.

Crosstalk: custserv@software.hill.af.mil, or 801 777 8045

Author Biography

Tom Gilb was born in California 1940, Joined IBM 1958 Norway, became a freelance consultant 1960. He resides in Norway with his Norwegian Family, and has a lot of American Family in London, as well as a flat in Covent Garden, so as to be steps away from the Ballet and Theaterland. He spends most of his working season teaching and consulting for industry worldwide, mainly for high tech industries, mainly for software people, but there is quite a variety which includes top management and aircraft design, as well as some charities like Environmental and Third World Aid planning. Off season, mainly all Summer he enjoys sitting at his Cabin on the Oslo Fjord for 9 weeks, trying to write books, so as to justify watching the sailboats passing by. AKA ships passing in the night.

He was a consultant to the Simula 67 Project (Dahl, Nygaard, Myrhaug) which is the acknowledged inventor of the OO concept.

Author Email: Gilb@ACM.org,

Office: for Information on Gilb publications, and public and in-house courses on Planguage, Evolutionary Delivery, Inspection, Requirements Specification.

Web Site for more information

http://www.stsc.hill.af.mil/www/stc/gilb.html is a Hill AirForce Base Web Site which has many Gilb things such as the Results book Manuscript ( now being rewritten this Summer as "Planguage", the main effort is 100 illustrations) and slides for talks (Advanced Inspection and What is Level Six) and courses.(Inspection, Requirements)

Sidebar quote from the 'Planguage' manuscript.

From Chapter.section 9.6. Advanced ideas: Evolutionary Management

How to decompose systems into small evolutionary steps.

Moved to this page