INCREMENTAL
DEVELOPMENT AND DELIVERY
FOR LARGE SOFTWARE SYSTEMS
DOROTHY R. GRAHAM
Software Engineering Consultant
Grove Consultants,
Grove House, 40 Ryles Park Road,
Macclesfield, Cheshire SK11 8AH, UK
Tel +44 1625 616279
Fax +44 1625 619979
ABSTRACT
Life cycle models have arisen in order to bring control to
the process of software development. Two aspects of life cycles
have provided benefits: the ordering of development phases, and
the modularisation of the development process to give discipline
and control. The third aspect, production of systems monolithically,
is not beneficial for large systems.
Incremental development is the development of a system in a
series of partial products, generally with increasing functionality,
throughout the project timescale; incremental delivery gives those
increments to the users when they are completed. An increment
is complete when all the associated life cycle products are finished,
including testing, training, and documentation. There are significant
benefits both for developers and users, but there are also significant
problems. In particular, more discipline is needed to manage incrementally,
particularly good configuration management.
An earlier paper [1] gave an extensive bibliography of incremental
development and delivery. This paper summarises the types of incremental
development with particular reference to large system development,
and gives additional recent references.
1. INTRODUCTION
Life Cycle models such as the Waterfall or V-models have been developed in order to bring control to the process of software production. There are three key aspects embodied in such models:
It is my contention in this paper that the first two aspects
of monolithic life cycle models have provided the benefits which
have been achieved, but the third aspect does not scale up for
large system development and may actually be detrimental.
2. SOFTWARE DEVELOPMENT PROBLEMS
Although significant improvements in the development of software
systems have been achieved in recent years, there are still significant
problems remaining. Monolithic software development is generally
fixed at the earliest possible time, both in terms of cost and
schedule. In order to meet tight contractual restrictions, the
Requirements Specification is produced either before or soon after
the start of development, and is then "frozen"; i.e.
changes to the requirements are strongly discouraged.
2.1 Estimation
It is difficult to estimate the cost, effort, time or size
of large systems with a high degree of accuracy. This is not unique
to software development projects, it also affects other engineering
disciplines. Being able to foresee the future with unerring accuracy
is not a task which is easily done, by people or by machines.
The real world does not stand still while large systems are developed;
new products and processes are discovered, underlying assumptions
are invalidated, new laws are passed, and developers learn new
things, which would have enabled them to build a better system,
if only they had known those things at the beginning. It is a
common experience at the end of development to know how it should
have been done.
2.2 Changing Requirements
Knowing how it should have been done is also a common reaction
of users when they first see the new system they have asked for.
A widespread response when using the new system for the first
time is: "That is not what I want". It is much easier
to "know it when you see it" than to say what it is
that you want before you see it.
This effect occurs even when users have been extensively involved
in specifying the system (but then they should add, "even
though it is what I asked for"). User knowledge also grows
throughout system development; the users cannot ask for what they
will really need because they cannot see into the future with
unerring accuracy either.
The point is made by Floyd, Reisen and Schmidt that "requirements
are not 'given' and therefore cannot, strictly speaking, be analyzed."
Rather, they are gradually established through interaction between
users and developers. [2]
These two aspects of developing knowledge are illustrated in
Figure 1 below. Both user knowledge and developer knowledge increase
with time. System A, which is specified at the beginning of development
reflects the minimum of that growing body of knowledge for both
parties. System B is the system which the developer would have
built with hindsight, and System C is the system which the customer
would have asked for with hindsight. System D is the system which
is actually required at the end of development.
Figure 1. Increasing knowledge throughout development.
2.3 Testing and Maintenance
Significant problems are experienced in the testing and maintenance
of large systems. Testing typically takes 40% of the software
development effort for any sized system, with testing of real-time
systems often taking up to 80% of the development time. Testing
of large software systems is difficult, but with the monolithic
life cycle, the major dynamic execution testing of the software
is done at the end of development. Verification and validation
procedures help to produce better quality intermediate life cycle
products, but they are no substitute for testing the real end
product.
If development takes longer than expected and schedules slip,
then it is likely that testing will also take longer than expected.
However, the end date has a tendency to remain fixed in time,
so the testing schedule is compressed rather than extended. Problems
which are built into the software are only discovered late in
development, when there is no time to put them right. If testing
is difficult, why do we leave it till last?
With some types of development, testing is constrained by the
development of hardware, for example, so that testing cannot be
done early. However, other types of system can begin testing much
earlier in the development timescale than the monolithic life
cycle implies, with considerable benefits.
Maintenance can typically account for 80% of the total system
costs. A major aspect of the early maintenance of a finished system
is to create the system which should have been asked for and built,
if only users and developers had the benefit of hindsight. In
the diagram above, System A is delivered, but maintenance then
progresses the system to System D (and beyond). A frozen specification
can only hope to meet a frozen need.
2.4 Customer Confidence (lack of)
Figure 2 below illustrates the confidence levels which occur with both monolithic and incremental development models.
Confidence is initially high with both approaches, probably higher for the monolithic approach at the start because it is the accepted way to do things.
During lengthy development, however, customer confidence drops off, as no working product is yet visible, developers talk in techno-speak, and delays are announced. If delivery of the system comes before all confidence is lost, there is hope of recovery, but if confidence goes below a certain point, then no matter how good the system is technically, it will not be accepted by users.
This user resistance may be dormant during monolithic analysis and implementation, emerging when the system becomes operational, and taking the form of sabotage, scapegoating, or incorrect use of the system. [3]
An incremental approach produces a working product much earlier than a monolithic approach. The initial reaction to seeing this is usually "This is not what I want", leading to an initial loss of confidence, although the delivery of an increment on time is generally a refreshing change for customers.
By developing incrementally towards the needed system (rather
than the specified system), confidence is restored to a high level
which is then maintained and increased.
Figure 2. Customer Confidence throughout development.
3. THE THREE-DIMENSIONAL WATERFALL MODEL
The Waterfall model is taken as illustrative and representative
of monolithic life cycle models. It is usually shown in a form
similar to Figure 3 below. The Requirements Specification (RS)
should be done first, followed by System Design (SD), Detailed
Design (DD), Code and Test (C&T), and Integration and Test
(I&T).
Figure 3. The Waterfall Model
Monolithic life cycle models recommend that each phase be completed before starting on the next phase. In practice, this never occurs; there is always some reworking of baselined products from previous phases.
In monolithic development, the working system in its entirety is delivered at the end of the whole development process.
The waterfall model does not show what is actually being recommended,
however. It is really a two-dimensional representation of a three-dimensional
object. A three-dimensional model, Figure 4, shows more accurately
the time sequence of development. Each horizontal "slab"
should be completed before starting on the next one.
Figure 4. The Three-dimensional Waterfall Model.
4. INCREMENTAL DEFINITIONS
An incremental approach postpones detail in some or all phases
to produce working software earlier in the project development
timescale. The basic idea is to develop the system in a vertical
slice rather than a horizontal slab.
4.1 Incremental Development
Incremental Development is the development of a system in a
series of partial products (increments) throughout the project
timescale.
4.2 Incremental Delivery
Incremental Delivery is the delivery of increments to the customer/users at intervals throughout the project timescale.
Note: A system can be developed incrementally without being
delivered incrementally to users, but not vice versa.
4.3 Increment
An Increment is a self-contained functional unit of software, together with all supporting material, including:
- requirements specification,
- design documentation,
- test plans, cases and results,
- user manuals and training,
- estimates, plans, schedules, resourcing,
- quality assurance information (e.g. review reports)
- configuration management information.
An increment produces (or alters) a cross-section of the final system deliverables, related to a functional portion of the final system.
Incremental development is the construction of a software system in a series of small mini-life-cycles, rather than construction in one large monolithic life cycle.
In the following sections, a number of incremental life cycle
models are shown.
5. INCREMENTAL LIFE CYCLE MODELS
5.1 Incremental Build and Test
Figure 5. Incremental Build and Test.
The incremental build and test approach begins the incremental
development in the coding phase, with the previous phases being
monolithic. Examples are given in Deutsch [4] and Wong [5] among
others.
Many developers go some way toward this approach informally,
although often without the complete set of life cycle documentation.
Since this is not what is recommended by the monolithic waterfall
model, the developers may feel guilty that they are not following
the model correctly; better results will be obtained from following
the incremental build and test approach intentionally rather than
accidentally.
5.2 Evolutionary Delivery
Figure 6. Evolutionary Delivery
Evolutionary delivery as described by Gilb [6] is shown in Figure 6, and is the most extreme incremental approach, defining the increments from the top of the life cycle. Gilb's method includes incremental delivery as well as incremental development, and therefore has useful working facilities available to the customers much earlier than other life cycle models.
The diagram shown actually does not do justice to the evolutionary delivery method, however, since there is a higher level process which precedes the incremental steps, consisting of setting system and business objectives, open architecture design, planning and quality assurance.
The evolutionary deliveries are made at frequent intervals (possibly as small as a week), and consist of some function, facility, or organisational change which is useful to the customer and relatively easy to produce. In fact that ratio is used to determine the order of the increments.
A major effect of evolutionary delivery is to elicit requests
for change, mainly from users ("that isn't what we want").
However, these change requests may be "folded back"
into the development process at significantly less cost than for
monolithic models, for two reasons. First, change is expected
and planned for, so it does not come as an unwelcome surprise.
Secondly, when requirements have been completely detailed and
designed (in the monolithic approach), changes which are requested
will affect the work already invested in the frozen specification.
With incremental development, requested changes which affect those
areas which have not yet been completely detailed, do not result
in discarding work already done. Change can be turned to advantage,
provided it is controlled.
5.3 Framework Incremental Life Cycle
Figure 7. Framework Incremental Life Cycle
Just as one extreme being wrong does not imply that the other
extreme is right, a framework incremental approach may be the
"best of both worlds", by providing a compromise between
the monolithic waterfall and Gilb's evolutionary delivery. Enough
of the initial requirements specification and architectural design
is done so that the direction and structure of the system produced
is clear enough to direct the software development process. This
approach can still give useful products very early in the development
timescale. An example of this type of approach is the specification
of the structure and interfaces for a database, with detailed
facilities to be specified later.
A similar method recently described by Hill [7] as a hybrid
between "conservative" and "radical" top-down
approaches has been found to work successfully. See also Redmill
[8] and Krzanik [9].
5.4 Phased Development
Figure 8. Phased Development
Phased development has frequently been used in the development
of large systems, and is a step in the right direction. The difference
between a very small phase and a large increment is not distinct.
However, phases tend to be large and growing; there is a tendency
to put as much as possible into the current phase. There is also
a temptation to compensate for timescale slippage by bringing
forward the later phases to overlap the earlier ones. This approach
can result in severe incompatibility between successive phase
products. The emphasis with incremental development is to include
as little as possible, i.e. as little as would be useful into
each increment.
5.5 Prototyping
Prototyping is usually regarded as the building of a small
system (or two) before building the big one. The knowledge gained
in either building or using the prototype is then used in building
the "real" system. A good description of the prototyping
approach, combined with risk analysis, is given by Boehm [10].
"Rapid prototyping" is used to describe the building
of systems using software tools such as code generators, report
generators, or Fourth Generation Languages (4GL's). A recent book
called "Structured Rapid Prototyping" by Connell and
Shafer [11] gives guidance in the development of systems incrementally.
Prototypes are used to reduce risk in the applications area
(a novel design, function or performance), or to reduce risk in
the user interface area. (The users should then recognise the
final system as what they had approved as a prototype.)
However, in a long development timescale, with several years
between deliveries of prototypes, the requirements may change
extensively, so that it is difficult to relate new requirements
to the old prototypes. New hardware capacity, newer user interface
styles and graphical capabilities seen on personal computers can
make a prototyped user interface seem very old-fashioned.
It is not always possible to predict in advance of development
whether a prototype will be thrown away, incorporated or built
upon. Prototypes are classified in terms of their use into experimental,
exploratory, or evolutionary prototypes. (Floyd [12]) Prototypes
can also be classified in terms of their relationship to the life
cycle as Throw-Away, Incorporated, or Incremental,. (Ince and
Hekmatpour [13]).
5.5.1 Throw-Away Prototype
Figure 9. Throw-Away Prototype
If the prototype is written with the intention of throwing
it away, then it is unlikely (and unnecessary) for it to be developed
to the same standards as a "real" development. This
type of prototyping is sometimes called "quick and dirty".
The prototype can be discarded as soon as development of the large
system begins (the first bin in the diagram), or it can be kept
and possibly used by the customer until the large system is actually
delivered and finished (the second bin). There is a tendency for
users to become fond of a well-used prototype and be reluctant
to throw it away.
A throw-away prototype can be used as an animated requirements
specification; if the prototype looks and interacts in a way which
is liked and approved by the customer, then the final system can
be tested to ensure that it looks and behaves in the same way.
5.5.2 Incorporated Prototype
Figure 10. Incorporated Prototype
Part or all of the prototype can be incorporated into the final
system software product, either code, design, or requirements.
If a prototype has any possibility of becoming incorporated, the
software engineering discipline and standards applied to its development
should be consistent with the desired quality of the final product.
5.5.3 Incremental Prototype
An incremental prototype is the development of the system in
a series of prototypes which are integrated together to become
the final system. This therefore looks the same as the Evolutionary
approach shown in Figure 6, but possibly without incremental delivery
or the higher level of overall objectives, open architecture and
planning of Gilb's approach.
5.6 Tool-intensive Incremental Development
Figure 11. Tool-Intensive Incremental Development
Software development tools are becoming capable of automating "higher up" the life cycle, i.e. code can now be generated directly from detailed design specifications. At the same time, requirements are being specified in increasingly more formal notations.
These two trends give the possibility of eventually automating the entire life cycle. Semi-formal specification methods such as SSADM, Jackson, Yourdon and others may eventually be automated. Mathematically based formal methods such as VDM, Z, and OBJ are rigourous enough to automate now. There is already a commercial tool (ObjEx) [14] which automates specifications written in OBJ.
Although current approaches suffer from severe performance
limitations, future hardware and software development will continue
to ameliorate performance problems. Ultimately we are left with
only two phases: specifying requirements and testing. If this
happens, it would not be sensible to specify all requirements
before getting something working, and software development will
naturally follow an incremental cycle.
6. INCREMENTAL DEVELOPMENT RESEARCH
There is a large body of research in the area of incremental development as described in Graham [1] for a comprehensive overview, Gilb [6] for historical development and general inspiration, Ince and Hekmatpour [13] on prototyping, Agresti [15] for discussion of new paradigms, and Law and Longworth [16] for strategic considerations.
Guidance for implementing incremental development and delivery can be found in Deutsch [4] for incremental build and test, Gilb [6] for evolutionary delivery, Connell & Shafer [11] for tool-intensive (4GL) development. An interactive incremental development methodology called "STEPS" (Software Technology for Evolutionary Participative System Development) is given in Floyd, Reisen and Schmidt [2]. Michael Jackson has recently launched an incremental development methodology called "ISE" (Incremental Software Engineering) [17].
Reported experience in the use of incremental development is scarce. The design of a prototype for a large system is described in Harker [18]. Other references have been given in Graham [1].
It is interesting to note three instances of successful use in other papers included in this publication: Warboys [19] on the success of CADES, Chatters [20] on the fault-free factory approach, and Malcolm [21] on the first on-time delivery in ten years, albeit in a project which was probably already doomed (due to the loss of user confidence). Willmott [22] also felt that incremental re-development of an air traffic control system would be safer than one-time replacement.
The successful IBM "Cleanroom" approach [23], described
at an earlier CSR conference, is also an incremental approach.
7. INCREMENTAL STRATEGY
7.1 Suitability
Large systems are particularly suitable for incremental development. Monolithic development is suitable only for small systems of short duration, where the requirements are well known at the beginning of development and unlikely to change, according to Krzanik [9], and for commercial packages such as a database package, operating system, or word processor, according to Connell and Shafer [11]. Gilb maintains that any system can be developed using evolutionary delivery. [6]
Deciding which form of incremental development to use should
be based on the risk factors for the particular system to be developed.
If the system requirements are very uncertain and highly likely
to change, then prototyping may be suitable. If the system architecture
is critical, e.g. a database, then the framework model may be
suitable. If development funding is uncertain, then evolutionary
delivery may be most suitable. Incremental development is quite
feasible for fixed-price contracts, but may well be easier without
incremental delivery.
7.2 Partitioning the System into Increments
Deciding how the system can be divided up into self-contained functional increments, particularly for incremental delivery to users, can be difficult when incremental techniques are initially tried. (Deciding how best to partition a system for development is never easy anyway.) There are a number of good design methodologies for producing a loosely coupled architecture of non-overlapping functions (which is good design practice in any case); incremental development does not require any particular or specific design technique. The involvement of outside consultants with knowledge and experience of incremental development can be very helpful in the initial stages.
The initial difficulties experienced in designing increments are often due to thinking about the system in ways which are quite different to the monolithic approach. Monolithic thinking is directed toward completing all requirements first, but incremental thinking takes a small requirement subset toward implementation first.
The objectives which drive the partitioning process are to
keep the increments as small as possible, provided they will provide
a useful function to the users. The temptation to continually
increase the functionality within each increment should be resisted.
It is also essential to retain control over the content of each
increment, and prevent developers from incorporating other "good
changes at the same time", which then becomes undisciplined
and uncontrolled.
7.3 Prioritising and Scheduling
The scheduling and sequencing of development is based on three aspects: first, any parts of the system which must be in place before functional increments can be implemented should be completed first, but only the minimum needed.
Second, the broad strategy for the next series of increments should be defined. Alternatives include the development of the most critical increments first to minimise risk, the development of interface increments first to test control, or the development of functional threads first to achieve a working partial product. The latter is needed as early as possible in order to use incremental delivery effectively.
Within the broad strategy, there will be a choice of increments among equal priority. The third scheduling aspect is based on a ratio of two things: the user benefits and the development cost. The user benefits for the selection of proposed increments should be analysed and prioritised by the user organisation, if that is feasible. The development costings should be estimated by the developer organisation.
This ratio of benefit over cost determines the scheduling priority. This is technically called "the juiciest bit" (Gilb, [6]) Thus an increment with a high perceived benefit with high cost may be developed before one for a low cost and low perceived benefit, but those with both high benefit and low cost will be earliest.
The increments to be prioritised will not stay fixed for long,
but need to be re-analysed in the context of recently delivered
increments; error correction increments may take precedence over
other planned increments, for example.
8. PROBLEMS OF INCREMENTAL DEVELOPMENT AND DELIVERY
In this section, various problems related to incremental development and delivery are outlined. Possible solutions are suggested where appropriate. Further discussion, particularly of management problems, can be found in Redmill [8].
There are many aspects of software development which are not
affected by incremental development or delivery, such as the need
for good management, quality assurance, configuration management,
and the training of staff in software engineering principles.
Additional training is needed, however, when departing from development
techniques which are widely accepted, even when extensive benefits
can be gained.
8.1 Hardware Related Problems
8.1.1 Risk of inadequate choice
The choice of hardware for a system to be developed incrementally
will be based on intentionally incomplete specifications, with
the risk that an inadequate choice will be made. In fact monolithically
developed systems often make the wrong choice of hardware as well,
but the risk is increased in incremental development. A hardware
upgrade may be a possible solution.
8.1.2 Response times
If a small number of increments have been delivered on the
target hardware, response times should be extremely good at first,
but will gradually deteriorate with increasing functionality,
to the disappointment and frustration of users. To overcome this,
it is possible to "simulate the final user environment",
i.e. put in slowing-down code to be removed during system tuning.
If response times are severely affected by a single increment,
this can be an aid to identifying system bottlenecks, which can
be difficult to localise with a slow system which has been developed
monolithically.
8.1.3 Development hardware
If the target hardware is delivered with the early increments,
additional hardware may be needed to continue the development
of the system. This problem is normally postponed until the maintenance
phase of monolithic development, but needs to be faced earlier
in incremental delivery.
8.1.4 Embedded systems hardware
The parallel development of hardware for embedded systems may
constrain the definition of the increments.
8.2 Life Cycle Problems
Incremental development is not an alternative to applying life
cycle discipline; the phases of the life cycle still need to be
followed in the right order and with all of the associated controls.
Each increment is a small life cycle in its own right.
8.2.1 Requirements specification
Requirements still need to be specified for the limited area
which comprises an increment, and need to be frozen while the
increment is developed (icecubes instead of icebergs). This enables
planning, estimation and scheduling to be done in the small. The
Requirements Specification is needed to define the boundaries
of the system and of the increment.
8.2.2 Design
Design is needed in order to preserve a coherent structure
to the software system throughout the changes which will occur
during incremental development. The overall design should be defined
in the first increment, but each increment should also be designed,
and the design must work towards preserving the integrity of the
overall architecture. This requires effort, as Lehman and Belady
point out in their second law of program evolution [24]. The difficulty
of providing a good overall design without a full definition of
requirements should not be underestimated.
8.2.3 Testing
Testing is needed to ensure that each increment fulfils its requirement, and has not adversely affected the rest of the system. With incremental delivery, more extensive regression testing will probably be needed, since any change to the system results in a changed system. All tests should be run again to ensure that there are no adverse side-effects of the change.
There are additional considerations for a system which is tested extensively using a purpose-built simulator or test harness. If the production software is developed incrementally, the test software needs to be ready for use early in the development timescale, and so may also need to be developed incrementally. This will have an effect on the scheduling of effort between the software product and the test software. Note that the test software needs to be tested as well, before it can be relied upon to test the production software.
Redmill has found that there is a tendency for users to perform
thorough acceptance testing only for the first increment with
the testing of subsequent increments being skimped. [8]
8.2.4 Other life cycle products
Documentation, user manuals, training, and quality control
procedures should not be skimped in the excitement of having something
working. They are still needed in order to retain control over
the development process. A good configuration management system
is essential for keeping track of increments in various stages
of completion. Although working code may be produced quickly,
the extra documentation required for additional releases may be
seen as a greater overhead than with monolithic development.
8.3 Management Problems
8.3.1 Living with uncertainty
It is unsettling to live with uncertainty; this is one reason
why developers prefer to specify complete requirements before
beginning to design a system. However, incremental development
requires a certain level of uncertainty to be tolerated within
the context of controlled development. There are levels of uncertainty
with monolithic development as well, but we tend to hide them
from ourselves by attempting to resolve specification uncertainties
on paper.
8.3.2 Team coordination
The coordination of teams of people working on different parts
of the system, and being in different life cycle phases at once,
presents a challenge to management. Corrections found in the use
of a delivered increment have to be incorporated into the system
as part of an increment further "down-stream". Configuration
management is essential.
8.3.3 System releases
Releasing a system to a large user base incrementally is even
more of a challenge, and may prove very difficult even with a
good configuration management system.
8.3.4 Scheduling and prioritising
The scheduling and prioritising of increments is a process
which is constantly being altered by the results of earlier incremental
deliveries; management must be prepared to spend effort in supporting
this continuing process.
8.3.5 Balance between original specification and desired changes
Development may tend to proceed in two directions at once;
pulled toward the original specification by the developers (who
can easily become "locked in" to local goals), and pushed
toward new changes by the users. Management needs to keep the
balance between these two. The requests for change should not
be allowed to "hijack" the original system objectives,
but some change must be allowed or the benefit of incremental
delivery will be lost. Changes need to be controlled at a strategic
level, in order to take the widest view of the system objectives
into account.
8.3.6 Organisational cultural change
Changing the way a large organisation develops software is not easy and cannot be done overnight. Effort is needed in introducing incremental development ideas, to assess and then convince of the benefits. Effort is also needed to ensure that the concepts are being implemented correctly; for example, the temptation to merge increments together in order to meet a timescale should be resisted. Without continuing pressure, attitudes and habits will revert to the earlier ideas, even if the new words are used.
The development organisation may find strong resistance even
from those departments which stand to gain from incremental development
or delivery, for example contracts, quality assurance, and higher
management.
8.4 Financial/Contractual Problems
8.4.1 Contracts
Contracts for the development of software systems are generally based on a statement of requirements, under the assumption that development will proceed monolithically. There is no reason why incremental development or even delivery cannot proceed from a "completed", i.e. frozen, Requirements Specification. However, many of the benefits of the incremental approach will be lost if user and developer knowledge were prohibited from being incorporated into the development process.
To allow the format of the contract to determine the development
strategy appears to be putting the cart before the horse; surely
the contract should be the servant of development, not its master.
However, a new contractual approach may also involve significant
organisational culture changes.
8.4.2 Competitive tendering
Competitive tenders using an incremental approach as opposed to a fully specified requirement may not enable the purchaser to evaluate like with like. The unspoken basis for pre-specifying the full requirement is the assumption that the requirements can be fully known in advance, which is rarely completely true.
A working prototype may be more convincing to a potential customer
than a 250-page document, however.
8.4.3 Estimation
Estimation of system development effort is even more difficult
when you don't know what you are going to build. Estimation methods
and tools are the wrong way around for incremental development;
they tell how long for a known (guessed) size, whereas an incremental
estimate wants to know what functionality (size) can be achieved
with known effort.
8.4.4 Management accountability
Accountability to higher management may be distorted by incremental
development. Effort will be spent on the original specification,
corrections and change requests. If management are tracking only
effort on the original specifications, they may not have a true
picture of the system being produced. On the other hand, incremental
development may be the means of forcing out into the open what
is really happening, which can be hidden in monolithic development.
8.4.5 De-stabilised development environment
If a developer is funded for only a small number of increments rather than for a full development, this may lead to a de-stabilised development environment, i.e. insufficient commitment to the project from the developer because of the uncertain future of the project. This may produce an inadequate product, a new search for a more suitable developer, with delays and additional expense. However, it is preferable for the purchaser to discover the limitations of the supplier after only a few increments than after a lengthy monolithic development.
If a contract is awarded before a full specification is produced,
the buyer may become "locked in" to an unsuitable supplier.
It may well be cost-effective to fund more than one developer
to produce an early increment, particularly for a large system.
8.5 User-Developer Relationship Problems
8.5.1 User expectations
Users need to be involved in the process of setting goals and
requirements for each increment, and need a thorough understanding
of what is to be achieved by each one; their expectations can
become somewhat inflated if they are not kept in touch with realistic
proposals. User perceptions of what is easy and difficult to achieve
are notoriously inaccurate.
9. ADVANTAGES OF INCREMENTAL DEVELOPMENT
The advantages of incremental development are given separately
to those of incremental delivery. Incremental development without
delivery gives advantages to the developers, and incremental delivery
gives advantages to the users.
9.1 Improved Team Morale
Success breeds success; if the team can see the end product
actually working, it is a great boost to morale and can lead to
greater productivity.
9.2 Early Solution of Implementation Problems
Problems which are discovered during implementation can be
put right before the rest of the system is built according to
the same (faulty) assumptions. Testing the software early in development
leads to improved quality of the finished product.
9.3 Reduced Risk of Disaster
If you are going to have a disaster, it is much cheaper to
have an incremental one; you will have lost only thousands or
tens of thousands rather than millions.
9.4 Improved Maintenance
The maintenance of incrementally developed systems is easier
than monolithically developed systems because a maintenance environment
is started early on in development. Good maintenance is continuous
controlled change, and so is incremental development. Increments
which are not designed to be easy to change will not survive the
incremental development process.
9.5 Control of Over-engineering or Gold-plating
If best-value increments are developed first, that leaves the
worst-value increments until last, where their true cost can be
seen. Additions to specifications are often agreed for political
reasons at the beginning of a project; if the benefits to end
users drive the production schedule, the political enhancements
may well be quietly forgotten.
9.6 Measurement of Productivity
In monolithic development, productivity is measured in terms
of lines of code produced per day, or pages of documentation.
If parts of the system are working, "product"-ivity
can be measured in terms of the actual product.
9.7 Estimation Feedback
Estimates are produced in the large, for the whole system, but also in the small for the increments. If the first increment's actual effort is out by a factor of two, for example, it is possible that the global estimates are also out by the same factor. In monolithic development for a project taking say 6 years, you may not find out how inaccurate your estimates are until 4 years have elapsed; with incremental development you will find out much earlier, say after 2 months.
The feedback from incremental estimation can either modify
or confirm both the global estimates and the estimates for subsequent
increments. Either way management has better information sooner
than with monolithic development.
9.8 Smoother Staffing Requirements
With monolithic development, the project will need teams of
analysts, followed by teams of designers, coders, and testers
in sequence. With incremental development the need for specialised
teams is distributed throughout the development process.
10. ADVANTAGES OF INCREMENTAL DELIVERY
The benefits of incremental development for developers are
significant, but the greatest benefits come from both incremental
development and delivery to users.
10.1 Useful Product Early
Seeing something working which will actually benefit their
needs is the greatest reward for users. Users may well be worried
about their decision to invest in a software system and may remember
hearing tales of disaster in related areas. Having something which
can bring business or operational benefits early in the development
process gives them the beginning of a return on their investment
without having to wait years for anything tangible.
10.2 Increased Confidence in Developer
The users' confidence in their developer is greatly enhanced
by having something working early; actually this applies whether
the increment is delivered or not. An improved working relationship
results from better morale.
10.3 Better Quality Software
The software which is produced benefits from the developers'
increasing knowledge being "folded" back into the developing
product.
10.4 Longer Useful Life
The system will have a longer useful life, because it will
be easy to maintain, having been subject to a great deal of controlled
change throughout development. Products which can evolve easily
to satisfy changing business needs last longer and are more cost-effective
than those which are so difficult to change that they are abandoned
and replaced.
10.5 More Flexible Options
If a project does suffer from cost or time overruns, some later
increments can be eliminated, and the most useful parts of the
system will still be produced within the original financial constraints.
10.6 Something Useful if Cancelled
If a monolithic project is cancelled, the only things produced
are piles of paper (requirements, design documents, etc.). If
an incremental project is cancelled, the increments already produced
form a working product of some sort.
10.7 Increased User Acceptance
If users actually have some say in the way the system is developed,
their sense of "ownership" is improved, which leads
to better acceptance of the system by the user organisation.
10.8 Increased System Assimilation
The assimilation of a major new system into working practices
can be traumatic for the user organisation. People are resistant
to change, but they resist large changes more than small ones.
Most problems of system use stem from people problems rather than
technical ones. Incremental delivery allows small areas of the
organisational procedures to be altered at any one time; when
the problems are overcome, those areas are then established, so
the system has a foothold in the organisation. This also increases
the sense of system "ownership".
10.9 Increased Understanding of Requirements
Incremental delivery gives the users something real. Their
first reaction is "this is not what I want", but now
they have a much better idea of what they do want, and can progress
toward their true requirements.
10.10 System can Meet Real Need, not Frozen Need
The increasing user knowledge gained during development can
be "folded" back into the development process, so that
the eventual system is much closer to what is really needed at
the end, rather than what was thought to be wanted at the beginning.
11. SUMMARY OF EFFECTS
A summary of the effects of incremental development and delivery
is shown in Figure 11 below.
Figure 11. Effect of incremental development and delivery.
Incremental development moves the final product from the initial specification, System A, towards System B, what the developer would have produced with hindsight.
Incremental delivery cannot occur without incremental development, so it is not possible to move from System A to C.
Incremental development and delivery moves the final product from System A towards System D, the system which is wanted at the end of development.
Monolithic development confines the system to System A, the
frozen specification. Neither user nor developer knowledge can
be incorporated into the software system product.
12. COMPARATIVE COSTS
The comparative costs of monolithic and incremental development
for a hypothetical project are shown in Figure 12 below.
Figure 12. Comparative Costs of Monolithic and Incremental
Development
12.1 Original Specification
With monolithic development, the cost of developing the original
specification must be met in full in order for the project to
produce anything useful. With incremental development, it is possible
that the original specification would still be produced, but is
more likely that only part of the original specification will
be produced.
12.2 Corrections
The cost of corrections is no more, and should be less, for
incremental development than for monolithic, since errors of analysis,
design or implementation will be discovered earlier in development
and will not have propagated through the rest of the system requirements,
designs and code.
12.3 Inevitable Change
It will not be any more costly to respond to inevitable change,
such as a changed hardware environment or new work practices.
It should in fact be less costly for incremental development,
since a system which has been developed incrementally is accustomed
to frequent change.
12.4 Requirements Evolution
It will not be any more costly to implement evolving requirements,
and in fact should be less in incremental development, since many
changed requirements can be incorporated into the final system
without having to discard work already done on superseded requirements.
12.5 Configuration Management
Configuration management costs are likely to be higher with
incremental development than with monolithic development, particularly
if the existing configuration mangement method and/or tools are
fairly primitive. Configuration management is more essential for
incremental development and delivery, although it is a good idea
for any software development.
13. COMPARATIVE SCHEDULING
The comparison of scheduling differences between monolithic
and incremental development is shown in Figure 13 below.
Figure 13. Comparative Scheduling for Monolithic and Incremental
Development
13.1 Timescale to Finish
The elapsed time to finish the total development may be longer
for incremental development than for monolithic development, but
it is not as critical.
13.2 Timescale to Produce Useful Function
The timescale until something useful is produced is much shorter
with incremental development and delivery; this is the main reason
for using incremental models. This is also why it is not as critical
if the elapsed time to finish the total system is greater than
with monolithic development.
13.3 Requirements Evolution
Real requirements do evolve, whether the developers take note of the changes or not. In monolithic development, only a limited number of requirement changes are permitted; the remaining changes are therefore stacked up waiting until after handover to be implemented. In incremental development, the evolving requirements can be incorporated into the evolving product.
After handover, the incrementally developed system merely continues to adjust to changing requirements in much the same way as it did during development, but probably at a reduced rate (fewer new facilities being included).
The monolithic system has two problems after handover. First,
it must become a maintenance-type environment capable of handling
changes in a controlled way, which may involve some "teething"
troubles. Second, it must deal with the backlog of changed requirements
which have been stacked up and prohibited during development,
in addition to coping with the on-going requests for changes.
14. SUMMARY AND CONCLUSIONS
This paper has given definitions of incremental development and delivery, and emphasized that an increment should be a complete self-contained functional unit of software, including all life cycle documentation, test documentation, and other support such as user manuals and training.
Essentially the incremental idea for large system development is to develop a series of small systems, which will eventually become the large system. The main strategy is to postpone some detail in order to get something working as soon as possible. A vertical "slice" of the life cycle model is taken, rather than a horizontal "slab" at once.
Incremental development can reduce many of the risks of large system development, but it is not without problems. The problems of incremental development centre around the management and control of software and associated products in a different time ordering to monolithic development.
In order to "think incremental", there are three guidelines:
As software developers begin "taking their own medicine" by adopting software tools, I have predicted [1] that incrementally developed tools will be more successful than monolithically developed tools; this is already supported in the experiences of implementing IPSE's as reported by LeQuesne [25]
Barry Boehm once said that developing software from requirements is like walking on water; it's easier if it's frozen. However, it is easier to freeze a pond than an ocean.
Software is the most flexible, malleable and adaptible medium
ever known; isn't it strange that the first thing we do with a
software requirement is to freeze it? Incremental development
and delivery can help to put the "soft" back into software.
15. REFERENCES
Dorothy R. Graham: Incremental Development and Delivery for Large Software Systems,
published in Software Engineering for Large Software Systems, ed. B. A. Kitchenham,
Proceedings of the Sixth Annual
CSR Conference on Large Software Systems, Elsevier, 1990.
|
Homepage without frames please ......................................................... |
