QUALITY AND RISK MANAGEMENT

Spring 1997 Term Paper

Raymond Miller
Presented April 21, 1997

Prepared for:
Dr. Richard Bechtold
SWSE 625

Approved by:
________________

P. McNeece
MJY Program Manager

The purpose of this paper is to compare the relationship between software quality and software risk management. The paper first reviews the basic principles, techniques, and their application to the development of quality software. The reader is then provided with a review of basic risk management concepts, including Boehmís six step risk management process. The balance of the paper discusses how quality software techniques are both a contributor to, as well as a mitigator of, a software development programís risk.

Introduction to Quality

According to the International Standards Organization (ISO), quality is the totality of features and characteristics of a product or service that bear on the ability to satisfy specific or implied needs. This definition implies that quality is intrinsic to the product, and represents the production view of quality. A second school of thought holds that one must go beyond this definition to attain quality. For this school, quality is not solely product-centric, but is a relationship between the customer and the product, where the customer is any stakeholder or affected party, and the product includes goods and services. Further, the concept of quality can change over time in response to changing environments and values, where values shape the customer(s) beliefs concerning what is ìgoodî or ìbad.î Therefore, software quality, in addition to product or service features/characteristics and needs, must also be addressed in terms of the customer and the organizational context. (R.T. Vidgen, A.T. Wood-Harper) This is known as the use view of quality. A review of each of these elements is contained in the following paragraphs, the first being the human element.

Views of Quality

Each player in the development process has a different, and often conflicting, view on quality. The following thumbnail sketches are provided for quality as perceived by each of the key development process players : (Gilles)

• Program Manager: product is reliable, maintainable, keep customer happy, and do so within project deadlines and constraints (which lead to the need for compromise).
• Business analyst: The clientís ally in the development camp and wants to protect user defined functionality and requirements from external change.
• QA auditor: detect departure from the quality solution/product -- his or her actions to move the process back to the quality path will be resisted by all project related personnel.
• End user: junior employees with little input into system development, but will be responsible for its operation. The end user will demand the ìbells and whistlesî and, while he or she doesnít pay the bills for system development, will (in the long term) determine the acceptability of the system.
• Line Manger: the end userís boss that usually adopts an attitude of ìgive my people what they want without busting the budget or completion time frame.î
• Project sponsor: bill payer; wants the system delivered on time, on budget

Lastly, there is the developerís view of quality, which will influence the approach he or she selects for constructing the end product. It is derived not only from the developerís view of quality (production versus use), but also by how knowledge of the requirements (subjective or objective) is gained, and how they perceive the organizational environment in which they will work (consensus versus conflict). R.T. Vidgen and A.T. Wood-Harper identify four possible paradigms that indicate how the developer will perceive quality:

• Objective/Consensus: the developer sees quality as a rational engineering process where the objective will not be problematic once set down in an unambiguous specification. Quality is concerned with meeting the specification, implying a need for rigorous control of the development process. The developer tends to perceive quality as an attribute of the product (this is the view of much of the software engineering community today.)
• Objective/Conflict: the developer sees quality as the objective, but also understands conflicting interests are involved, so will try to determine whose quality requirement is being addressed (that of the managementís, workersí, etc.) as it will not be possible to satisfy fully everybodyís quality requirements.
• Objective/Consensus: the developer considers quality to be socially constructed, involving as many different groups (stakeholders/affected parties) as possible in order to include multiple perspectives and interests. The resultant definition of quality reflects a consensus of views involved.
• Objective/Conflict: the developer considers quality to be formed from multiple perspectives and interests, but assumes there will be conflict and exercise of power. Emancipator is concerned with constructing a notion of quality that takes into account the interests of the less powerful with the aim of human advancement through emancipation. End result of such is an accommodation rather than a consensus.

Quality Features and Attributes

Both schools recognize that quality software have two distinguishing features: first, that it conforms to its specification (i.e., is it a good solution?); and second, that it is fit for its intended purpose (does it address the problem?). In addition, both schools recognize that there is a set of attributes that constitute high quality software. A literature search of various quality texts contain many such attribute lists; the following is a seven top level attributes suggested by Glass.

• Portability: the ability of the software to be transferred easily from one computer to another for the purposes of execution.
• Reliability: the ability of the software to satisfy its requirements without error.
• Efficiency: the ability of software to perform with minimum use of computer resources such as internal memory, external memory, and machine cycles.
• Human engineering: the ability of the software to be easily understood and used by human users.
• Testability: the ability of the software to be easily verified by execution.
• Understandability: the ability of the software to be read by a software maintainer.
• Modifiability: the ability of the software to be revised by a software maintainer.

This list of attributes are presented in no particular order of merit. Like quality itself, there is no absolute hierarchy or ranking established for these attributes. Not all of these attributes will be required in every engineering project. Furthermore, the techniques used to implement these attributes may have a positive, neutral, or negative impact on each other. Therefore, a prioritized list of quality attributes must be determined early in the programís life to fix the programís quality goals and establish the ìtrade spaceî between the various attributes.

Quality Principles

There is one last item to be covered before looking at how software quality assurance enters the development process, which is quality principles. Itís becoming an accepted belief within the software development community that quality canít be inspected into software, nor can it be tested inóit must be built in with the aid of the process which produces it. In their book Software Quality: A framework for success in software development and support, Curran and Sanders indicate that this quality process must adhere to four basic principles:

• Prevent defects from being introduced. At least as much effort should be placed in keeping defects out of the code as detecting their presence in the code. Methods for doing this include the use of appropriate software engineering standards and procedures; independent quality auditing to ensure standards and procedures are followed; establish a formal method of accumulating and disseminating lessons learned from past experiences and mistakes; high quality inputs such as software tools and subcontracted software.
• Ensure that defects are detected and corrected as early as possible, as the longer the errors go undetected, the more expensive they are to correct. Therefore, quality controls must be put in place during all stages of the development life cycle, and to all key development products such as requirements, designs, documentation and code. These should all be subjected to rigorous review methods such as inspections, walkthroughs, and technical reviews.
• Eliminate the causes as well as the symptoms of the defects. This is an extension of the previous principleóremoval of the defect without eliminating the cause is not a satisfactory way to solve the problem. By removing the cause, you have in effect improved the process (and recall that continuous process improvement is another key tenet in how Total Quality Management principles are applied to quality software); and lastly
• Independently audit the work for compliance with standards and procedures. This is a two part audit conducted at the process level using SEI or SPR assessment methodologies, as well as audits at the project level which will determine if project activities were carried out in accordance to the standards and procedures established in the quality process, and whether those standards and procedures are adequate to ensure the quality of the project in general.

Quality Factors and Risk

Having discussed quality, we should now ask the question is software quality, or the quality assurance program, considered to be a risk factor in software development projects. In the book Assessment and Control of Software Risks , Jones describes the results of his experiences with formal assessment of software development. Using Software Productivity Research (SPR) or Software Engineering Institute methods to review projects in several hundred enterprises. These software produced by these projects sorted into six generic classes:

• Management Information Systems (MIS): accounting and claims systems
• Systems software projects such as operating systems, telecommunications, or other systems which control physical devices
• Commercial Off-The-Shelf (COTS) development projects where products are leased/sold to end users
• Military software projects that are constrained to follow MILSPEC
• Contract/outsourced software projects (civilian domain) where bulk of software is produced by contract personnel versus employees of the client organization
• End user software projects where the software was developed by the intended user versus professional programming staff.

Over 100 risk factors were observed within these program. Few projects have more than 15 risk factors at any time, but many have 6 simultaneously. Analysis of risk patterns from these assessments indicate that elements of significant risk are not the same across all software domains. These lists contain the top risk factors, followed by their occurrence in the sample assessed programs. Those items which are directly or indirectly influenced by the quality assurance program, or impact the quality software issue, are shown in bold.

MIS:

• Creeping user requirements (80%)
• Excessive schedule pressure (65%)
• Low Quality (60%)
• Cost overruns (55%)
• Inadequate configuration control (50%)

Low Quality is defined as delivered software which either does not work at all, or which fails repeatedly in operation. Jones defined low quality software as user reports of more than 0.5 bugs/defects per Function Point per calendar year. Low quality for MIS is endemic, and normally results from 2 things: (1) inadequate defect removal such as failing to use inspections and carrying out testing in a casual or unprofessional fashion; and (2) inadequate defect prevention such as failing to use standard techniques like Joint Application Design (JAD) or Information Engineering (IE), and sometimes failing to produce adequate specifications for the project.

Systems Software Risk:

• Long schedules (70%)
• Inadequate cost estimating (65%)
• Excessive paperwork (60%)
• Error-prone modules (50%)
• Canceled projects (35%)

Excessive paperwork has no exact definition, but paperwork can be considered to be excessive under the following conditions: (1) more than 50 discrete document types are produced; (2) the cost of paperwork approaches or exceeds 50% of the total software project expenses; (3) more than 2000 words are created per Function Point. System software ranks behind military software in the magnitude of paperwork production, with most of the paperwork appearing to be redundant or overlarge for the work at hand. (Note: a more subtle derivative problem that underlies excessive paperwork: to date, there has been no published literature on the optimum quantity, volume, structure, or style of the set of paper documents that surround software projects.)

Commercial Software Risks

• Inadequate user documentation (70%)
• Low user satisfaction (55)
• Excessive time to market (50%)
• Harmful competitive actions (45%)
• Litigation expense (40%)

Inadequate user documentation is defined as user information which is incomplete, unclear, wrong, or inconvenient. User information includes on-line help and published material. This is the most widespread problem of the commercial s/w world. This problem can be traced to the following factors:

• technical writing is such a comparatively scarce skill.
• user manuals normally inadequate:
• Documenting the first release of any new software package is notoriously difficult;
• Some vendors canít afford or donít use competent technical writers and illustrators;
• The state of the art of s/w user documentation is still primitive and evolving.

Low user satisfaction means clients are displeased with one or more of the following factors (as of 1993, some of these problems occurred on more than half the commercially-marketed software):

• low quality
• inadequate functionality
• complex or arcane command structures
• steep learning curves
• tricky or troublesome installation procedures
• poor service or customer support
• excessive utilization or monopolization of disk space or other hardware components by the software.

Military Software

MILSPEC are fairly rigorous standards and ensure reasonable consistency from project to project, they also create a number of very expensive problems and create risks in their own right.

• Excessive paperwork (90%)
• Low productivity (85%)
• Long schedules (75%)
• Creeping user requirements (70%)
• Unused or unusable software (45%)

Excessive paperwork -- large military projects routinely create up to 100 discrete document types and more than 6 pages of paper per function point. This paperwork, in all forms, eats up 53% of the total cost of military software projects. While Jones acknowledges some paperwork adds value and rigor to process; however, most appears to be redundant. He also observed that itís tougher to delete from military standards than add to military standards.

Unused or unusable software is defined as software projects which, when delivered, are in fact not used by the intended user community. This is common to governmental software in general and the military software in particular. This tendency is due partly to long time span to complete military projects, such that the intended users have changed jobs or the jobs themselves have changed to the point where the software has little to no utility. Additionally, some of the hardware for which the s/w was created has become obsolete. This long project time is being further extended by litigation surrounding contract awards, which is on the increase.

Contract or Outsourced S/W risks

• High maintenance costs(60%)
• Friction between contractor and client personnel (50%)
• Creeping user requirements (45%)
• Unanticipated acceptance criteria (30%)
• Legal ownership of s/w and deliverables (20%)

Maintenance costs are defined as projects where annual maintenance costs for defect repairs and minor updates are notably higher than U.S. norms, and where the amount of existing software which one person can maintain is notably lower than U.S. norms.

Unanticipated acceptance criteria is defined as new and sometimes stringent criteria levied on a contractor by a client as a condition of product acceptance or payment of funds, outside the scope of the initial contract or agreement. Examples of unanticipated acceptance criteria can include very stringent quality or performance targets for the software, or the need to retroactively specify or document the software. This situation can result from a failure to establish criteria early in the contract, or can be due to hard feelings on part of client that the work was not satisfactorily performed. This can result in damage to the project, client-contractor relationship and in extreme cases litigation or contract termination.

End User S/W risks

• Nontransferable applications (80%)
• Hidden errors (65%)
• Unmaintainable software (60%)
• Redundant applications (50%)
• Legal ownership of software and deliverables (20%)

Hidden errors are defined as logical or programming errors that are latent within an end-user application without the knowledge of the author or anyone else. This is extremely common due to lack of formal reviews, inspections, testing, audits, or QA activity.

Unmaintainable software. Once the software author leaves the company, who is then responsible for its maintenance? Some applications are so poorly structured, commented, and specified that maintenance becomes essentially impossible after the original author leaves.

Risk Management ñ Boehmís Six Steps

As Jones assessments show, quality assurance activities do pose a direct or indirect risk to the software development process. Current software risk management has been adapted from concepts, practices, and principles which has been successfully used within other engineering and manufacturing arenas. The objectives of software risk management is to identify, address, and eliminate potential elements of risk before they become either threats to successful software operation or major sources of software rework. Such threats are stated in terms of ìrisk exposure,î ìrisk impactî or ìrisk factor.î Regardless of the term, it is defined by the relationship of the probability of an unsatisfactory outcome for a particular item and the loss to the affected players if the outcome is unsatisfactory. (This relationship can be displayed by a family of probability versus loss curves). A decision tree is constructed showing the combined risk exposure for each decision option, where combined risk exposure is the sum of individual risk exposures obtained from specified probabilities of occurrence). This decision tree provides a numeric discriminator between the various options, as well as facilitate sensitivity analyses of the preferred option to the risk exposure parameters. Such analysis is of particular benefit when the occurrence probabilities and loss costs are not well enough known to permit precise analysis.

Boehm lays out a six step risk management process consisting of two primary steps, each having three subsidiary steps, as a means of implementing the concept in practice. Boehm further recommends a number of techniques appropriate to implement each primary and subsidiary step. The first step is risk assessment, which consists of:

• Risk identification, which identifies specific project risk items that are likely to endanger a programís chances of success;
• Risk analysis, which determines the probabilities of individual and consolidated loss probabilities and costs for each risk item;
• Risk prioritization produces a ranked ordering of risk items that were identified and analyzed;

Once a prioritized listing of project risk items has been developed, the second step, risk management, is initiated. This step, used to bring these risk items under control, consists of:

• Risk management planning, which outlines how each individual item of risk will be addressed, and how individual risk plans are integrated into the overall project management plan;
• Risk resolution in which implemented actions or activities either eliminate or resolve the risk involved with the particular item; and
• Risk monitoring, where the projectís progress towards resolving risk items or taking corrective action is tracked.

Application of Risk Management to Quality Factors

As we noted in the Quality Factors and Risk section of this paper, several generic classes of software development either directly or indirectly suffered from quality- or software quality assurance-related problems, as well as the factors causing those problems. In this section, we will propose some techniques that can help control, mitigate, or prevent these risks. (Jones)

Element: Creeping User requirements:

Risk mitigation techniques:

• Use of prototypes;
• Use of JADS for requirements creation in MIS domain;
• Use of Information Engineering (IE) methodologies for requirements creation -- again, used primarily in the MIS domain.
• Use of functional metrics to monitor requirements growth. Since metrics are normally quantified during requirements phase, research now underway to couple and automate requirements gathering process and metrics creation. Now appears technically feasible to build requirements automation tools built around tables of common functions and features. Advantages of such tools: add speed and rigor to requirements gathering, but feed data to both Function Point enumerators and cost estimators--could also feed into CASE, data modeling, and design tools.
• New technique -- base contract on number of function points to be delivered and estimated cost per function point. This will force the user to recognize that requirements creep comes at a financial, as well as schedule, cost.

Element: Low Quality and Error-Prone Modules

Risk Mitigation techniques: Quality control on critical path to schedule reduction and cost control. Four classes of technologies that have proven to be effective for software quality control:

• Quality estimation and reliability estimation tools. Quality/reliability estimation tools are relatively new to market (only 6 commercial grade tools in 1993) used by less than 10% of all software development managers.
• Defect prevention methods. Defect prevention includes all technologies which can reduce the chance of making mistakes or errors and include: (a) any of the structured analysis and design techniques; (b) prototypes; (c) high-level and object-oriented languages; (d) rigorous usage of structured coding techniques for procedural languages; (e) the quality function deployment (QFD) approach; (f) the TQM approach; (g) SQA departments; and (h) clean room development methods.
• Defect removal methods. Defect removal methods includes design reviews, structured walkthroughs, formal code inspections, correctness proofs, and all forms of testing. Formal reviews and inspections have the highest measured defect removal efficiencies of any forma of defect removal, and are in fact used by all of the U.S. quality leaders. Testing works best if carried out by trained specialists in a formal manner.; and
• Quality measurement programs. Jones noted that all U.S. leaders in software quality control (i.e., Bladrige Award winners) have established full quality measurement programs. One of the newest extensions of functional metrics is in the quality domain. The older ìlines of codeî metrics were so ambiguous and paradoxical for measuring requirements , design, and documentation error levels that there was essentially no literature on many important quality topics. Function points were used in 1991 to establish U.S. national quality averages or military systems and MIS projects. As of 1993, functional metrics are also being used to measure and predict the number of test cases and test runs for s/w projects.

Wallmueller, Ernest. Software Quality Assurance ñ A practical approach. Prentice Hall, Inc 1994. ISBN 0-13-819780-6

Schulmeyer, G. Gordon Zero Defect Software. McGraw-Hill, Inc. 1990. ISBN 0-07-055663-6

Glass, Robert. Building Software Quality. Prentice-Hall, Inc. 1992. ISBN 0-13-086695-4

Boogaard, Martin. Defusing the Software Crisis ñ Information Systems flexibility through data independance. Thesis Publishers, Amsterdam, 1994. ISBN 90-5170-289-2

Curran, E. and Sanders, J. Software Quality: A framework for success in software development and support. Addison-Wesley Publishing Co., Inc. 1994, ISBN 0-201-63198-9

Blackman, M., Jeffreys, M. Quality systems by prototyping. Extracted from Software Quality
Management, Elsevier Science Publishers, London. 1993 ISBN 1-85166-963-9

Vidgen, R.T. and Wood-Harper, A.T., Establishing and managing a relevant notion of quality Extracted from Software Quality Management, Elsevier Science Publishers, London. 1993 ISBN 1-85166-963-9