Capability Models: ISO and CMM

Dr. William Hankley

CIS 841

Software Validation and Verification, Summer 1998

INTRODUCTION 

VOCABULARY 

TUTORIAL 

TOOLS 

EXAMPLES

CONCLUSION

EVALUATION 

REFERENCES 

TIMELOG 
 
 

STATEMENT

The software development industry is famous for being out of control in terms of   scheduling, cost estimation and reliability.  Many software organizations are investing a large amount of resources implementing software practices and methodologies that can bring discipline to their operations.  These organizations are turning to industry standards to improve their software management process.

The Capability Maturity Model (CMM) by the Software Engineering Institute and the International Standard Organization (ISO) with its 9000 series share a common concern with software quality improvements.  The purpose of this report is to present an overview of the two standards mapping their correlation and identifying their differences.  In addition, this paper reviews several tools available in the market to track the compliance of the ISO9000 and CMM standards

INTRODUCTION

The CMM and ISO9001 methods were introduced in 1987 with the purpose of assessing the ability of suppliers to meet their commitment and requirements of their customers. 

In September 1987, SEI released a brief description of the process maturity framework [1] which was later expanded in Humphrey's book, Managing the Software Process [2]. Two methods, software process assessment and software capability evaluation and a maturity questionnaire were developed to appraised software process maturity.  In the same year, ISO published the 9000 standard describing the key requirements for companies and organizations doing business in the European community.   In 1991, ISO released another standard, the ISO9000-3 Guidelines for the Applications of ISO9001 to the Development, Supply, and Maintenance of Software.

Software organizations, especially in Europe and the US, have raised the  concern on how to evolve their processes to meet the requirements of both models.   Many organizations that are ISO compliant try to map their certification with levels in the Capability Model; however, because each model addresses topics not included in the other, compliance with one model does not guarantee the compliance with the other.   This document provides the necessary information to identify the similarities and differences of both models.  However, this document is not intended to be a tutorial for implementation or registration of either model.

VOCABULARY

1. software process:  The process or set of processes used by an organization to plan, execute, monitor control and improve its software related activities.
2. process assessment:  Is an appraisal to determine the state of an organization's  current software process, to determine the high-priority software process related issues facing an organization, and to obtain the organizational support for process improvement.
3. capability level:  A set of common features (i.e. generic practices) that works together to provide a major enhancement in the capability to perform a process.
4. capability evaluation:  Is an appraisal to identify contractors who are qualified to perform the software work or to monitor the state of the software process used on an existing software effort.
5. process:  A set of activities.
6. process capability:  The range of expected results that can be achieved by following a process.
7. assessed capability:  The output of one or more recent, relevant process assessments conducted in accordance with an standard.
8. base practice:  A management activity that directly addresses the purpose of a particular process and contributes to the creation of its output.
9. practice:  A management activity that contributes to the creation of the output (work products) of a process or enhances the capability of a process.

TUTORIAL

The Capability Maturity Model

The Capability Maturity Model (CMM) is a framework of software development and process management.   Companies that  implement the CMM want to improve their ability to meet cost, schedule, and product functionality goals. The CMM defines five levels of maturity based on process capability.  Except for Level 1, each maturity level is composed of several key process areas that indicate where the company should focus its improvement. A company first evaluates which level it is currently functioning at, then works toward advancing to the next level by mastering significant skills and process.   The following characterizations of the five maturity levels highlight the primary process changes made at each level  

• Level 1: Initial The software process is characterized as ad hoc, and occasionally even chaotic. Few process are defined, and success depends on individual effort.  
• Level 2: Repeatable Basic project management processes are established to track cost, schedule, and functionality. The necessary process discipline is in place to repeat earlier successes on projects with similar applications.  
• Level 3: Defined The software process for both management and engineering activities is documented, standardized, and integrated into a standard software process for the organization. All projects use an approved, tailored version of the organization's standard software process for developing and maintaining software. 
• Level 4: Managed Detailed measures of the software process and product quality are collected. Both the software process and products are quantitatively understood and controlled.  
• Level 5: Optimizing Continuous process improvement is enabled by quantitative feedback from the process and from piloting innovative ideas and technologies.

Figure 1.  The five levels of Software Process Maturity

 

Key Process Areas:  Except for level 1, each maturity level is composed of key areas that an organization must address to achieve the corresponding maturity level.  

Level 2:  The key process areas at level two are focused on establishing basic project management controls.  Those key areas are:

• Requirements Management
• Software Project Planning
• Software Project Tracking and Oversight
• Software Subcontract Management
• Software Quality Assurance
• Software Configuration Management

Level 3:  The key process areas at level three address project and  organizational issues, as the organization establishes an infrastructure for software engineering across all projects.  Those key areas are:

• Organization Process Focus
• Organization Process Definition
• Training Program
• Integrated Software Management
• Software Product Engineering
• Intergroup Coordination
• Peer Reviews

Level 4:  The key process areas at level four focus on establishing quantitative understanding of the software process and the software work products being built.  Those key areas are:

• Quantitative Process Management
• Software Quality Management.

Level 5:  The key process areas at level five cover the issues that both the organization and the projects must address to implement continuous and measurable software process improvements.  Those key areas are;

• Defect Prevention
• Technology Change Management
• Process Change Management

International Standards Organization (ISO) 9000 series

The International Standards Organization (ISO) 9000 series of standards is a set of documents dealing with quality systems that can be used for external quality assurance purposes. They specify quality system requirements for use where a contract between two parties requires the demonstration of a supplier's capability to design and supply a product. The two parties could be an external client and a supplier, or both could be internal, e.g., marketing and engineering groups in a company.  

ISO relies on audits to provide assurance that an organization is meeting the requirements stated in the model.  In an audit, the documents and records that make the quality system are reviewed, but even more important, the audit inspects the way people work in the organization and the knowledge they have about the operation of the quality system.

The following list provides some tasks that are required to comply with ISO 9000:

• writing a quality manual, describing the organization's quality system at a high level
• writing procedure documents describing the work is carried out in the organization
• creating a system to control distribution and re-issue of documents
• identifying training needs for most positions in the organization
• training people in the organization on operation of the quality system
• planing and conducting internal audits

There are several standards and guidelines in the ISO 9000 series, including ISO 9000, ISO 9001, ISO 9002, ISO 9003, ISO 9004, ISO 8402. There are also a number of guides, such as ISO 9000-3, which are additional parts to standards in the ISO 9000 series. 

ISO 9000, "Quality management and quality assurance standards - Guidelines for selection and use," clarifies the distinctions and interrelationships between quality concepts and provides guidelines for the selection and use of a series of international standards on quality systems that can be used for internal quality management purposes ( ISO 9004 ) and for external quality assurance purposes (ISO 9001, 9002 and 9003 ). The quality concepts addressed by these standards are: 

• An organization should achieve and sustain the quality of the product or service produced so as to meet continually the purchaser's stated or implied needs.
• An organization should provide confidence to its own management that the intended quality is being achieved and sustained.
• An organization should provide confidence to the purchaser that the intended quality is being, or will be, achieved in the delivered product or service provided. When contractually required, this provision of confidence may involve agreed demonstration requirements.

ISO 9001, "Quality systems - Model for quality assurance in design/development, production, installation, and servicing," is for use when conformance to specified requirements is to be assured by the supplier during several stages, which may include design, development, production, installation, and servicing. Of the ISO 9000 series, it is the standard that is most pertinent to software development and maintenance.  

ISO 9002, "Quality systems - Model for quality assurance in production and installation, " is for use when conformance to specified requirements is to be assured by the supplier during production and installation. 

ISO 9003, "Quality systems - Model for quality assurance in final inspection and test," is for use when conformance to specified requirements is be assured by the supplier solely at final inspection and test. 

ISO 9004, "Quality management and quality system elements - Guidelines," describes a basic set of elements by which quality management systems can be developed and implemented. 

ISO 9000-3, provides "Guidelines for the application of ISO 9001 to the development, supply, and maintenance of software." Annexes A and B in ISO 9000-3 cross-reference ISO 9000-3 and ISO 9001. 

Mapping ISO 9001 to the CMM

ISO has 20 clauses that define the areas of process compliance.   The following table is provided by the SEI to map the ISO clauses with key process areas in the CMM.  A strong relationship indicates that both ISO and CMM address the particular area that is being compared; a weak relationship indicates that there is not a clear mapping between the two models, but the requirement can be met under a different category.

ISO 9001 Clause	Strong Relationship	Weak Relationship
4.1: Management Responsibility	Commitment To Perform, Software Project Planning, Software Project Tracking and Oversight	Ability To Perform, Verifying Implementation, Software Quality Management
4.2: Quality System	Verifying  Implementation, Software Project Planning, Software Quality Assurance, Software Product Engineering	Organization Process Definition
4.3: Contract Review	Requirements Management, Software Project Planning	Software Subcontract Management
4.4: Design Control	Software Project Planning, Software Project Tracking and Oversight, Software Configuration Management, Software Product Engineering	Software Quality Management
4.5: Document and Data Control	Software Configuration Management, Software Product Engineering
4.6: Purchasing	Software Subcontract Management
4.7: Control of Customer-Supplied Product	(none)	Software Subcontract Management
4.8: Product Identification and Traceability	Software Configuration Management, Software Product Engineering
4.9: Process Control	Software Project Planning, Software Quality Assurance, Software Product Engineering	Quantitative Process Management, Technology Change Management.
4.10: Inspection and Testing	Software Product Engineering, Peer Review
4.11: Control of Inspection, measuring and Test equipment	Software Product Engineering
4.12: Inspection and Test Status	Software Configuration Management, Software Product Engineering
4.13: Control of Nonconforming Product	Software Configuration Management, Software Product Engineering
4.14: Corrective and Preventive Action	Software Quality Assurance, Software Configuration Management	Defect Prevention
4.15: Handling, storage, packaging, preservation, and delivery	(none)	Software Configuration Management, Software Product Engineering
4.16: Control of Quality Records	Software Configuration Management, Software Product Engineering, Peer Reviews
4.17: Internal Quality Audits	Verifying Implementation, Software Quality Assurance
4.18: Training	Ability to Perform, Training Program
4.19: Servicing	(none)
4.20: Statistical Techniques	Measurement and Analysis	Organization Process Definition, Quantitative Process Management, Software Quality Management

Although there is a great overlap between ISO and CMM, there are some areas in the ISO9000 model that are not covered in the CMM and vice-versa.    Most of the clauses of the ISO model can be mapped into key practices in the first three levels of the CMM.  Implementing CMM up to level three almost guarantees a certification in ISO 9000 assuming the organization will also add processes to comply with the ISO clauses that deal with "control of customer supplied product (4.7)" and "handling, storage, packaging, preservation, and delivery (4.15)".  However, the higher levels of CMM are not addressed in ISO 9000.   Also there is controversy among ISO and CMM when it comes to measuring the coverage and depth of the audits from the two organizations.  The argument usually employed by the CMM is that because ISO is a standard that applies to many different industries besides software, auditors are not necessarily well trained when it comes to evaluating key areas in the software quality process such as design reviews.

Another fundamental difference between ISO and CMM is the existence of levels of quality in CMM and the lack of that classification in the ISO.  The essence of the CMM is that several quality levels can be recognized.  An organization can plan a quality improvement process based on the levels of maturity.  In ISO there is not such classification.  ISO 9000 series identifies a minimal set of requirements for a quality system and does not necessarily stress the need for a continuous process improvement.

TOOLS

Neither the ISO9000 or the Capability Maturity Model standards dictate on the tools that must be used to achieve process compliance. Most of the software organizations that are ISO or CMM certified tailor their own quality process combining existing tools with new ones to meet the requirements for all or some of the areas of certification. There are literally hundreds of tools in the market that can help organizations achieve their goal. Tools range all the way from complete solutions for all CMM levels or ISO categories, to more specialized tools that provide support for only a particular area.

Although the authors of this article were not able to test any of the tools available in the market, due to high costs and the lack of free trial packages, we are classifying the tools under several categories giving and example of a tool for such category:

Tools for ISO compliance:

1. Complete Integrated Environment:  These are tools that provide all-in-one package features for defining, tracking and controlling software processes for ISO certification.

• Computer Integrated Quality Solutions: QMS/9000+ (http://www.cnj.digex.net/~keane/). QMS/9000+ is composed of 23 quality application Modules, each of which executes procedural, record keeping and reporting tasks needed to comply with its respective ISO/QS 9000 clause requirements. The modules available under this package are Management Reporting Module, Quality Manual Framework, Contract Review Module Design Support Function, Document Control Module, Supplier Ratings Module, Inspection & Test Module, ProductId/Traceability Module, Statistical Process Control Module, Inspection & Testing Module, Calibration Module, Inspect/Test Status Module, Nonconformance Control Module, Corrective Action Module, Product Life/Stability Module, QMS Basic Module and Database, Audit Module, Training Module, Product Servicing Module, Statistical Analysis Module

  One of the key features of QMS/9000 is that it provides an integrated environment for ISO implementation and tracking. Integrated environments tend to be more user friendly but less flexible when it comes to interactions with other systems. We have requested a CD trial version of this product, but at the time this review was written, no information has been received yet.

2. Audit Support Tools:  These are tools that assist when preparing audit materials for an ISO 9000 quality system.   These tools are generally used to help organizations establishing their own internal audit program.

• The ISO 9000 Quality System Checklist - http://www.iso9000checklist.com/ckmenu.htm: The ISO 9000 Quality System Checklist is a complete tool for setting up an internal auditing program and training auditors. It contains all the materials needed to establish and maintain an ISO 9000 auditing program. This product assumes that the person responsible for setting up the ISO 9000 audit program has minimal or no knowledge of ISO 9000. A project time line is included that tells the project manager exactly what to do and what materials to use.

3. Assessment:  Process assessment tools: This tools generally evaluate the existing process and provide a suggested path to achieve ISO certification.

• The Assessor for ISO 9000 - http://www.cgocable.net/~sdonnell/index.html.   The Assessor for ISO 9000 is a Windows based software tool that can help develop your own ISO 9000 road map. Determine if you really need ISO 9000, or to what classification you should be certified. The Assessor will educate you and provide a customized approach.

4. Document Control Systems:  These tools provide a centralized control system for documentation management.  Some of the most frequent features found in these tools are document linkage, queries, report generation and version control.

• Procedure WRITE - http://www.comprose.com/write.htm.  ProcedureWRITE 2.0 is database software system specially designed for creating, managing and publishing structured documents including processes, procedures, work instructions, corporate and departmental policies, job descriptions, and more.   This product allows you to create, manage and publish documents whether in hard copy form or electronically over a LAN, intranet or internet.

Software Management Tools: This category groups tools that are not necessarily design to support an standard such as ISO or CMM.  In general software management tools help organizations organizing elements of a software process such as risk management, metrics, and quality assessment.

1. CHECKPOINT - www.spr.com/html/checkpoint.htm:  The tool provides three capabilities for managing the software cycle:

• Estimation:  Checkpoint predicts effort at four levels of granularity: project, phase, activity, and task.   Estimates also include resources, deliverables, defects, costs, and schedules.
• Measurement:  Checkpoint enables users to capture project metrics to perform benchmark analysis, identify best practices, and develop internal estimation knowledge bases (known as Templates).
• Assessment:  Checkpoint facilitates the comparison of actual and estimated performance to various industry standards included in the knowledge base. Checkpoint also evaluates the strengths and weaknesses of the software environment.  Process improvement recommendations can be modeled to assess the costs and benefits of implementation.

EXAMPLES

The following list contains the results of organizations that successfully implemented the Software Capability Maturity Model to drive software quality improvements:

1. Improvement Found by the U.S. Air Force[3]. 
   • Explored relationship between SEI levels and project success as measured by cost and schedule performance.
   • Data set: 11 DOD contractors who had been rated by CMM protocols and   31 software projects examined
   • Findings summary:

     The more mature the organization is, the more likely it will meet planned budget and schedule.

     The least mature organizations were likely to have difficulty adhering to cost and schedule baselines; the more mature organizations were likely to have on-baseline cost and schedule performance.

2. Boeing Results [4].

• Customer Satisfaction at SEI Level 1: 87% positive
• Customer Satisfaction at SEI Level 3: >97% positive
• Defect Containment Effectiveness at Level 1: 31%
• Defect Containment Effectiveness at Level 3: 72%
• Effort Reduction in Level 3 compared with Level 1: 31%
• Development Interval Reduction in Level 3 compared with Level 1: 36%

3. Lucent Technologies: Autoplex 1000 project (1987-1994)[5].

• This organization develops software for the switch component of the Autoplex 1000 cellular system. The organization consists of approximately 200 developers and support personnel.  The results found after implementing the CMM methodology are:

  System outage time reduced by factor of 30

  Customer satisfaction dramatically improved

  Reduced development intervals by 1/3

CONCLUSION

There is enough evidence in the software industry of projects that have improved their overall quality by implementing processes based on the ISO 9000 or the SEI's CMM.  Examples such as the ones presented in this article are just a small subset of organizations that have been able to reduce software bugs, improve schedule estimation, and reduce software cycle.  However, a process certification does not guarantees quality improvement by itself.  Measurable improvements in a software process should be the continuous goal of an organization. 

Some of the criticism that follows ISO and the CMM model is that they do not scale well for small organizations or that the cost of implementing either process outgrows the benefits.  The CMM, sponsored by the Department of Defense and the SEI,  is perceived as a sector-specific practice guide for large, software-intensive projects.  ISO, on the other hand, has the reputation of being a model that requires process compliance and not process improvement. 

What should we expect in the future coming from ISO and SEI?  The SEI and the ISO organization are currently working on initiatives that attemp to address the concerns expressed before.  ISO is developing a suite of standards for software process assesment under the SPICE  project.  The SPICE project intends to harmonize the efforts and models around the world to manage the software process.   The goal of SPICE is to provide a guideline that will be software sector independant.  On the other hand, the SEI is currently working on developing an integrated framework for maturity models and associated products.  The new SEI initative is called the "Capability Maturity Model Integration(CMMI) Product Suite" and it will  consist of a framework for integrating different sector especific models.  The CMMI will allow orgazations to tailor their process using the framework. 

What it is clear is that  in the years ahead, standards and certificates should take into account the diversity of software projects in the real-world and the need to track continuous process improvement.    Standards will have to accommodate to this reality and organizations trying to implement a new quality process should assess carefully the benefits of implementing either or both process models.

EVALUATION

Self Evaluation  

(1) How many levels are in the Capability Maturity Model (CMM)?  

(a) 1 Level

(b) 2 Level

(c) 7 Level

(d) 5 Level

Key

(2) which level is it in when the software process in ad hoc, and occasionally even chaotic state? 

(a) 5 Level

(b) 4 Level

(c) 1 Level

(d) 2 Level

Key

(3) what is the process principle of Level 2 Repeatable? 

(a) repeat earlier successes on projects with similar applications. 

(b) the software process are quantitatively understood and controlled. 

(c) few process are defined, and success depends on individual effort.

(d) products are quantitatively understood and controlled.

Key

(4) which of the following activities happen to software process in Level 3? 

(a) documented into a standard software process.

(b) standardized into a standard software process.

(c) integrated into a standard software process.

(d) documented, standardized, and integrated into a standard software process.

Key

(5) where should the feedback from for continuous improvement made in Level 5?

(a) the process.

(b) piloting innovative ideas and technologies.

(c) the process and piloting innovative ideas and technologies.

(d) do not need feedback.

Key

(6) In ISO 9000 series, which one is most pertinent to software development and maintenance?            

(a) ISO 9000.

(b) ISO 9001.  

(c) ISO 9003. 

(d) ISO 9004. 

(e) ISO 9002. 

(7) Which one in the following choice is the guide to interpret ISO 9001? 

(a) ISO 9000.

(b) ISO 9002.  

(c) ISO 9003. 

(d) ISO 9004. 

(e) ISO 9000-3. 

Key

(8) Which one below describe the characterization of ISO 9001? 

(a) Quality management and quality assurance standards - Guidelines for selection and use. 

(b) Quality systems - Model for quality assurance in production and installation. 

(c) Quality management and quality system elements - Guidelines. 

(d) Quality - Vocabulary. 

(e) Quality systems - Model for quality assurance in design/development, production installation and servicing. 

(f) Quality systems - Model for quality assurance in final inspection and test. 

Key

(9) Which one below describe the function of ISO 9001? 

(a) Defines the basic and fundamental terms relating to quality concepts, as they apply to products and services, for the preparation and use of quality standards and for mutual understanding in international communications. 

(b) Is for use when conformance to specified requirements is to be assured by the supplier solely at final inspection and test. 

(c) Is for use when conformance to specified requirements is to be assured by the supplier during several stages, which may include design, development, production, installation, and servicing. 

(d) Clarifies the distinctions and interrelationships between quality concepts and provides guidelines for the selection and use of a series of international standards on quality systems that can be used for internal quality management purposes and for external quality assurance purposes. 

(e) Is for use when conformance to specified requirements is to assured by the supplier during production and installation. 

(f)Describes a basic set of elements by which quality management systems can be developed and implemented.

 

Key

 

(10) Which one below is an internal quality management? 

(a) ISO 9001. 

(b) ISO 9002. 

(c) ISO 9003. 

(d) ISO 9004. 

(e) ISO 8402. 

Key

(11) As a software management tool, what are the capabilities Checkpoint provides?

(a) Estimation, Assessment.

(b) Measurement.

(c) Assessment.

(d) Estimation, Measurement, Assessment.

Key

(12) Which key process area is in the Capability Maturity Model (CMM) Level 2?

(a) Organization Process Focus.

(b) Organization Process Definition.

(c) Software Project Planning.

(d) Software Quality Management.

Key

(13) What are the key process areas in Capability Maturity Model (CMM) Level 4?

(a) Software Project Planning.

(b) Measurement.

(c) Quantitative Process Management

(d) Quantitative Process Management, Software Quality Management.

Key

(14) What are the benefits Lucent Technologies gets from implementing Capability Maturity Model?

(a) System outage time reduced.

(b) Customer satisfaction improved.

(c) Reduced development intervals.

(d) (a), (b) , (c).

Key

(15) Which of the following key process areas in CMM have strong relationship with ISO 9001 Clause 4.2 Quality System?

(a) Verifying Implementation, Software Project Planning, Software Quality Assurance, Software Product Engineering.

(b) Software Project Planning, Software Quality Assurance.

(c) Training.

(d) Software Project Planning.

Key

REFERENCES

[1].  W.S. Humphrey, Characterizing the Software Process: A Maturity Framework, Software Engineering Institute, CMU/SEI-90-TR-24, ADA182895, June 1987.  Also published in IEEE Software, Vol. 5, No 2, March 1988, pp.73-79.

[2].  W.S. Humphrey, Managing the Software Process, Addinson Wesley, Reading, MA, 1989.

[3].  1994 Air Force Institute of Technology Study.   Software Process Newsletter, Committee on Software Process; No. 7, Fall, 1996, IEEE Computer Society TCSE.

[4].  From a talk given at SEPG 97 by John Vu, Associate Technical Fellow, Boeing Corp.

[5]. The Autoplex 1000 Switch Quality Story, Bill Skeen's, Autoplex; George Tucker, QUEST; August, 1994.

Some important links in software engineering:

• Software Engineering Institute http://www.sei.cmu.edu 
• International Standards Organization (ISO) http://www.iso.ch  
• ISO Easy www.seattleu.edu/~kimayars/cmm/cmm.html 
• SEI Capability Maturity Model http://www.qpmg.com/sei.htm 
• CMM online  Tutorial  http://www.ati.stevens-tech.edu/pal/lessn_2/sld001.htm
• ISO 9000 New Service http://www.iso.ch/9000e/moreinfo.htm
• ISO 9000 News http://www.iso.ch/9000e/9000697e.pdf
• ISO Country Code http://www.bcpl.lib.md.us/~jspath/isocodes.html
• California ISO 9000 Consulting Service http://www.caliso9000.com
• ISO 9000 Network http://www.isonet.com
• ISO 9000 Group http://www.commerce-associates.com

TIMELOG

Key

(1) d (2) c (3) a (4) d (5) c (6) b (7) e (8) e (9) c (10) d (11) d (12) c (13) d (14) d (15) a