3.3 Technical Explanations

Each assertion listed in the Technical Performance Checklist is supported by a

Cause Description. In the case of the performance checklist, the support is to

broaden and deepen the understanding of the assertion. Following are explanations

of the assertions found in the Technical Performance Checklist.

51 ARCHITECTURE

51a All Critical Success Factors (CSFs) such as Mean Time To

Repair (MTTR), Mean Time Between Failure (MTBF), etc.,

have been documented and understood.

All Critical Success Factors (CSFs) such as MTTR, MTBF, etc., must have

been documented and fully understood as the same by both parties. When those

CSFs are incorporated into the design, a clear trail must exist from each CSF to

its incorporation into the design.

51b All modules/subsystems are well defined.

All modules/subsystems are well defined whenever all parameters that go

into making up the module or subsystem are understood. While this statement

can be somewhat subjective, it must be answered in objective terms. If there are

parameters that are not understood, they must be defined. Look carefully at the

old saw: ‘‘We know what we know and we know what we don’t know, but our

problems are evidenced when issues arise that we don’t know we don’t know.’’

There is another old saw that states: ‘‘You certainly have a keen grasp of the

obvious.’’ The point is that all issues and considerations must be ‘‘thought

through’’ in order to discover possibilities that are not mentioned. Whenever

one of these possibilities is uncovered, it should be investigated and documented.

51c All key functions such as time, length, weight, performance

requirements, and interfaces, etc., listed in the requirements

are appropriately covered.

All key functions, performance requirements, and interfaces, etc., listed in

the requirements are appropriately covered when all are listed on the ordinate

(the ‘‘Y’’ axis—along the side) of the Requirements Traceability Matrix (RTM)

(see Attachment 7) and the Requirements Flow-Down Matrix (RFM) (see Attachment

8), and the WBS locations are listed on the abscissa (the ‘‘X’’ axis—

across the top or bottom). The requirements are appropriately covered when

there is an ‘‘intersect’’ between all requirements and all the WBS locations

showing dispositions.

51d All major elements (physical and data) are described and

justified.

All major elements (physical and data) are described when they are understood

equally by all parties involved. All major elements (physical and data) are

justified when they contribute to the whole of the unit or system.

51e All key aspects of user interfaces are well defined.

All key aspects of user interfaces are well defined when they follow the accepted

standards established for the industry such as IBM’s User Access Guide1

and/or Microsoft’s Interface Guidelines2 or other appropriate interface guidelines.

Accepted psychological, physical, and human factors standards should be

followed as well.

51f The Architecture hangs together conceptually.

The Architecture hangs together conceptually when the system specification

describes the functional components of the system in terms of their behaviors

and provides component-to-component interfaces resulting in a sum of the

parts to make a whole.

52 DESIGN

52a The design process is correct and traceable to enterprise,

customer, and standard processes.

The design process is correct and traceable to enterprise, customer, and standard

processes whenever the required numbers and types of Design Reviews

and the content of each Design Review are traceable to the standard, the customer,

and the enterprise processes.

The design process is driven by enterprise, customer, and standard policies

and processes. The links between these processes should appear on your Standards

Traceability Matrix. The Standards Traceability Matrix is further explained

in Attachment 13.

52b The design is correct and traceable to the requirements.

The design is correct and traceable to the requirements when every element

of the produced product is directly traceable to a requirement. All the elements

of the design process are brought into focus through the Requirements Traceability

Matrix (RTM). The RTM will trace the requirement from the specification

through the design process, through the qualification (preproduction) test

process, through the manufacturing process, and through the final (operational)

test to end with the final delivery to the customer. The RTM is an inherent part

of the Technical Plan, which is attached to the Project Plan.

Let’s take a moment to look at ethics. As you review the requirements that

you will perform to, you must look at these requirements as a professional.

After all, that’s why you are being hired. If you find a requirement that you

know is wrong or will not work, you have an obligation to advise your customer

of that fact. If you ‘‘sandbag’’ this issue and try to make it up with a change

later on, you are guilty of unethical conduct. Under some cases, you may be

guilty of illegal conduct. To simply trace the requirement to its source is not

sufficient. It must, to the best of your professional knowledge, be a valid requirement.

52c The design is efficient.

The design is efficient when it performs as the requirements document (contract)

demands, has the inherent reliability, maintainability, and availability demanded

by the requirements document (contract) or meets at least the same

qualifications for these factors for competing products, and is economical in its

design and production and throughout its life-cycle. Ensure that all CSFs (See

Cause Description 51a) are incorporated as well.

52d The design adequately addresses issues that were identified

and deferred to design at the architectural level.

The design adequately addresses issues that were identified and deferred to

design at the architectural level when those architectural elements have been

TEAMFLY

defined and are traceable to the design. Further, they must be addressed in the

appropriate Design Review and clearly identified in the design.

52e The design is partitioned into manageable segments.

The design is partitioned into manageable segments when the segments are

logical, can be defined, can be tested, can be scheduled, and can be costed. The

purpose of this partitioning is to create groupings for the Work Breakdown

Structure (WBS) that are manageable and consistent with the resources available

(i.e., assigned internally or subcontracted, outsourced, or purchased as necessary).

52f The design accounts for supportability, Life Cycle Cost

(LCC), total cost of ownership, and future expansions.

The design accounts for supportability, Life Cycle Cost (LCC), total cost

of ownership, and future expansions whenever all these factors are taken into

consideration in the design, production, implementation, and operations and

maintenance alternatives.

The objective of an LCC analysis is to provide a basis for choosing the most

cost-effective approach to the entire life cycle/total cost of ownership system,

product, or unit within the available resources. Sometimes this can include

planned or estimated future expansions as well. Pre Planned Product Improvement

(P3I) and the phasing in of those improvements should be a part of your

up-front planning. The analysis must cover the entire lifespan of the system,

product, or unit.

The LCC process provides a systematic methodology for evaluating and

quantifying the cost impacts of alternative courses of action. It can be used to

support trade-off analyses between several product design configurations or the

sensitivity of a specific design to changes. The LCC can, and probably will, affect

the distribution of costs between up-front design or production costs and field

operation and maintenance costs. Care must be taken to involve the entire lifespan

of the system, product, or unit. Frequently, only design or production

costs are considered, leaving operation and maintenance costs to be added later.

If the specification calls for a Design To Cost (DTC) approach, its result may

well be different from the LCC. Be certain to check with the customer regarding

intent. The customer could have intended LCC but said DTC or some similar

term. The results will likely be different.

John C. Sterling provides an excellent comparison of LCC models at: www.

nissd.com/sdes/papers/deslcc.htm.

52g Technical Performance Measures (TPMs) such as data

retrieval time, weight, error rate, etc., have been defined and

accommodated.

Technical Performance Measures (TPMs) such as data retrieval time, weight,

error rate, etc., have been defined and accommodated whenever the TPMs are

fully understood and appear in the related WBS sections and the related test

procedures.

53 DESIGN REVIEWS

53a All Design Reviews were completed according to required

processes.

All Design Reviews were completed according to required processes when

the events of the Design Review are directly traceable to the requirements stipulated

in standard processes, customer (contract and contract referenced) processes,

and enterprise processes. In the event these processes are not available to

you, see Cause Description 53a (NO).

53b The customer approved each Design Review.

The customer will have approved each Design Review if the customer has

signed a sheet that confirms that the customer (through its representative, if

necessary) agrees to the Design Review package, the Design Review, and the

Design Review minutes, including Design Review action items.

54 IN-PROCESS REVIEWS

54a All required In-Process Reviews were conducted according

to required processes.

All required In-Process Reviews were conducted according to required processes

when the events of the In-Process Reviews are directly traceable to the

requirements stipulated in standard processes, customer (contract and contract

referenced) processes, and enterprise processes.

54b Each In-Process Review was approved by the appropriate

authority.

The appropriate authority will have approved each In-Process Review if the

appropriate authority has signed a sheet that confirms that the appropriate authority

(through their representative, if necessary) agrees to the In-Process Review

package, the In-Process Review, and the In-Process Review minutes, including

In-Process Review action items.

55 PROTOTYPES

55a The prototypes reflect the requirements.

The prototypes reflect the requirements when the customer or client agrees

that the prototype satisfies or demonstrates the requirements.

Prototyping is a technique for gathering and validating requirements

through an early representation of the product. In prototyping, you gather preliminary

requirements that you use to build a representation of the solution—a

prototype. You review this with the customer, who may or may not give you

additional or different requirements. You incorporate these changes and review

them with the customer again. This repetitive process continues for an agreed

number of iterations or until the product meets the customer’s needs. The limitations

are established by the contract, agreement, or understanding.

Gathering requirements can mean the difference between a project’s success

and its failure. Some developers tend to gather requirements forever and some

do not gather them at all. Your project timeline must include the time required

for gathering and developing a comprehensive list of requirements. Developers

typically gather requirements in one-on-one interviews, but you can take advantage

of a number of alternative techniques as well.

The most certain (and slowest) method of keeping up with prototype configuration

or content is to follow the basic guidelines of any product development

(requirement, baseline, change process, incorporation of change, etc.). As

an alternative, it is generally agreed nowadays that the prototypes reflect the

requirements if a physical inventory and/or functional test of the prototype(s)

reflect the actual requirements at critical points.

Prototypes are not the end products; consequently, you should modify, truncate,

or abbreviate the tests to expend the minimum amount of effort and resources

consistent with proving the issue.

55b Prototypes were constructed incrementally.

For software projects, incremental construction is usually a necessity for

complex programs. The purpose of incremental construction is to create modules

that are in themselves testable and provable. This becomes a timesaver

whenever you are testing a complex system, and the test fails somewhere along

the line. Whenever a test fails, the system reverts back to the last increment that

passed its tests. Whenever two tested modules are put together and the test fails,

it is likely that the interfaces are incorrect. It is also possible that the specification

for one or more of the modules is incorrect. All these conditions must be

taken into consideration.

Incremental construction is generally less critical for hardware projects, but

it follows the same general precepts as software projects.

55c Prototype changes were incorporated into the design using

the Change Control Process.

Prototype changes were incorporated into the design using the Change Control

Process (See Technical Cause Descriptions 61a, 61b, and 61c) when the

changes are traceable through the product to the documentation that authorized

the change.

Prototypes should be treated the same as First Articles in their development.

This is particularly true if your process takes the prototype (or Breadboard or

Brassboard) directly to development or production. Granted, many changes are

devised during the prototype process and incorporated into the prototype to

validate their efficacy. Indeed, that’s what the prototype process is all about.

However, if it is concluded that the in-process change should be a part of the

prototype, its constituents must be incorporated at some point through the

Change Control Process.

See Cause Description 55a. See the glossary for definitions of these terms.

55d Each prototype change was reviewed and accepted by the

originator of the requirements.

Each prototype change was reviewed and accepted by the originator of the

requirements whenever a clear trail exists from the requirement to the acceptance.

In the world of prototypes, this is the equivalent of a change process. Many

times, prototype changes will be verbal from the originator. When you get a

verbal requirement, stop and document the requirement, even if it’s just a note.

Otherwise, when you get to the end of the prototype, you will have no idea of

its contents.

If you get a change to the prototype and it does not pan out, don’t discard the

documented requirement. It is a good way to keep up with resources such as computer

time and labor hours that have been consumed and could be reimbursable.

56 SUBCONTRACTS

56a The sum of all subcontracts reflects all tasks allocated.

The sum of all subcontracts reflects all tasks allocated if the totality of all

subcontracts and all work to be performed internally add up to the total requirements

in the requirements document (contract).

The Requirements Traceability Matrix (RTM) is the tool that ties together

the requirements from the requirements document (contract) to where and

how the requirements are being performed. The Work Breakdown Structure

(WBS) should follow the RTM directly with respect to specified equipment or

modules and indirectly with respect to tasks or performance parameters. In the

vernacular, this situation is referred to as the program or project having no

holes or overlaps.

When evaluating subcontracts for holes and overlaps, consideration must be

given to the fact that there will be some overlaps with respect to required processes.

For instance, if a certain quality program is required of the overall program,

it must be ‘‘flowed down’’ to each of the subcontractors. In this case, it

might appear to be an overlap but it is not. It is, in fact, an appropriate allocation

of requirements.

56b Each subcontract contains all tasks allocated.

Each subcontract contains all tasks allocated when the tasks contained in the

subcontract are equal to the assigned tasks from the Requirements Flow-Down

Matrix (RFM) and are reflected in the Requirements Traceability Matrix

(RTM).

57 PURCHASE ORDERS

57a The sum of all Purchase Orders reflects all purchases to be

made.

The sum of all Purchase Orders reflects all commercially available products

to be purchased for the program or project.

The Requirements Traceability Matrix (RTM) is the tool that ties together

the requirements from the requirements document (contract) to where and

how the requirements are being performed. The RTM should follow the Work

Breakdown Structure (WBS) directly with respect to specified equipment or

modules. In the vernacular, this situation is referred to as the program or project

having no holes or overlaps.

When evaluating Purchase Orders for holes and overlaps, consideration must

be given to the fact that there will be some overlaps with respect to required

processes. For instance, if the ‘‘Buy America’’ clause is required of the overall

program, it must be ‘‘flowed down’’ to each of the Purchase Orders. In this

case, it might appear to be an overlap but it is not. It is, in fact, an appropriate

allocation of requirements.

57b Each Purchase Order is complete.

Each Purchase Order is complete and properly written when it contains:

Reference Number, Order Date, Vendor, Contact Information, Name of Item,

Stock (Catalog) Number, Number of Units, Price, Delivery Schedule, Delivery

Location, Purchaser, and Authorizing Signature.

Most companies have preprinted Purchase Order forms. If yours does not,

create your own. Even if your Purchase Order form is nothing more than a

memo, at least it is documentation of what has been ordered and provides a

basis for financial accountability.

58 PRODUCTION/MANUFACTURING

58a All production/manufacturing processes are traceable to

standard, customer, or enterprise processes (see glossary).

All production/manufacturing processes are traceable to standard, customer,

or enterprise processes when the heritage of the process is clearly referenced in

the process.

Manufacturing processes are usually very involved and have tremendous effect

on the final product. If the processes have been in effect for some time and

neither the line nor materials have changed, chances are the processes are okay.

Changing either the line (Cause Description 58b) or the materials (Cause Description

58d) can have an effect on the processes that was unanticipated.

58b The line(s) were properly designed and set up for this

(these) product(s).

The line(s) were properly designed and set up for this (these) product(s) if

the line produces the product according to the requirements.

If the line design, the processes, or the materials have not changed, chances

are that the design of the line is okay. Changing either the processes (Cause

Description 58a) or the materials (Cause Description 58d) can have an effect

on production that was unanticipated. If the line design has changed, there has

likely been an effect on the product that was unanticipated.

58c Shop orders were correct and thorough.

Shop orders were correct and thorough when the end product produced is

the product that was specified by the customer.

Shop orders contain the need for an output product. Shop orders usually

refer back to the specification (specified or derived) that the end product must

meet. Obviously the specifications will be applicable to the end-product in measurement

terms such as size, weight, functionality, etc. Usually, shop orders

refer to the techniques or process, tools, raw products, etc., that must be used

to produce the end product.

Many products require that a number of piece parts be provided to create

the final end product. Shop orders control each step of the overall process to

ensure that the final end product is the product specified. This process is true

of either hardware or software.

Shop orders differ from work orders in that shop orders specify a product to

be provided (e.g., rod, software module, etc.) whereas work orders specify a

service to be provided (i.e., replace light bulb, etc.).

58d The materials were proper for the processes and the

product(s) and do meet the requirements.

The materials were proper for the product(s) when the product meets the

requirements. The materials must meet the requirements of the processes and/

or materials list, and they must be the materials that were specified in the subcontract

or Purchase Order.

59 UNIT TEST

59a Each Unit3 Test correctly reflects the requirement.

Each Unit Test correctly reflects the requirement when each element of the

Unit Test is directly traceable to each element of the unit requirement (specification)

through the Requirements Traceability Matrix (RTM) or Subcontract

Requirements Traceability Matrix (SRTM).

59b Each design element that applies to the routine/module/

subsystem has its own test case.

Each design element that applies to the routine/module/subsystem has its

own test case when there is a direct correlation between the requirement and the

elements tested in the unit, subsystem, or system test through the Requirements

Traceability Matrix (RTM) or Subcontract Requirements Traceability Matrix

(SRTM).

59c Unit Test findings were reviewed for completeness and

forwarded to be incorporated into Subsystem Tests and the

System Test.

Unit Test findings were reviewed for completeness and forwarded to be incorporated

into Subsystem Tests and the System Test when the Unit Tests are

traceable to the next level tests. When the Program Test Plan (PTP) is developed,

it must include ‘‘stacking’’ the qualification and acceptance tests from the

lowest level (Unit Tests) to the highest level (System Test). Care must be taken

in the PTP to ensure that all units or subsystems placed under subcontract as

well as units and subsystems you have developed adhere to this philosophy.

This technique is sometimes called the ‘‘Test Flow Forward Technique.’’

When developing the PTP and the test requirements for the subcontracts,

the Requirements Traceability Matrix (RTM) must be used to account for the

requirements being ‘‘flowed down’’ into the PTP. The PTP must accommodate

the ‘‘flow-down’’ and then, in turn, account for the ‘‘flow-up’’ of the product

responses. The PTP should allow for building the system from the bottom up

through the use of the test flow.

59d All Problem Test Reports (PTRs) were captured,

dispositioned (allocated for action), and worked off.

All Problem Test Reports (PTRs) were captured, dispositioned, and worked

off when there is complete accountability for every error that occurred during

test conduct and every error was captured, dispositioned, corrected, and verified.

The System Test, as written, must subsequently run without error.

60 SYSTEM TEST

60a The System Test Plan/Procedure was approved by the

customer.

The System Test Plan/Procedure was approved by the customer when the

System Test Plan/Procedure has been provided to the customer with lead time

adequate for customer review, and the customer agrees with and approves the

final content. The review/approval cycle may require iteration before approval

is achieved.

60b The System Test is traceable to the requirements.

The System Test is traceable to the requirements when each requirement is

forward traceable through the unit and the subsystem to the system via the

Requirements Traceability Matrix (RTM). Each unit or subsystem requirement

must be tested at least once at the appropriate level (i.e., at the unit level or at

the subsystem level). System level requirements must be visible and backward

traceable to the requirement through the RTM. The exceptions to this statement

are those requirements that are only visible at the system level. Your RTM

should reflect this particular situation by showing the requirement in the leftmost

column and the place where it is tested in the System Test. All columns

in-between will have no entries or dashes.

60c The System Test tested all elements of the system

concurrently.

System Test tested all elements of the system concurrently when all elements

of the system are called into play as they will be whenever the system is operating

in its normal mode.

60d The System Test was performed under appropriate load(s).4

The System Test was performed under appropriate load(s) when the loads

on the system are the loads required by the specification.

60e The System Test was performed using the same kind of

personnel that will be used by the customer.

The System Test was performed using the same kind of personnel that will

be used by the customer with regard to training, education, experience, etc. To

conduct with engineers a system test that was designed to be run by Level 1

technicians is an invalid test even when you follow all the other procedures of

the test. Many times the customer will stipulate that the actual users must perform

the System Test. In that case, there is no question. When you are planning

your project, ensure you have set aside time to train or expose these persons to

the system and its demands.

60f The System Test was properly documented and

incorporated the test results of all prior-level tests.

The System Test was properly documented and incorporated the test results

of all prior-level tests when the results of the Unit Level Tests and the Subsystem

Level Tests are clearly visible in the construct and conduct of the System Test.

The System Test should be an incremental test that relies upon the successful

completion of all Unit and Subsystem Tests operating together under appropriate

system loads.

61 CONFIGURATION MANAGEMENT

61a The Configuration Management Plan (CMP) is thorough,

complete, and authorized.

The Configuration Management Plan (CMP) is thorough, complete, and authorized

when it follows the format required by the customer and/or the enterprise

and maintains the required content as specified in customer or enterprise

configuration management policy. Further, the CMP must be signed by an authority

that is authorized to sign such documents (usually the vice president or

director of engineering, or an equivalent position).

The Configuration Management process could easily be looked upon as the

TEAMFLY

Janus process. Remember the Roman god Janus who looked both backward

and forward? That’s what the Configuration Management process does. It looks

backward to the baseline, as established, and forward to the test process that

will prove the viability of a change.

61b Change requests were presented and approved by an

appropriate level of the Review Board.

Change requests were presented and approved by an appropriate level of the

Review Board when the presentations and approvals followed the Configuration

Management Plan (CMP). See Attachment 5 for more information.

61c Version controls are in place and are reflected on (in) the

product.

Version controls are in place and are reflected on (in) the product when the

affected product is appropriately marked with the version that describes it in

the Version Description Document (VDD) (see glossary) and to which it has

been tested.

62 SYSTEM EFFECTIVENESS FACTORS

62a All required System Effectiveness Factors5 have been

appropriately considered.

To say that all required System Effectiveness Factors have been appropriately

considered, means that all the System Effectiveness Factors have been appropriately

considered in both the product and the processes.

Notes

1. Systems Application Architecture—Common User Access Guide to User Interface Design.

IBM Corporation, 1991 IBM Document Number SC34-4289 (available through

IBM field offices).

2. The Windows Interface Guidelines for Software Design. Redmond, Wash.: Microsoft

Press, 1995.

3. Defined as the smallest stand-alone component that produces a definable output

from a definable input. The unit may be hardware or software. In the case of hardware,

power can be external (i.e., a separate unit).

4. Loads are stresses placed upon a system. Loads are those stresses in units typical

for the product such as pounds, watts, ergs, numbers of subsystems, etc.

5. The System Effectiveness Factors refer to Reliability, Availability, Maintainability,

Supportability (including Logistics), Susceptability, Producibility, Human Engineering,

Safety, and Security.