3.3 Technical Explanations
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67
68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84
85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101
102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118
119 120 121 122 123
Each assertion listed in the Technical Performance Checklist is supported by a
Cause Description. In the case of the performance checklist, the support is to
broaden and deepen the understanding of the assertion. Following are explanations
of the assertions found in the Technical Performance Checklist.
51 ARCHITECTURE
51a All Critical Success Factors (CSFs) such as Mean Time To
Repair (MTTR), Mean Time Between Failure (MTBF), etc.,
have been documented and understood.
All Critical Success Factors (CSFs) such as MTTR, MTBF, etc., must have
been documented and fully understood as the same by both parties. When those
CSFs are incorporated into the design, a clear trail must exist from each CSF to
its incorporation into the design.
51b All modules/subsystems are well defined.
All modules/subsystems are well defined whenever all parameters that go
into making up the module or subsystem are understood. While this statement
can be somewhat subjective, it must be answered in objective terms. If there are
parameters that are not understood, they must be defined. Look carefully at the
old saw: ‘‘We know what we know and we know what we don’t know, but our
problems are evidenced when issues arise that we don’t know we don’t know.’’
There is another old saw that states: ‘‘You certainly have a keen grasp of the
obvious.’’ The point is that all issues and considerations must be ‘‘thought
through’’ in order to discover possibilities that are not mentioned. Whenever
one of these possibilities is uncovered, it should be investigated and documented.
51c All key functions such as time, length, weight, performance
requirements, and interfaces, etc., listed in the requirements
are appropriately covered.
All key functions, performance requirements, and interfaces, etc., listed in
the requirements are appropriately covered when all are listed on the ordinate
(the ‘‘Y’’ axis—along the side) of the Requirements Traceability Matrix (RTM)
(see Attachment 7) and the Requirements Flow-Down Matrix (RFM) (see Attachment
8), and the WBS locations are listed on the abscissa (the ‘‘X’’ axis—
across the top or bottom). The requirements are appropriately covered when
there is an ‘‘intersect’’ between all requirements and all the WBS locations
showing dispositions.
51d All major elements (physical and data) are described and
justified.
All major elements (physical and data) are described when they are understood
equally by all parties involved. All major elements (physical and data) are
justified when they contribute to the whole of the unit or system.
51e All key aspects of user interfaces are well defined.
All key aspects of user interfaces are well defined when they follow the accepted
standards established for the industry such as IBM’s User Access Guide1
and/or Microsoft’s Interface Guidelines2 or other appropriate interface guidelines.
Accepted psychological, physical, and human factors standards should be
followed as well.
51f The Architecture hangs together conceptually.
The Architecture hangs together conceptually when the system specification
describes the functional components of the system in terms of their behaviors
and provides component-to-component interfaces resulting in a sum of the
parts to make a whole.
52 DESIGN
52a The design process is correct and traceable to enterprise,
customer, and standard processes.
The design process is correct and traceable to enterprise, customer, and standard
processes whenever the required numbers and types of Design Reviews
and the content of each Design Review are traceable to the standard, the customer,
and the enterprise processes.
The design process is driven by enterprise, customer, and standard policies
and processes. The links between these processes should appear on your Standards
Traceability Matrix. The Standards Traceability Matrix is further explained
in Attachment 13.
52b The design is correct and traceable to the requirements.
The design is correct and traceable to the requirements when every element
of the produced product is directly traceable to a requirement. All the elements
of the design process are brought into focus through the Requirements Traceability
Matrix (RTM). The RTM will trace the requirement from the specification
through the design process, through the qualification (preproduction) test
process, through the manufacturing process, and through the final (operational)
test to end with the final delivery to the customer. The RTM is an inherent part
of the Technical Plan, which is attached to the Project Plan.
Let’s take a moment to look at ethics. As you review the requirements that
you will perform to, you must look at these requirements as a professional.
After all, that’s why you are being hired. If you find a requirement that you
know is wrong or will not work, you have an obligation to advise your customer
of that fact. If you ‘‘sandbag’’ this issue and try to make it up with a change
later on, you are guilty of unethical conduct. Under some cases, you may be
guilty of illegal conduct. To simply trace the requirement to its source is not
sufficient. It must, to the best of your professional knowledge, be a valid requirement.
52c The design is efficient.
The design is efficient when it performs as the requirements document (contract)
demands, has the inherent reliability, maintainability, and availability demanded
by the requirements document (contract) or meets at least the same
qualifications for these factors for competing products, and is economical in its
design and production and throughout its life-cycle. Ensure that all CSFs (See
Cause Description 51a) are incorporated as well.
52d The design adequately addresses issues that were identified
and deferred to design at the architectural level.
The design adequately addresses issues that were identified and deferred to
design at the architectural level when those architectural elements have been
TEAMFLY
defined and are traceable to the design. Further, they must be addressed in the
appropriate Design Review and clearly identified in the design.
52e The design is partitioned into manageable segments.
The design is partitioned into manageable segments when the segments are
logical, can be defined, can be tested, can be scheduled, and can be costed. The
purpose of this partitioning is to create groupings for the Work Breakdown
Structure (WBS) that are manageable and consistent with the resources available
(i.e., assigned internally or subcontracted, outsourced, or purchased as necessary).
52f The design accounts for supportability, Life Cycle Cost
(LCC), total cost of ownership, and future expansions.
The design accounts for supportability, Life Cycle Cost (LCC), total cost
of ownership, and future expansions whenever all these factors are taken into
consideration in the design, production, implementation, and operations and
maintenance alternatives.
The objective of an LCC analysis is to provide a basis for choosing the most
cost-effective approach to the entire life cycle/total cost of ownership system,
product, or unit within the available resources. Sometimes this can include
planned or estimated future expansions as well. Pre Planned Product Improvement
(P3I) and the phasing in of those improvements should be a part of your
up-front planning. The analysis must cover the entire lifespan of the system,
product, or unit.
The LCC process provides a systematic methodology for evaluating and
quantifying the cost impacts of alternative courses of action. It can be used to
support trade-off analyses between several product design configurations or the
sensitivity of a specific design to changes. The LCC can, and probably will, affect
the distribution of costs between up-front design or production costs and field
operation and maintenance costs. Care must be taken to involve the entire lifespan
of the system, product, or unit. Frequently, only design or production
costs are considered, leaving operation and maintenance costs to be added later.
If the specification calls for a Design To Cost (DTC) approach, its result may
well be different from the LCC. Be certain to check with the customer regarding
intent. The customer could have intended LCC but said DTC or some similar
term. The results will likely be different.
John C. Sterling provides an excellent comparison of LCC models at: www.
nissd.com/sdes/papers/deslcc.htm.
52g Technical Performance Measures (TPMs) such as data
retrieval time, weight, error rate, etc., have been defined and
accommodated.
Technical Performance Measures (TPMs) such as data retrieval time, weight,
error rate, etc., have been defined and accommodated whenever the TPMs are
fully understood and appear in the related WBS sections and the related test
procedures.
53 DESIGN REVIEWS
53a All Design Reviews were completed according to required
processes.
All Design Reviews were completed according to required processes when
the events of the Design Review are directly traceable to the requirements stipulated
in standard processes, customer (contract and contract referenced) processes,
and enterprise processes. In the event these processes are not available to
you, see Cause Description 53a (NO).
53b The customer approved each Design Review.
The customer will have approved each Design Review if the customer has
signed a sheet that confirms that the customer (through its representative, if
necessary) agrees to the Design Review package, the Design Review, and the
Design Review minutes, including Design Review action items.
54 IN-PROCESS REVIEWS
54a All required In-Process Reviews were conducted according
to required processes.
All required In-Process Reviews were conducted according to required processes
when the events of the In-Process Reviews are directly traceable to the
requirements stipulated in standard processes, customer (contract and contract
referenced) processes, and enterprise processes.
54b Each In-Process Review was approved by the appropriate
authority.
The appropriate authority will have approved each In-Process Review if the
appropriate authority has signed a sheet that confirms that the appropriate authority
(through their representative, if necessary) agrees to the In-Process Review
package, the In-Process Review, and the In-Process Review minutes, including
In-Process Review action items.
55 PROTOTYPES
55a The prototypes reflect the requirements.
The prototypes reflect the requirements when the customer or client agrees
that the prototype satisfies or demonstrates the requirements.
Prototyping is a technique for gathering and validating requirements
through an early representation of the product. In prototyping, you gather preliminary
requirements that you use to build a representation of the solution—a
prototype. You review this with the customer, who may or may not give you
additional or different requirements. You incorporate these changes and review
them with the customer again. This repetitive process continues for an agreed
number of iterations or until the product meets the customer’s needs. The limitations
are established by the contract, agreement, or understanding.
Gathering requirements can mean the difference between a project’s success
and its failure. Some developers tend to gather requirements forever and some
do not gather them at all. Your project timeline must include the time required
for gathering and developing a comprehensive list of requirements. Developers
typically gather requirements in one-on-one interviews, but you can take advantage
of a number of alternative techniques as well.
The most certain (and slowest) method of keeping up with prototype configuration
or content is to follow the basic guidelines of any product development
(requirement, baseline, change process, incorporation of change, etc.). As
an alternative, it is generally agreed nowadays that the prototypes reflect the
requirements if a physical inventory and/or functional test of the prototype(s)
reflect the actual requirements at critical points.
Prototypes are not the end products; consequently, you should modify, truncate,
or abbreviate the tests to expend the minimum amount of effort and resources
consistent with proving the issue.
55b Prototypes were constructed incrementally.
For software projects, incremental construction is usually a necessity for
complex programs. The purpose of incremental construction is to create modules
that are in themselves testable and provable. This becomes a timesaver
whenever you are testing a complex system, and the test fails somewhere along
the line. Whenever a test fails, the system reverts back to the last increment that
passed its tests. Whenever two tested modules are put together and the test fails,
it is likely that the interfaces are incorrect. It is also possible that the specification
for one or more of the modules is incorrect. All these conditions must be
taken into consideration.
Incremental construction is generally less critical for hardware projects, but
it follows the same general precepts as software projects.
55c Prototype changes were incorporated into the design using
the Change Control Process.
Prototype changes were incorporated into the design using the Change Control
Process (See Technical Cause Descriptions 61a, 61b, and 61c) when the
changes are traceable through the product to the documentation that authorized
the change.
Prototypes should be treated the same as First Articles in their development.
This is particularly true if your process takes the prototype (or Breadboard or
Brassboard) directly to development or production. Granted, many changes are
devised during the prototype process and incorporated into the prototype to
validate their efficacy. Indeed, that’s what the prototype process is all about.
However, if it is concluded that the in-process change should be a part of the
prototype, its constituents must be incorporated at some point through the
Change Control Process.
See Cause Description 55a. See the glossary for definitions of these terms.
55d Each prototype change was reviewed and accepted by the
originator of the requirements.
Each prototype change was reviewed and accepted by the originator of the
requirements whenever a clear trail exists from the requirement to the acceptance.
In the world of prototypes, this is the equivalent of a change process. Many
times, prototype changes will be verbal from the originator. When you get a
verbal requirement, stop and document the requirement, even if it’s just a note.
Otherwise, when you get to the end of the prototype, you will have no idea of
its contents.
If you get a change to the prototype and it does not pan out, don’t discard the
documented requirement. It is a good way to keep up with resources such as computer
time and labor hours that have been consumed and could be reimbursable.
56 SUBCONTRACTS
56a The sum of all subcontracts reflects all tasks allocated.
The sum of all subcontracts reflects all tasks allocated if the totality of all
subcontracts and all work to be performed internally add up to the total requirements
in the requirements document (contract).
The Requirements Traceability Matrix (RTM) is the tool that ties together
the requirements from the requirements document (contract) to where and
how the requirements are being performed. The Work Breakdown Structure
(WBS) should follow the RTM directly with respect to specified equipment or
modules and indirectly with respect to tasks or performance parameters. In the
vernacular, this situation is referred to as the program or project having no
holes or overlaps.
When evaluating subcontracts for holes and overlaps, consideration must be
given to the fact that there will be some overlaps with respect to required processes.
For instance, if a certain quality program is required of the overall program,
it must be ‘‘flowed down’’ to each of the subcontractors. In this case, it
might appear to be an overlap but it is not. It is, in fact, an appropriate allocation
of requirements.
56b Each subcontract contains all tasks allocated.
Each subcontract contains all tasks allocated when the tasks contained in the
subcontract are equal to the assigned tasks from the Requirements Flow-Down
Matrix (RFM) and are reflected in the Requirements Traceability Matrix
(RTM).
57 PURCHASE ORDERS
57a The sum of all Purchase Orders reflects all purchases to be
made.
The sum of all Purchase Orders reflects all commercially available products
to be purchased for the program or project.
The Requirements Traceability Matrix (RTM) is the tool that ties together
the requirements from the requirements document (contract) to where and
how the requirements are being performed. The RTM should follow the Work
Breakdown Structure (WBS) directly with respect to specified equipment or
modules. In the vernacular, this situation is referred to as the program or project
having no holes or overlaps.
When evaluating Purchase Orders for holes and overlaps, consideration must
be given to the fact that there will be some overlaps with respect to required
processes. For instance, if the ‘‘Buy America’’ clause is required of the overall
program, it must be ‘‘flowed down’’ to each of the Purchase Orders. In this
case, it might appear to be an overlap but it is not. It is, in fact, an appropriate
allocation of requirements.
57b Each Purchase Order is complete.
Each Purchase Order is complete and properly written when it contains:
Reference Number, Order Date, Vendor, Contact Information, Name of Item,
Stock (Catalog) Number, Number of Units, Price, Delivery Schedule, Delivery
Location, Purchaser, and Authorizing Signature.
Most companies have preprinted Purchase Order forms. If yours does not,
create your own. Even if your Purchase Order form is nothing more than a
memo, at least it is documentation of what has been ordered and provides a
basis for financial accountability.
58 PRODUCTION/MANUFACTURING
58a All production/manufacturing processes are traceable to
standard, customer, or enterprise processes (see glossary).
All production/manufacturing processes are traceable to standard, customer,
or enterprise processes when the heritage of the process is clearly referenced in
the process.
Manufacturing processes are usually very involved and have tremendous effect
on the final product. If the processes have been in effect for some time and
neither the line nor materials have changed, chances are the processes are okay.
Changing either the line (Cause Description 58b) or the materials (Cause Description
58d) can have an effect on the processes that was unanticipated.
58b The line(s) were properly designed and set up for this
(these) product(s).
The line(s) were properly designed and set up for this (these) product(s) if
the line produces the product according to the requirements.
If the line design, the processes, or the materials have not changed, chances
are that the design of the line is okay. Changing either the processes (Cause
Description 58a) or the materials (Cause Description 58d) can have an effect
on production that was unanticipated. If the line design has changed, there has
likely been an effect on the product that was unanticipated.
58c Shop orders were correct and thorough.
Shop orders were correct and thorough when the end product produced is
the product that was specified by the customer.
Shop orders contain the need for an output product. Shop orders usually
refer back to the specification (specified or derived) that the end product must
meet. Obviously the specifications will be applicable to the end-product in measurement
terms such as size, weight, functionality, etc. Usually, shop orders
refer to the techniques or process, tools, raw products, etc., that must be used
to produce the end product.
Many products require that a number of piece parts be provided to create
the final end product. Shop orders control each step of the overall process to
ensure that the final end product is the product specified. This process is true
of either hardware or software.
Shop orders differ from work orders in that shop orders specify a product to
be provided (e.g., rod, software module, etc.) whereas work orders specify a
service to be provided (i.e., replace light bulb, etc.).
58d The materials were proper for the processes and the
product(s) and do meet the requirements.
The materials were proper for the product(s) when the product meets the
requirements. The materials must meet the requirements of the processes and/
or materials list, and they must be the materials that were specified in the subcontract
or Purchase Order.
59 UNIT TEST
59a Each Unit3 Test correctly reflects the requirement.
Each Unit Test correctly reflects the requirement when each element of the
Unit Test is directly traceable to each element of the unit requirement (specification)
through the Requirements Traceability Matrix (RTM) or Subcontract
Requirements Traceability Matrix (SRTM).
59b Each design element that applies to the routine/module/
subsystem has its own test case.
Each design element that applies to the routine/module/subsystem has its
own test case when there is a direct correlation between the requirement and the
elements tested in the unit, subsystem, or system test through the Requirements
Traceability Matrix (RTM) or Subcontract Requirements Traceability Matrix
(SRTM).
59c Unit Test findings were reviewed for completeness and
forwarded to be incorporated into Subsystem Tests and the
System Test.
Unit Test findings were reviewed for completeness and forwarded to be incorporated
into Subsystem Tests and the System Test when the Unit Tests are
traceable to the next level tests. When the Program Test Plan (PTP) is developed,
it must include ‘‘stacking’’ the qualification and acceptance tests from the
lowest level (Unit Tests) to the highest level (System Test). Care must be taken
in the PTP to ensure that all units or subsystems placed under subcontract as
well as units and subsystems you have developed adhere to this philosophy.
This technique is sometimes called the ‘‘Test Flow Forward Technique.’’
When developing the PTP and the test requirements for the subcontracts,
the Requirements Traceability Matrix (RTM) must be used to account for the
requirements being ‘‘flowed down’’ into the PTP. The PTP must accommodate
the ‘‘flow-down’’ and then, in turn, account for the ‘‘flow-up’’ of the product
responses. The PTP should allow for building the system from the bottom up
through the use of the test flow.
59d All Problem Test Reports (PTRs) were captured,
dispositioned (allocated for action), and worked off.
All Problem Test Reports (PTRs) were captured, dispositioned, and worked
off when there is complete accountability for every error that occurred during
test conduct and every error was captured, dispositioned, corrected, and verified.
The System Test, as written, must subsequently run without error.
60 SYSTEM TEST
60a The System Test Plan/Procedure was approved by the
customer.
The System Test Plan/Procedure was approved by the customer when the
System Test Plan/Procedure has been provided to the customer with lead time
adequate for customer review, and the customer agrees with and approves the
final content. The review/approval cycle may require iteration before approval
is achieved.
60b The System Test is traceable to the requirements.
The System Test is traceable to the requirements when each requirement is
forward traceable through the unit and the subsystem to the system via the
Requirements Traceability Matrix (RTM). Each unit or subsystem requirement
must be tested at least once at the appropriate level (i.e., at the unit level or at
the subsystem level). System level requirements must be visible and backward
traceable to the requirement through the RTM. The exceptions to this statement
are those requirements that are only visible at the system level. Your RTM
should reflect this particular situation by showing the requirement in the leftmost
column and the place where it is tested in the System Test. All columns
in-between will have no entries or dashes.
60c The System Test tested all elements of the system
concurrently.
System Test tested all elements of the system concurrently when all elements
of the system are called into play as they will be whenever the system is operating
in its normal mode.
60d The System Test was performed under appropriate load(s).4
The System Test was performed under appropriate load(s) when the loads
on the system are the loads required by the specification.
60e The System Test was performed using the same kind of
personnel that will be used by the customer.
The System Test was performed using the same kind of personnel that will
be used by the customer with regard to training, education, experience, etc. To
conduct with engineers a system test that was designed to be run by Level 1
technicians is an invalid test even when you follow all the other procedures of
the test. Many times the customer will stipulate that the actual users must perform
the System Test. In that case, there is no question. When you are planning
your project, ensure you have set aside time to train or expose these persons to
the system and its demands.
60f The System Test was properly documented and
incorporated the test results of all prior-level tests.
The System Test was properly documented and incorporated the test results
of all prior-level tests when the results of the Unit Level Tests and the Subsystem
Level Tests are clearly visible in the construct and conduct of the System Test.
The System Test should be an incremental test that relies upon the successful
completion of all Unit and Subsystem Tests operating together under appropriate
system loads.
61 CONFIGURATION MANAGEMENT
61a The Configuration Management Plan (CMP) is thorough,
complete, and authorized.
The Configuration Management Plan (CMP) is thorough, complete, and authorized
when it follows the format required by the customer and/or the enterprise
and maintains the required content as specified in customer or enterprise
configuration management policy. Further, the CMP must be signed by an authority
that is authorized to sign such documents (usually the vice president or
director of engineering, or an equivalent position).
The Configuration Management process could easily be looked upon as the
TEAMFLY
Janus process. Remember the Roman god Janus who looked both backward
and forward? That’s what the Configuration Management process does. It looks
backward to the baseline, as established, and forward to the test process that
will prove the viability of a change.
61b Change requests were presented and approved by an
appropriate level of the Review Board.
Change requests were presented and approved by an appropriate level of the
Review Board when the presentations and approvals followed the Configuration
Management Plan (CMP). See Attachment 5 for more information.
61c Version controls are in place and are reflected on (in) the
product.
Version controls are in place and are reflected on (in) the product when the
affected product is appropriately marked with the version that describes it in
the Version Description Document (VDD) (see glossary) and to which it has
been tested.
62 SYSTEM EFFECTIVENESS FACTORS
62a All required System Effectiveness Factors5 have been
appropriately considered.
To say that all required System Effectiveness Factors have been appropriately
considered, means that all the System Effectiveness Factors have been appropriately
considered in both the product and the processes.
Notes
1. Systems Application Architecture—Common User Access Guide to User Interface Design.
IBM Corporation, 1991 IBM Document Number SC34-4289 (available through
IBM field offices).
2. The Windows Interface Guidelines for Software Design. Redmond, Wash.: Microsoft
Press, 1995.
3. Defined as the smallest stand-alone component that produces a definable output
from a definable input. The unit may be hardware or software. In the case of hardware,
power can be external (i.e., a separate unit).
4. Loads are stresses placed upon a system. Loads are those stresses in units typical
for the product such as pounds, watts, ergs, numbers of subsystems, etc.
5. The System Effectiveness Factors refer to Reliability, Availability, Maintainability,
Supportability (including Logistics), Susceptability, Producibility, Human Engineering,
Safety, and Security.