К оглавлению
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 
34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 
51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 
68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 
85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 
102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 
119 120 121 122 123 

Every test run should have PTR forms available to capture any anomalies

that occur during the conduct of the test. Further, there must be a PTR log in

which to record the PTRs and account for each and every one. Usually, a sequence

number is assigned to a PTR preceded by a unique Alpha that relates to

the test. For example, the first PTR for the System Test could be numbered as


If you do not have a PTR system, consider using the information to create a

form for your own use:





Test Conductor

Test Title Run No.

Short Title of Problem

Description of Problem

Disposition: Responsibility

Scheduled Correction Date

Action Taken

Completed By


Accepted By


In addition to a PTR Form, you should have a PTR Log to collect the actions

of all the PTRs opened. The PTR Log should contain:




Short title of problem

Name of person to whom assigned

Date initiated

Date to be completed

Date actually completed

Date accepted

Name of person accepting the PTR closure


60a (NO) The System Test Plan/Procedure was not approved by the


The System Test Plan was not approved by the customer when the System

Test Plan/Procedure has not been provided to the customer with lead time adequate

for customer review or the customer does not agree with or approve the

final content or the customer returns the plan/procedure without review and/

or approval or the customer has not been previously apprised of the content of

the plan/procedure.

If the first time the customer sees the System Test Plan is when the customer

arrives for the System Test, you can plan on a lot of stoppages, a lot of explanations,

and possibly a disapproval of the entire System Test.

At this point, it should be clear that the System Test Plan should be finalized

and forwarded to the customer with adequate time for review. If the customer

does not review the plan or does not approve the plan, revise the schedule to

ensure that the plan has been approved by the customer. Proceeding into test

without customer approval of the Test Plan and Test Procedure is a sure way to

ensure failure. Believe it or not, there are some customers around who will

sandbag the process just to keep their options open. You cannot allow this to



The best solution to this problem is to not let it happen in the first place.

That is achieved by scheduling the completion of the System Test Plan and

System Test Procedure as a part of the Data Plan. An understanding and agreement

must be made that the documents will be reviewed and approved by the

customer within a certain time period (two weeks to one month are usual).

Your agreement and schedule can allow for iteration, if necessary, but the System

Test should not be scheduled until final approval of the System Test Plan

and the System Test Procedure are approved by the customer.

Your agreement with the customer should be such that nonapproval will

result in a project stoppage. Due diligence will determine who is at fault for

nonapproval and thus be the basis for compensation, if any.

If you did not get this approval early in the project and are now faced with

going into System Test without approval, it is my recommendation that you

stop and sit down with the customer and get approval before proceeding. Unless

you have mutual understanding of all elements of the test, many issues will be

unresolvable and must be run again and again. This will be extremely costly,

probably to you.

For an outline of a System Test Plan, see Data Item Description DI-ATTS-


60b (NO) The System Test is not traceable to the requirements.

The System Test is not traceable to the requirements when each requirement

is not forward traceable through the unit and the subsystem to the system via

the Requirement Traceability Matrix (RTM) or each requirement is not tested

at least once at the appropriate level (i.e., at the unit level or at the subsystem

level) or system level requirements are not visible and backward traceable to

the requirement through the RTM. The exceptions to this statement are those

requirements that are only visible at the system level. Your RTM should reflect

this situation by showing the requirement in the leftmost column and the place

where it is tested in the System Test. All columns in between will have no entries

or dashes.


At this point, you are in for a lot of work. It is not adequate to simply trace

a single requirement back through the RTM; each and every requirement must

be traced back through the RTM. The reason is that if a requirement is visible

at the system level but has not been accommodated at the subsystem or unit

level, there exists a potential point of failure buried within the system.

Return to Cause Description 60d and accomplish the tasks listed there. Then,

repeat the testing advocated in Cause Descriptions 60a, 60b, and 60c.

60c (NO) The System Test has not tested all elements of the system


The System Test has not tested all elements of the system concurrently when

all elements of the system are not called into play as they will be whenever the

system is operating in its normal mode.

The purpose of the System Test is to test the entire system together. This

process tests the interfaces and the loadings of the system. In many systems, it

is normal that all units or subsystems or modules are not operating at the same

time but rather are operating at some predetermined or commanded sequence.

This is the way the system should be tested, as if it were performing the tasks it

is required to perform. If the system is not tested in its operating mode, it is

not a system test at all but rather a series of subsystem or unit tests.

Generally, the customer will define the system test coverage and scenarios,

and you will design the Test Plan around these criteria. The best way to design

a System Test is to create and document the System Test as the system is being

designed. This is not necessarily a day-to-day activity but should certainly be

accomplished before each major review so that the design and the testing are

concurrent. Using a documented Configuration Management Process, the test

process should require revision of the test after each major revision and follow

the same level of review as the system itself.


If you are at this point and recovery is necessary, it is clear the System Test

was not created concurrent with the system. You are probably the recovery

project manager because the one that got the project to this point is somewhere


There are four steps that must be taken:

First, a lot of documentation must be reviewed. The original requirement

and all changes to the baseline must be reviewed. The baseline must be updated

(or, if you are really in trouble, created) to reflect the last documented baseline

of the system. The baseline would now be established, but it must be validated.

This is the tricky part. The only way to validate the current requirements base-

line is to negotiate it with your customer. From the customer’s standpoint, this

could be an opportunity to add in all those things they wanted but couldn’t

afford. From your standpoint, you must insist that the customer provide documentation

of the original baseline and each change afterward. Verbal changes

must not be accepted.

Second, the system must be physically and functionally baselined. This will

probably take a lot of time, but it must be done. Keep very close time records

of this activity so the time can be properly allocated during negotiation.

Third, you must negotiate the differences with your customer, and responsibility

must be assigned. If the customer has established and documented a valid

baseline with changes and you (your company) has accepted these changes or,

at least, not refused the changes, that’s what you must work to, no mater how

much it hurts. Any change to that statement must be a management decision

because there are all kinds of legal ramifications.

Fourth, when the physical and functional baseline has been established and

the requirements negotiated, they must be brought together. Usually, this requires

changing the system. At this point, you can write (rewrite) the System

Test with reasonable assurance it will be correct. Sometimes, at this point, there

are other changes the customer sees he wants. This time, keep up with the

changes in the System Test!

60d (NO) The System Test was not performed under appropriate


The System Test was not performed under appropriate load(s) when the

loads on the system are not the loads required by the specification.


If the specification does not stipulate loads, it is best to create ‘‘reasonable’’

loads, as determined by engineering analyses, and perform the system tests

under those loads. Those loads and that fact must be documented and presented

to the customer/client prior to final acceptance. The best time to present these

issues is in the first Design Review. If the customer/client has an issue with the

loads, it is a point for negotiation.

60e (NO) The System Test was not performed using the same kind of

personnel that will be used by the customer.

The System Test was not performed using the same kind of personnel that

will be used by the customer with regard to training, education, experience, etc.,

or the personnel specified by the customer. To conduct a System Test with

engineers instead of Level X technicians is an invalid test even when you follow

all the procedures in the test. If the specification does not stipulate operating

personnel level, it is best to assume reasonable operating levels, as determined

by engineering analyses, and perform the system tests using those personnel.

Those operating levels must be documented and presented to the customer/

client prior to final acceptance. The best time to present these issues is in the

first Design Review. If the customer/client has an issue with the created loads,

that is a point for negotiation.

60f (NO) The System Test was not properly documented and did not

incorporate the test results of all prior-level tests.

The System Test was not properly documented and did not incorporate the

test results of all prior level tests when the results of the unit level tests and the

subsystem level tests are not clearly visible in the construct and conduct of the

system test.