Subjective Measures

Subjective evaluation of the experimental outcomes was conducted three months after

the comparative experiment under the guidance of an assistant. The rating process was

supported by an evaluative questionnaire (available from the authors). The questionnaire

consisted of eight maps or figures (six maps from the outcome of the comparative

experiment, one map of the management team, and one synthesized map). Subjects were

not told which maps were which. The questionnaire was designed based on a multiattribute

value (MAV) model (Massey and O’Keefe, 1993; Massey and Wallace, 1996;

Sakman, 1985) with three attributes or criteria: problem representation (C1), solution

implication (C2), and stakeholder implication (C3). Subjects were asked to evaluate the

figures in two steps. In step one, they identified the information in each map that was

critical. In step two, they rated the map on the three criteria. The first step provided the

basis for the second step. For example, for problem representation, the raters were asked

to mark elements (factors and relations between factors) with þ (or 0) symbols if the raters

agreed (disagreed) that these elements represented the problem situation. For solution

implication, they were asked to mark relationships that had important implications for

solving the problem. For stakeholder implication, subjects were asked to indicate how

well various groups of stakeholders and their needs, interests, and power5 were incorporated

in the map. They rank ordered these groups in terms of which were represented

the best, second best, and so forth.

All HALONG subjects were contacted and asked to evaluate the maps. Half of them

completed the questionnaire. To obtain a more robust evaluation of whether the maps

represented the situation at HALONG, three senior management personnel who had not

participated in building the maps were also recruited to evaluate them. In addition to

HALONG personnel, 30 M.B.A. students who had been in management positions and ten

lecturers in a Vietnamese school of management were recruited to rate the maps. Twentyeight

usable questionnaires (21 from M.B.A. students, seven from lecturers) were

obtained from this sample.

The measures in this study were three attributes of the group map that were defined

above: problem representation (C1), solution implication (C2), and stakeholder implication

(C3). The attributes were measured on a 0-10 scale, in which 0 indicates “strongly

disagree” and 10 indicates “strongly agree.”

As illustrated in Table 9, the comparison of the results is organized in the following

manner. For each method (M1, M2, M3), we compared the two groups of raters (HALONG

raters, indicated by G1, versus non-HALONG raters, indicated by G2) in terms of the three

criteria (C1, C2, C3) to determine whether independent evaluations (non-HALONG raters)

were significantly different from participant evaluations (HALONG raters). Originally we

had planned to contrast HALONG participants’ perceptions with the perceptions of the

three non-participating HALONG managers, but there were no significant differences

between these two sets of ratings, so they were combined for this analysis. We also

compared the pairs of group maps derived with the same method on the three criteria to

determine whether they received significantly different evaluations. Finally, we compared

the three methods over the three criteria for all raters combined to determine the

relative performance of the methods.

Comparison of means for the two aggregate collective maps constructed by groups A

and B indicated that the two maps differed (see Figure 1). Although the maps for groups

A and B were derived using the same method, B received significantly higher ratings than

A for problem representation (t = -2.24, df = 84, p = .02, 2 tailed). The differences are near

significant in terms of solution implication (t = -1.85, df = 83, p = .068) and stakeholder

5

5.5

6

6.5

7

7.5

C1 C2 C3

G1-A G1-B G2-A G2-B

Figure 1. Evaluation of the aggregate method over three criteria

Notes: G1 stands for the HALONG rater group; G2 for the non-HALONG rater group. A and B

are the two group maps constructed in the experiment using the aggregate method. C1 is the

average ratings of the groups for problem representation criterion, C2 for solution implication

criterion, and C3 for stakeholder implication criterion. So, G1-A represents the ratings given by

HALONG raters (G1) to the first aggregate group map (A). G1-B represents the ratings of the

second aggregate group map (B) by the HALONG raters (G1). G2-A represents the ratings of

the first aggregate group map (A) by the non-HALONG raters (G2). G2-B represents the ratings

of the second aggregate group map (B) by the non-HALONG raters (G2).

Table 9. A plan for comparison of results of mapping across the groups

Treatments M1 (Aggregate) M2 (Congregate) M3 (Workshop)

Group A B C D E F

Raters G1 (HL) / G2 (non-HL) G1 (HL)/ G2 (non-HL) G1 (HL)/ G2 (non-HL)

implication (t = -1.891, df = 83, p = .062). No significant difference was found in the

evaluations between the HALONG and non-HALONG raters for C1 and C2 (p>.1).

HALONG raters gave slightly higher ratings to C3 than independent raters with a 10%

level of significance (t = 1.777, df = 83, p = .07).

Figure 2. Evaluation of the congregate method over three criteria

5

5.5

6

6.5

7

C1 C2 C3

G1-C G1-D G2-C G2-D

Notes: G1 stands for HALONG raters; G2 for non-HALONG raters; C and D are the two group

maps constructed in the experiment using the congregate method. C1 stands for problem

representation criterion, C2 for solution implication criterion, and C3 for stakeholder implication

criterion. See Figure 1 note for further explanation.

Figure 3. Evaluation of the workshop mapping method over three criteria

Notes: G1 stands for HALONG raters; G2 for non-HALONG raters; E and F are the two group

maps constructed in the experiment using the workshop method. C1 stands for problem

representation criterion, C2 for solution implication criterion, and C3 for stakeholder implication

criterion. See Figure 1 note for further explanation.

5

5.5

6

6.5

7

7.5

C1 C2 C3

G1-E G1-F G2-E G2-F

Comparison of means for groups C and D for the congregate mapping method (see Figure

2) indicated no significant differences between groups. The difference was near significance

for problem representation (t = -1.956, df = 84, p = .054, 2 tailed), but it was not

significant for either solution implication (t = -1.754, df = 84, p = .083) or stakeholder

implication (t = -1.612, df = 84, p = .111). Although the figure suggests a tendency for

HALONG subjects to rate outcomes higher than non-HALONG subjects, the difference

was not statistically significant (p > .10).

The result of the means comparison between the maps of groups E and F, which utilized

the workshop mapping method, indicated significant differences between the two maps

(see Figure 3) for the non-HALONG raters. While the HALONG group rated the maps for

groups E and F as approximately equal on the three criteria, the non-HALONG subjects

rated group F’s map as better than group E’s on all three criteria (t = 2.44, df = 84, p = .017

for problem representation; t = 2.02, df = 84, p = .046 for solution implication; and t = 2.30,

df = 84, p = .024 for stakeholder implication).

The effectiveness ratings for the three methods of constructing maps for all raters

combined are shown in Figure 4. To test for differences in effectiveness between the three

methods we conducted one way ANOVAs with methods as the factor for each of the three

dependent variables (C1, C2, and C3). The ANOVAs revealed a significant main effect

for the method factor for solution implication (C2: F = 4.123, df = 2/254, p = .017). A

marginally significant main effect was also found for stakeholder implication (C3: F =

2.263, df = 2/254, p = .106). No significant effect was found for problem representation

(C1). Post-hoc tests revealed that the aggregate mapping method was rated as superior

to the other two methods on solution implication (p < .015) and to the congregate method

in terms of stakeholder implication (p < .03)

Figure 4. Comparison of the three methods over three criteria

5.4

5.6

5.8

6

6.2

6.4

6.6

6.8

7

C1 C2 C3

M1 M2 M3

Notes: M1 stands for the ratings averaged across all raters for both maps constructed with the

aggregate method, M2 the ratings averaged across all raters for both maps constructed with the

congregate method, and M3 the ratings averaged across all raters for both maps constructed with

the workshop method. C1 stands for problem representation criterion, C2 for solution implication

criterion, and C3 for stakeholder implication criterion