Subjective Measures
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67
68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84
85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101
102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118
119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135
136 137 138 139 140 141 142 143 144 145 146
Subjective evaluation of the experimental outcomes was conducted three months after
the comparative experiment under the guidance of an assistant. The rating process was
supported by an evaluative questionnaire (available from the authors). The questionnaire
consisted of eight maps or figures (six maps from the outcome of the comparative
experiment, one map of the management team, and one synthesized map). Subjects were
not told which maps were which. The questionnaire was designed based on a multiattribute
value (MAV) model (Massey and O’Keefe, 1993; Massey and Wallace, 1996;
Sakman, 1985) with three attributes or criteria: problem representation (C1), solution
implication (C2), and stakeholder implication (C3). Subjects were asked to evaluate the
figures in two steps. In step one, they identified the information in each map that was
critical. In step two, they rated the map on the three criteria. The first step provided the
basis for the second step. For example, for problem representation, the raters were asked
to mark elements (factors and relations between factors) with þ (or 0) symbols if the raters
agreed (disagreed) that these elements represented the problem situation. For solution
implication, they were asked to mark relationships that had important implications for
solving the problem. For stakeholder implication, subjects were asked to indicate how
well various groups of stakeholders and their needs, interests, and power5 were incorporated
in the map. They rank ordered these groups in terms of which were represented
the best, second best, and so forth.
All HALONG subjects were contacted and asked to evaluate the maps. Half of them
completed the questionnaire. To obtain a more robust evaluation of whether the maps
represented the situation at HALONG, three senior management personnel who had not
participated in building the maps were also recruited to evaluate them. In addition to
HALONG personnel, 30 M.B.A. students who had been in management positions and ten
lecturers in a Vietnamese school of management were recruited to rate the maps. Twentyeight
usable questionnaires (21 from M.B.A. students, seven from lecturers) were
obtained from this sample.
The measures in this study were three attributes of the group map that were defined
above: problem representation (C1), solution implication (C2), and stakeholder implication
(C3). The attributes were measured on a 0-10 scale, in which 0 indicates “strongly
disagree” and 10 indicates “strongly agree.”
As illustrated in Table 9, the comparison of the results is organized in the following
manner. For each method (M1, M2, M3), we compared the two groups of raters (HALONG
raters, indicated by G1, versus non-HALONG raters, indicated by G2) in terms of the three
criteria (C1, C2, C3) to determine whether independent evaluations (non-HALONG raters)
were significantly different from participant evaluations (HALONG raters). Originally we
had planned to contrast HALONG participants’ perceptions with the perceptions of the
three non-participating HALONG managers, but there were no significant differences
between these two sets of ratings, so they were combined for this analysis. We also
compared the pairs of group maps derived with the same method on the three criteria to
determine whether they received significantly different evaluations. Finally, we compared
the three methods over the three criteria for all raters combined to determine the
relative performance of the methods.
Comparison of means for the two aggregate collective maps constructed by groups A
and B indicated that the two maps differed (see Figure 1). Although the maps for groups
A and B were derived using the same method, B received significantly higher ratings than
A for problem representation (t = -2.24, df = 84, p = .02, 2 tailed). The differences are near
significant in terms of solution implication (t = -1.85, df = 83, p = .068) and stakeholder
5
5.5
6
6.5
7
7.5
C1 C2 C3
G1-A G1-B G2-A G2-B
Figure 1. Evaluation of the aggregate method over three criteria
Notes: G1 stands for the HALONG rater group; G2 for the non-HALONG rater group. A and B
are the two group maps constructed in the experiment using the aggregate method. C1 is the
average ratings of the groups for problem representation criterion, C2 for solution implication
criterion, and C3 for stakeholder implication criterion. So, G1-A represents the ratings given by
HALONG raters (G1) to the first aggregate group map (A). G1-B represents the ratings of the
second aggregate group map (B) by the HALONG raters (G1). G2-A represents the ratings of
the first aggregate group map (A) by the non-HALONG raters (G2). G2-B represents the ratings
of the second aggregate group map (B) by the non-HALONG raters (G2).
Table 9. A plan for comparison of results of mapping across the groups
Treatments M1 (Aggregate) M2 (Congregate) M3 (Workshop)
Group A B C D E F
Raters G1 (HL) / G2 (non-HL) G1 (HL)/ G2 (non-HL) G1 (HL)/ G2 (non-HL)
implication (t = -1.891, df = 83, p = .062). No significant difference was found in the
evaluations between the HALONG and non-HALONG raters for C1 and C2 (p>.1).
HALONG raters gave slightly higher ratings to C3 than independent raters with a 10%
level of significance (t = 1.777, df = 83, p = .07).
Figure 2. Evaluation of the congregate method over three criteria
5
5.5
6
6.5
7
C1 C2 C3
G1-C G1-D G2-C G2-D
Notes: G1 stands for HALONG raters; G2 for non-HALONG raters; C and D are the two group
maps constructed in the experiment using the congregate method. C1 stands for problem
representation criterion, C2 for solution implication criterion, and C3 for stakeholder implication
criterion. See Figure 1 note for further explanation.
Figure 3. Evaluation of the workshop mapping method over three criteria
Notes: G1 stands for HALONG raters; G2 for non-HALONG raters; E and F are the two group
maps constructed in the experiment using the workshop method. C1 stands for problem
representation criterion, C2 for solution implication criterion, and C3 for stakeholder implication
criterion. See Figure 1 note for further explanation.
5
5.5
6
6.5
7
7.5
C1 C2 C3
G1-E G1-F G2-E G2-F
Comparison of means for groups C and D for the congregate mapping method (see Figure
2) indicated no significant differences between groups. The difference was near significance
for problem representation (t = -1.956, df = 84, p = .054, 2 tailed), but it was not
significant for either solution implication (t = -1.754, df = 84, p = .083) or stakeholder
implication (t = -1.612, df = 84, p = .111). Although the figure suggests a tendency for
HALONG subjects to rate outcomes higher than non-HALONG subjects, the difference
was not statistically significant (p > .10).
The result of the means comparison between the maps of groups E and F, which utilized
the workshop mapping method, indicated significant differences between the two maps
(see Figure 3) for the non-HALONG raters. While the HALONG group rated the maps for
groups E and F as approximately equal on the three criteria, the non-HALONG subjects
rated group F’s map as better than group E’s on all three criteria (t = 2.44, df = 84, p = .017
for problem representation; t = 2.02, df = 84, p = .046 for solution implication; and t = 2.30,
df = 84, p = .024 for stakeholder implication).
The effectiveness ratings for the three methods of constructing maps for all raters
combined are shown in Figure 4. To test for differences in effectiveness between the three
methods we conducted one way ANOVAs with methods as the factor for each of the three
dependent variables (C1, C2, and C3). The ANOVAs revealed a significant main effect
for the method factor for solution implication (C2: F = 4.123, df = 2/254, p = .017). A
marginally significant main effect was also found for stakeholder implication (C3: F =
2.263, df = 2/254, p = .106). No significant effect was found for problem representation
(C1). Post-hoc tests revealed that the aggregate mapping method was rated as superior
to the other two methods on solution implication (p < .015) and to the congregate method
in terms of stakeholder implication (p < .03)
Figure 4. Comparison of the three methods over three criteria
5.4
5.6
5.8
6
6.2
6.4
6.6
6.8
7
C1 C2 C3
M1 M2 M3
Notes: M1 stands for the ratings averaged across all raters for both maps constructed with the
aggregate method, M2 the ratings averaged across all raters for both maps constructed with the
congregate method, and M3 the ratings averaged across all raters for both maps constructed with
the workshop method. C1 stands for problem representation criterion, C2 for solution implication
criterion, and C3 for stakeholder implication criterion