Reliability
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67
68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84
85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101
102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118
119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135
136 137 138 139 140 141 142 143 144 145 146
To establish the reliability of the identification procedure, interview texts are coded by
multiple researchers/raters. The raters are deemed qualified to identify causal statements
if they have a familiarity with the technique and the domain under study. If the sample
is small, then complete sampling should be conducted. As the total number of pages of
transcripts increases, it becomes impossible for each rater to code each text. There are
usually two rounds of coding that cover a sample of the texts (5 - 10%). This subset of
the texts should be chosen at random. Comparisons are made for agreement and
disagreement between the researchers. Where disagreement occurred the discrepancies
are resolved through discussion. The reliability between the researchers is calculated by
measuring the level of agreement on the identification of causal statements and linkages.
The level of agreement between the researchers should be at least 0.75, to have an
acceptable level of reliability. For example, in her study of teaching methods, Nadkarni
(2003) reported Kendall’s coefficient of concordance (Siegel, 1956) to be 0.75 and argued
this was an acceptable level of reliability. A reliability less than 0.75 indicates that the
procedure is not robust enough for research purposes, and a modified identification
procedure will need to be developed.
Step 2: Construct Raw Causal Maps
In the second step, the causal statements identified in the first step are then separated
into “causes” and “effects” to construct the “raw causal maps.” See Table 4 for sample
causal statements.
A raw causal map is a map constructed using the language of the participants (See
Figure 3).
Step 3: Develop Coding Scheme
In CM research developing a coding scheme is important for several reasons, which
include: avoiding misclassification, interpretation and theory building. Carley and
Palmquist (1992) argue that aggregating actual raw phrases in the text into generalized
concepts can be used to move the coded text beyond explicitly articulated ideas to implied
1 Note: When the keyword “because” is in the sentence the cause comes after the keyword. Refer
to the sentences on page 28.
Cause Link Effect
You think of everything as an object Because1 Object oriented development is easy
I've got this object built up If then I go back and actually try to write some of
the methods
Once I have all of the information I need I think about What are the objects that will be needed
or tacit ideas and to avoid misclassification of concepts due to peculiar wording on the
part of individuals. In terms of interpretation, the coding scheme provides a mechanism
to reduce the cognitive load for both the researcher and the end user of the causal map.
For the researcher, a coding scheme is used to simplify the texts. Often the texts are
numerous pages in length and can be cumbersome to work with. By developing a coding
scheme, like terms can be combined and simplified into a standard format. This aids
analysis and interpretation of the maps. For the end user, the readability of the maps is
much improved when a word or short phrase can be substituted for a sentence. Again,
this provides consistency and clarity for the end user. From a theory building perspective
a coding scheme aids understanding of how the concepts (constructs) fit together into
a cohesive unit.
The steps involved in developing a coding scheme are dependent on the research context
of the study. Two different approaches have been employed to recast the content of
causal maps into a common scheme: benchmarking and theory-driven (Nadkarni &
Narayanan, in press). Each approach is described and associated with the appropriate
research context.
Benchmarking
With the discovery and evocative approaches, the relevant concepts are identified from
the participants’ statements (Nadkarni & Narayanan, in press; Nelson et al., 2000). This
process is referred to as benchmarking. In the benchmarking approach a list of ideal
concepts and links between concepts emerges from the causal maps of one or a group
of experts. This list is then used to compare the causal maps of other individuals. The
benchmarking approach has been widely used in expert-novice comparison studies (e.g.,
Hong & O’Neil, 1992). In these studies a causal map is developed based on the concepts
evoked from domain experts, with the expert map serving as the standard to which the
novice maps are compared. The benchmarking approach is useful in discovery and
evocative contexts and in particular studies linking causal maps to performance and
learning.
Figure 3. Raw causal map
Research Context Concepts Theory Guidance
Discovery Benchmarking from
participants
No
Evocative Benchmarking from
participants
Minimal
Hypothesis Testing From theory Yes
Intervention From theory Yes
Table 5. Coding scheme development
Theory-Driven
With the hypothesis testing and intervention approaches, the relevant concepts are
defined independent of, and prior to, coding from relevant literature. In the theory-driven
approach, the content in the individual causal maps is recast into theoretical categories
salient in the domain represented by the maps (e.g., Carley & Palmquist, 1992; Fahey &
Narayanan, 1989). In taking this approach, the researchers should first review the
relevant literature to determine if there are any theoretical classification schemes that
would be appropriate. If no single classification scheme is available, a composite
classification scheme encompassing the favorable aspects of the multiple schemes can
be used. Tying emergent categories to extant theory has been recommended to develop
standard categories (Carley & Palmquist, 1992) and build theory. See Table 5 for a
summary of the decision process.
The coding process begins with grouping frequently mentioned words in the statements.
A word or word group is created that captured the essence of the statement. For example,
the sentence fragment “You group the requirements document items based on functions”
could be labeled “Functions” or the fragment “bias on the part of management” could
be labeled “Management Bias.” Multiple researchers should review the statements and
independently place them into conceptual categories. Comparisons are made for agreement
and disagreement in the categorization of concepts. Where disagreement occurs
the discrepancies are resolved through discussion. The level of agreement between the
raters should be measured with the average no lower than 0.75. Once the conceptual level
scheme is developed, all of the statements are placed into the appropriate concept
category.
Once the concept-level coding is completed, a construct-level classification scheme can
then developed. Again, the benchmarking or theory-driven process should be used.
Table 6. Concept/construct level coding scheme
Raw Phrase Coded Concept Construct
You think of everything as an object Object Structure
Object-oriented development is easy OO Development Object-Oriented
Development Systems
I've got this object built up Object Structure
I go back and actually try to write some methods Method Behavior
Once I have all of the information I need OO Development Object-Oriented
Development Systems
What are the objects that will be needed Identifying Objects Object-Oriented Modeling /