Reliability

To establish the reliability of the identification procedure, interview texts are coded by

multiple researchers/raters. The raters are deemed qualified to identify causal statements

if they have a familiarity with the technique and the domain under study. If the sample

is small, then complete sampling should be conducted. As the total number of pages of

transcripts increases, it becomes impossible for each rater to code each text. There are

usually two rounds of coding that cover a sample of the texts (5 - 10%). This subset of

the texts should be chosen at random. Comparisons are made for agreement and

disagreement between the researchers. Where disagreement occurred the discrepancies

are resolved through discussion. The reliability between the researchers is calculated by

measuring the level of agreement on the identification of causal statements and linkages.

The level of agreement between the researchers should be at least 0.75, to have an

acceptable level of reliability. For example, in her study of teaching methods, Nadkarni

(2003) reported Kendall’s coefficient of concordance (Siegel, 1956) to be 0.75 and argued

this was an acceptable level of reliability. A reliability less than 0.75 indicates that the

procedure is not robust enough for research purposes, and a modified identification

procedure will need to be developed.

Step 2: Construct Raw Causal Maps

In the second step, the causal statements identified in the first step are then separated

into “causes” and “effects” to construct the “raw causal maps.” See Table 4 for sample

causal statements.

A raw causal map is a map constructed using the language of the participants (See

Figure 3).

Step 3: Develop Coding Scheme

In CM research developing a coding scheme is important for several reasons, which

include: avoiding misclassification, interpretation and theory building. Carley and

Palmquist (1992) argue that aggregating actual raw phrases in the text into generalized

concepts can be used to move the coded text beyond explicitly articulated ideas to implied

1 Note: When the keyword “because” is in the sentence the cause comes after the keyword. Refer

to the sentences on page 28.

Cause Link Effect

You think of everything as an object Because1 Object oriented development is easy

I've got this object built up If then I go back and actually try to write some of

the methods

Once I have all of the information I need I think about What are the objects that will be needed

or tacit ideas and to avoid misclassification of concepts due to peculiar wording on the

part of individuals. In terms of interpretation, the coding scheme provides a mechanism

to reduce the cognitive load for both the researcher and the end user of the causal map.

For the researcher, a coding scheme is used to simplify the texts. Often the texts are

numerous pages in length and can be cumbersome to work with. By developing a coding

scheme, like terms can be combined and simplified into a standard format. This aids

analysis and interpretation of the maps. For the end user, the readability of the maps is

much improved when a word or short phrase can be substituted for a sentence. Again,

this provides consistency and clarity for the end user. From a theory building perspective

a coding scheme aids understanding of how the concepts (constructs) fit together into

a cohesive unit.

The steps involved in developing a coding scheme are dependent on the research context

of the study. Two different approaches have been employed to recast the content of

causal maps into a common scheme: benchmarking and theory-driven (Nadkarni &

Narayanan, in press). Each approach is described and associated with the appropriate

research context.

Benchmarking

With the discovery and evocative approaches, the relevant concepts are identified from

the participants’ statements (Nadkarni & Narayanan, in press; Nelson et al., 2000). This

process is referred to as benchmarking. In the benchmarking approach a list of ideal

concepts and links between concepts emerges from the causal maps of one or a group

of experts. This list is then used to compare the causal maps of other individuals. The

benchmarking approach has been widely used in expert-novice comparison studies (e.g.,

Hong & O’Neil, 1992). In these studies a causal map is developed based on the concepts

evoked from domain experts, with the expert map serving as the standard to which the

novice maps are compared. The benchmarking approach is useful in discovery and

evocative contexts and in particular studies linking causal maps to performance and

learning.

Figure 3. Raw causal map

Research Context Concepts Theory Guidance

Discovery Benchmarking from

participants

No

Evocative Benchmarking from

participants

Minimal

Hypothesis Testing From theory Yes

Intervention From theory Yes

Table 5. Coding scheme development

Theory-Driven

With the hypothesis testing and intervention approaches, the relevant concepts are

defined independent of, and prior to, coding from relevant literature. In the theory-driven

approach, the content in the individual causal maps is recast into theoretical categories

salient in the domain represented by the maps (e.g., Carley & Palmquist, 1992; Fahey &

Narayanan, 1989). In taking this approach, the researchers should first review the

relevant literature to determine if there are any theoretical classification schemes that

would be appropriate. If no single classification scheme is available, a composite

classification scheme encompassing the favorable aspects of the multiple schemes can

be used. Tying emergent categories to extant theory has been recommended to develop

standard categories (Carley & Palmquist, 1992) and build theory. See Table 5 for a

summary of the decision process.

The coding process begins with grouping frequently mentioned words in the statements.

A word or word group is created that captured the essence of the statement. For example,

the sentence fragment “You group the requirements document items based on functions”

could be labeled “Functions” or the fragment “bias on the part of management” could

be labeled “Management Bias.” Multiple researchers should review the statements and

independently place them into conceptual categories. Comparisons are made for agreement

and disagreement in the categorization of concepts. Where disagreement occurs

the discrepancies are resolved through discussion. The level of agreement between the

raters should be measured with the average no lower than 0.75. Once the conceptual level

scheme is developed, all of the statements are placed into the appropriate concept

category.

Once the concept-level coding is completed, a construct-level classification scheme can

then developed. Again, the benchmarking or theory-driven process should be used.

Table 6. Concept/construct level coding scheme

Raw Phrase Coded Concept Construct

You think of everything as an object Object Structure

Object-oriented development is easy OO Development Object-Oriented

Development Systems

I've got this object built up Object Structure

I go back and actually try to write some methods Method Behavior

Once I have all of the information I need OO Development Object-Oriented

Development Systems

What are the objects that will be needed Identifying Objects Object-Oriented Modeling /