Observational Measurement of Behavior Book

List of Figures xiii

List of Tables xv

Foreword xvii

Preface xix

Acknowledgments xxiii

1 Introduction and Measurement Contexts 1

Overview 1

Systematic Observation 2

Count Coding Systems 3

Importance of Falsifiable Research Questions or Hypotheses 4

Behavior as "Behavior" Versus Behavior as a Sign or Indicant of a Construct 4

Two Interpretations of Operationalism 5

Distinction Between Context-Dependent Behavior and Generalized Tendencies to Behave 7

Rationale for Identifying How We Are Conceptualizing Our Object of Measurement 8

Influential Variables of a Measurement Context 9

Structuredness 9

Ecological Validity 10

Representativeness 10

Tension Between Structuredness and Ecological Validity 11

Recommendations for Measuring Generalized Characteristics From Observations 12

Potential Disadvantages of Systematic Observational Count Measurement 13

Recommendations 14

References 15

2 Improving Measurement of Generalized Characteristics Through Direct Observation and Generalizability Theory 17

Overview 17

Two Concepts of Measurement 17

Generalizability Theory as a Measurement Theory for Vaganotic Measures 19

Example: Generalizability (G) Study With Multiple Sessions as a Single Facet 20

Consequences of a Low G Coefficient 23

Decision Studies 24

McWilliam and Ware as an Example of a Two-Faceted Decision Study 25

Practice Using a G Calculator on Data From a Two-Faceted G and D Study 26

Accuracy of D Study Projections 30

Implications of the Lessons of G and D Studies for Single-Subject Research 31

A Dilemma 32

Recommendations 33

References 33

3 Designing or Adapting Coding Manuals 35

Overview 35

Selecting, Adapting, or Creating a Coding Manual 36

Definition of a Coding Manual 36

Relation of the Coding Manual to the Research Questions and Prediction 36

Recommended Steps for Modifying or Designing Coding Manuals 37

Conceptually Defining the Context-Dependent Behavior or the Generalized Characteristic 37

Deciding the Level of Detail at Which the Behaviors Should Be Distinguished 38

Physically Based Definitions, Socially Based Definitions, or Both? 39

Defining the Lowest Level Categories 40

Sources of Conceptual and Operational Definitions 42

Defining Segmenting Rules 46

Defining When to Start and Stop Coding 47

The Potential Value of Flowcharts 48

Do Coding Manuals Need to Be Sufficiently Short to Be Included in Methods Sections? 49

Recommendations 49

References 51

4 Sampling Methods 53

Overview 53

The Elements of a Measurement System 53

Behavior Sampling 54

Continuous Behavior Sampling 54

Intermittent Behavior Sampling 55

Interval Sampling 56

How Does Interval Sampling Estimate Number and Duration? 58

Participant Sampling 59

Focal Sampling 59

Multiple Pass Sampling 60

Conspicuous Sampling 60

Reactivity 60

Live Coding Versus Recording the Observation for Later Coding 62

Recording Coding Decisions 64

Practice Recording Session 66

Recommendations 69

References 70

5 Common Metrics of Observational Variables 73

Overview 73

Definition of Metric 74

Quantifiable Dimensions of Behavior 74

Proportion Metrics 75

Proportion Metrics Change the Meaning of Observational Variables 75

Scrutinizing Proportions 77

An Implicit Assumption of Proportion Metrics 78

Testing Whether the Data Fit the Assumption of Proportion Metrics 79

Consequences of Using a Proportion When the Data Do Not Fit the Assumption 80

Alternative Methods to Control Nuisance Variables 85

Statistical Control 85

Procedural Control 85

Transforming Metrics of Observational Variables in Group Statistical Analyses 86

Scales of Measurement for Observational Variables 88

Observational Variables in Parametric Analyses 90

Recommendations 90

References 91

6 Introduction to Sequential Analysis 93

Overview 93

Definitions of Terms Used in This Chapter 94

Sequential Versus Nonsequential Variables 94

Sequential Associations Are Not Sufficient Evidence for Causal Inferences 95

Coded Units and Exhaustiveness 96

Three Major Types of Sequential Analysis 98

Event-Lag Sequential Analysis 98

Time-Lag Sequential Analysis 99

Time-Window Sequential Analysis 100

The Need to "Control for Chance" 101

How Sequential Data Are Represented Prior to Contingency Table Organization 102

Contingency Tables 103

Proper 2 × 2 Contingency Table Construction of Two Streams of Data for Concurrent Analysis 105

Proper 2 × 2 Contingency Table Construction From One Stream of Data for Event-Lag Sequential Analysis 105

Simulation Study to Compare Results From Two Ways to Construct Contingency Tables 108

Contingency Tables for Time-Window Lag Sequential Analysis 109

Transitional Probability 111

Transitional Probabilities in Backward Sequential Analysis 113

Summary of Transitional Probabilities 115

Recommendations 116

References 116

7 Analyzing Research Questions Involving Sequential Associations 119

Overview 119

Computer Software to Aid Sequential Analysis 120

Practice Exercise Using MOOSES Software to Conduct Time-Window Analysis 120

Yule's Q 125

What Is "Enough Data" and How Do We Attain It? 126

Proposed Solutions for Insufficient Data 129

Sequential Association Indices as Dependent Variables in Group Designs 131

Testing the Significance of a Mean Sequential Association 131

Testing the Between-Group Difference in Mean Sequential Associations 132

Testing the Within-Subject Difference in Sequential Associations 132

Testing the Significance of the Summary-Level Association Between a Participant Characteristic and a Sequential Association Between Behaviors 133

Statistical Significance Testing of Sequential Associations in Single Cases 133

A Caveat Regarding the Use of Yule's Q 136

Recommendations 137

References 138

8 Observer Training, Observer Drift Checks, and Discrepancy Discussions 141

Overview 141

Three Purposes of Point-by-Point Agreement on Coding Decisions 141

Two Definitions of Agreement 142

Agreement Matrices 145

Discrepancy Discussions 148

Criterion Coding Standards 149

Observer Training 151

Method of Selecting and Conducting Agreement Checks 153

Retraining When Observer Drift Is Identified 155

Recommendations 156

References 156

9 Interobserver Agreement and Reliability of Observational Variables 159

Overview 159

Additional Purposes of Point-by-Point Agreement 159

Added Principles When Agreement Checks Are Used to Estimate Interobserver "Reliability" of Observational Variable Scores 160

Exhaustive Coding Spaces Revisited 164

The Effect of Chance on Agreement 167

Common Indices of Point-by-Point Agreement 168

Occurrence Percentage Agreement 168

Nonoccurrence Percentage Agreement 168

Total Percentage Agreement 169

Kappa 169

Base Rate and Chance Agreement Revisited 171

Summary of Point-by-Point Agreement Indices 172

Intraclass Correlation Coefficient as an Index of Interobserver Reliability from the Vaganotic Concept of Measurement 174

Options for Running ICC With SPSS 175

Between-Participant Variance on the Variable of Interest Affects ICC 175

Using ICC as a Measure of Interobserver Reliability for Predictors and Dependent Variables in Group Designs 177

The Interpretation of SPSS Output for ICC 177

The Conceptual Relation Between Interobserver Agreement and ICC 178

Consequences of Low or Unknown Interobserver Reliability 178

Recommendations 180

References 181

10 Validation of Observational Variables 183

Overview 183

The Changing Concept of Validation 184

Understanding Which Types of Validation Evidence Are Most Relevant for Different Research Designs, Objects of Measurement, and Research Purposes 185

Content Validation 186

Definition of Content Validation 186

Different Traditions Vary on the Levels of Importance Placed on Content Validation 187

Weaknesses of Content Validation 188

Sensitivity to Change 188

Definition of Sensitivity to Change 188

Influences on Sensitivity to Change 189

Weaknesses of Sensitivity to Change 190

Treatment Utility 190

Definition of Treatment Utility 190

Weaknesses of Treatment Utility 192

Criterion-Related Validation 193

Definition of Criterion-Related Validation 193

Primary Appeal of Criterion-Related Validation 193

Weaknesses of Criterion-Related Validation 194

Construct Validation 194

Definition of Construct Validation 194

Discriminative Validation 195

Nomological Validation 196

Multitrait, Multimethod Validation 197

An Implicit "Weakness" of Science? 200

Recommendations 202

References 202

Glossary 205

Index 221