Adding Systematic Measures of Classroom Implementation to a CS Education Theory of Change

Here, we present a theory of change that accounts for classroom implementation in a typical computer science education professional learning program.  We begin with the following theory of change which connects program activities to changes in students’ psycho-social characteristics which leads to desired outcomes:

Picture1

Figure 1. Typical computer science education theory of change

Oftentimes, we build evaluation plans around a relatively simple theory like Figure 1: program activities (usually 20 or more hours of face-to-face professional learning accompanied by a curriculum and teaching materials) are implemented in computer science classrooms in the hopes of altering student psycho-social characteristics (like motivation to succeed and belongingness) which lead to students’ demonstrations of improved content knowledge and greater intention to persist in computing education and careers.  This theory of change has primarily grown out of years of focus on student diversity in computer science.  Researchers and evaluators have correctly identified characteristics that prevent minority students, female students, and students with disabilities from participating in computer science.

We have not ignored implementation, we simply have not given it a place in the theory of change.

This focus has drawn our attention to student variables.  Further, our measures allow us to draw quantitative connections among these variables.  We have not ignored classroom implementation.  Instead, we often measure classroom implementation qualitatively, through teacher interviews and focus groups, and use those descriptions to help explain the quantitative connections among variables in the theory of change.  We have not ignored implementation, we simply have not given it a place in the theory of change.

While we have not fully explored student-level psycho-social characteristics that promote participation in computer science education, we have accounted for it enough to expand our view.  We recommend systematically accounting for classroom implementation using any of a number of classroom implementation frameworks.  Figure 2 revises our theory of change to systematically account for classroom implementation.

Picture2Figure 2.  Theory of change that accounts for classroom implementation

Adding classroom implementation affects the overall theory of change in important ways.  First, we have added the elements of a classroom implementation framework; specifically, we use the four elements of Century’s framework (structural procedural, structural educative, instructional pedagogical, and instructional student engagement).  However, adding these elements has also led to the inclusion of context and program sustainability.  Accounting for context is a part of most evaluation approaches, and it becomes critical in this model because we need to understand whether the program activities or school/teacher context accounts for changes in the four classroom implementation elements. Century warns that ‘the wily line that distinguishes innovation from context can be hard to pin down’ (Century & Cassata, 2016).  Ultimately, we would want to measure the connection between program activities and classroom implementation accounting for contextual variables.

Stretching the theory of change beyond the immediate classroom also requires that we acknowledge the effects of the program on students after the program has ended.

We also add program sustainability which is teachers’ continued use of program elements such as professional learning, curriculum, and accompanying materials, and we acknowledge that teachers may use these after the project has ended.  Stretching the theory of change beyond the immediate classroom also requires that we acknowledge the effects of the program on students after the program has ended, that students may persist into future education and may make computing a part of their career endeavors.  We also acknowledge that teachers’ changes to instruction may also affect future classrooms of students.  These elements may fall outside of measurable scope of the program’s evaluation, but they reflect the goals of many funders.

Century’s four aspects of classroom implementation require a more detailed description.  Structural procedural refers to the actions a teacher needs to take to enact professional learning.  Structural educative elements are those that encompass what a teacher needs to know in order to enact professional learning; most CS professional learning seeks to increase teachers’ content knowledge of CS.  Instructional pedagogical elements account for how a teacher behaves and interacts with students.  Some would call this ‘teacher moves,’ and in CS education this could refer to the ways in which a teacher facilitates student autonomy and risk taking.  The Instructional Student Engagement element refers to students’ actions and interactions.  For example, in a CS Unplugged intervention, an observer would expect to see students out of their seats and standing on a large network diagram drawn on the floor and stepping through the diagram.

A note from CSONIC: Once you register, please leave comments using the prompt at the bottom of this page.

Have you been able to describe implementation in a theory of change for any of your own evaluations? Why or why not?

One thought