A Starter Kit for Measuring Program Implementation

About the author: Tiffany Berry, PhD. is a research associate professor in the Division of Behavioral & Organizational Sciences. She is a core faculty member in two concentrations: Positive Developmental Psychology and Evaluation and Applied Research Methods. She regularly teaches and provides research supervision to master’s and doctoral students in both concentrations. In addition, she is an active educational program evaluator and associate director at the Claremont Evaluation Center (CEC) at CGU.

As a practicing educational evaluator for the past 18 years, I have examined the implementation of more than 100 different programs; this translates to thousands of different teachers, coaches, and staff trying to educate even more students with a range of individual abilities, personalities, and motivations. Further, it means I’ve worked with program directors/administrators with 100 different ideas about why their program is going to be particularly effective. All these discrete, yet interconnected moving parts makes measuring classroom implementation so complex (and also fun)! The purpose of this post is to share a few strategies I have used to embrace this complexity.

These discrete, yet interconnected moving parts make measuring classroom implementation so complex (and also fun)!

STEP 1: FIND AN IMPLEMENTATION FRAMEWORK RELEVANT TO THE PROGRAM.

Throughout my career, I have been exposed to several different implementation frameworks; however, I think Jeanne Century’s framework is the best. It is a wonderful, empirically-based implementation framework that can be flexibly applied to a range of educational programs under study. It identifies specific pedagogical and instructional dimensions of the implementation context. I have used this framework to evaluate a college readiness program as well as an English language arts curriculum for high school students.

STEP 2: WORK WITH STAKEHOLDERS TO UNDERSTAND THE CONTEXT OF THEIR PROGRAM.

Our team regularly collaborates with program stakeholders to identify the theory (or logic) of the program under study.  We discuss what strategies are critical for success, the essential ingredients for improving student outcomes, and, if necessary, what type of professional development is needed to implement the program correctly. This stakeholder-driven knowledge should be meaningfully must be incorporated into your measurement of program implementation if you want (1) stakeholders to buy-into the evaluation, (2) to collect data that will be relevant and useful to the stakeholders, and (3) to capture the essential ingredients that may help explain student learning outcomes.

Stakeholder knowledge should be meaningfully incorporated into your measurement of program implementation if you want stakeholders to buy-into the evaluation.

STEP 3:  ADAPT OR REVISE YOUR MEASUREMENT PROTOCOLS

You should adapt or revise your measurement protocols using stakeholder program theory, implementation frameworks, and existing educational research. This is no small feat and is probably the hardest part of the whole process. What specific implementation constructs are you going to measure? How are you going to measure it? Several methods are possible, from teacher or student surveys, to observations, to student interviews/focus groups. The choice of the method is dictated by several factors, including the budget, the timeline, or existing available measures.

STEP 4: PILOT TEST YOUR ADAPTED MEASURES FOR RELIABILITY AND VALIDITY

Although your implementation measures may be stakeholder-approved, they also need to be scientifically valid and reliable. Determine the psychometric properties of your scales, and share this information widely; consider posting the scale, along with its’ psychometric properties to the STEM repository. This will elevate the practice of evaluation and will accelerate our collective understanding around measuring classroom implementation.

A note from CSONIC: How have you incorporated implementation measurement in your own practice? Do you have any tips of your own? Let us know in the comment section or join the discussion on our forums!

One thought

  1. I like Century’s framework but find that I need to re-label her categories. Here’s my re-labeling:
    Structural Procedural: What teachers DO
    Structural Educative: What teachers KNOW
    Instructional Pedagogical: How teachers BEHAVE and INTERACT with students
    Instructional Student Engagement: How students ENGAGE

    I’m open to ideas. I find that I often have to translate these for PIs and project staff.