Program Overview: Program Evaluation8

The evaluation of the Civics Mosaic Program is multi-phased and uses a mixed methodology approach, collecting both quantitative and qualitative data to describe and report the development of the program, the implementation of program activities, and the outcomes for program participants. Phase 1 and Phase 2 evaluation activities have, over the program's first two years (October 2002 through September 2004), provided information to the program staff documenting the collaboration process, the initial outcomes resulting from professional development activities, and areas in which participants needed additional resources. Surveys, interviews and direct observations have been conducted both in the 12 regions of the United States and in the 12 regions in Russia. As of 2004, there are roughly 50 fellows involved in the program in each country, about 85 percent of those participants are classroom teachers. Formative feedback has assisted program staff to focus the professional development activities (e.g., exchange visits and summer seminars) and to concentrate program efforts to develop classroom materials to broaden the teaching of comparative civics. At this time, the development of the comparative textbook is primarily for use in high school classrooms; however middle school teachers have identified benefits for adapting these lessons for middle school social studies classes, as well.

Phase 3 of the evaluation, beginning in the third year of the program, starting October 1, 2004, is designed to address program impact by measuring specific teacher and student outcomes. The performance measures provided by the funding agency (the U.S. Department of Education) are to demonstrate improved instructional effectiveness (by teachers) and improved cognitive outcomes (for students). The initial step in the impact evaluation phase of evaluation is to pilot our textbook lessons in high school classrooms in Civics Mosaic regions in the United States. A comparative study, using a quasi-experimental design, is being conducted in the 2004-2005 school year in six of the Civics Mosaic regions to pilot the textbook.

This pilot study will include a total of 16 teachers representing 24 high school classrooms and approximately 500 students. The 8 teachers in the experimental group will be using some of our textbook lessons in their social studies classes; and the 8 comparison group teachers will use 'traditional' curriculum in each of the paired classrooms. Data are being collected to match/compare both the teachers and the students in their classrooms (collectively) so that differences in cognitive and/or affective measures for students can be attributed to the instructional differences from using the comparative civics lessons.

The results of this year's pilot will be used to inform and revise a larger comparative study in Year 4 of the program, beginning October 1, 2005. More comparative lessons will be field-tested in the subsequent year of the study. In addition, more teachers and classrooms will be recruited and a pre-test, post-test design will be implemented to enhance the potential for finding meaningful differences from instruction.

Phase 3 of the evaluation plan addresses the current need for evidence-based education. Since the No Child Left Behind legislation in 2001, teachers are required to demonstrate the use of scientifically based research (SBR) in curricular decision-making. The comparative study is an initial step in examining whether the use of comparative civics lessons makes a difference in student outcomes, namely, in their content knowledge and civic skills. The U.S Department of Education has been very specific about the need to demonstrate scientific evidence for new interventions. There are numerous guides and resources provided for teachers to develop skills in identifying and implementing rigorous (scientific) evidence to guide their decisions about which interventions (curriculum) to implement in their classrooms.9 The focus of the resources provided by the U. S. Department of Education, thus far, is on math and reading interventions. The use of a comparative study to investigate newly developed social studies curriculum is a first step in bringing this SBR approach to social studies teachers.