Sept 27th - Education Directors Meeting/Dinner [5-7pm]
Sept 28th - 8:30am – 3:30pm meeting/workshop
Location: Courtyard Marriott Chicago O’Hare; Meeting Rooms A&B
Free shuttle provided by the hotel from OHare [During the day (4am till 12:45am) the shuttle picks up from door number 2 at ~ 15 and 45 minutes after the hour intervals. Look for signs that direct you to the "bus and shuttle center " from the luggage pick up area.] To confirm: 1-847-824-7000
Download Agenda (155 KB)
Overall Goal: Evaluation Implementation - coordinated via collaboration with Tai group
Proposed meeting outcomes:
Participants will gain an increased knowledge of:
1. Existing and validated instruments for assessment of EO programs
2. Limitations of existing instruments
3. Methods of data analysis and implementation
The MRSEC Educators Network will:
1. Establish a plan for continued cross-site evaluation efforts
2. Will start a list of useful and validated instruments to be shared with network members
3. Will initiate the development [and sharing of] resources for data analysis and reporting
Workshop format:
Morning – short [30min max presentations] followed by discussion
Afternoon – workshop facilitated by Robert Tai
MRSECs will send [in advance] brief summaries of activities with potential for cross-site assessment.
The Tai group [ and others] will collect instruments with potential for utilization [and consideration and discussion at the workshop].
8:30 – 9:00am: Registration; Networking and continental breakfast
9:00 – 9:15am: Welcome and updates
- Christine Broadbridge and Dotti Pak: Welcome and goals for the meeting
- Sean Jones: Welcome from the NSF MRSEC Program
9:15 – 9:45am: “An Update on the MRSEC Cross-site REU Assessment with the Undergraduate Research Student Self Assessment Instrument”
- Tim Weston, Research Associate, University of Colorado Boulder Alliance for Technology, Learning and Society [ATLAS] Assessment and Research Center [ARC].
The Undergraduate Research Student Self Assessment (URSSA) is a survey instrument used by undergraduate research program administrators for evaluation of student outcomes. Two-hundred-nineteen students at 11 institutions with Materials Research Science and Engineering Centers (MRSEC) REU programs answered the URSSA survey during summers 2010 and 2011. We compared ratings from MRSEC across years and to students to those from the larger non-MRSEC comparison group from 83 other institutions with REU’s programs. MRSEC students in 2011 gave significantly higher average ratings than students in 2010 for one of the for URSSA indicators. Hispanic students, and first year MRSEC students also gave higher ratings than other students. Additional findings describe student reports on the types of research activities they engaged in, their satisfaction with program activities and their likelihood of participating in future educational and career activities.
9:45 – 10:15am: Presentation and Discussion led by Patricia Campbell, Ph. D.
In the past four years, NSF has come out with frameworks for evaluating impacts of informal science and broadening participation projects. These frameworks provide useful indicators and metrics as well as ways to evaluate specific types of informal science programs. Dr. Campbell, a coauthor of both frameworks, will provide an interactive overview of the frameworks with a focus on implications for education directors. In addition, from her current work, she will provide a series of tips on ways to improve evaluations with diverse populations.
10:15 – 10:30am: Coffee Break and Networking
10:30 – 11:00am: Presentation and Discussion led by Gil Noam, Ed.D., Ph. D.
The presenter will report on a recent summit on assessments in out-of-school time programs and will then introduce a method to evaluate quality programming (the Dimensions of Success tool) and a self-report method for engagement in STEM.
11:00 – 11:30am: Presentation and Discussion led by Cary Sneider, Ph. D.
The Next Generation Science Standards will be released in early 2013 for consideration by all 50 states. If widely adopted the new standards have the potential to fundamentally change the nature of science education in the US, and provide a solid foundation for collaboration and coordination of the patchwork of thousands of formal and informal science educational programs. The new standards will break substantially from previous documents in a number of ways, including an increased focus on the relationship between science and engineering, and specification of what students should know and be able to do in a form that will facilitate the alignment of curriculum, instruction, and assessment. This presentation will consider the implications of the new standards on the variety of K-12 programs currently being offered by the MRSEC consortium of universities, and suggest a process for coordinated planning that begins with assessment based on the Next Generation Science Standards.
11:30 – 12:00pm: Presentation and Discussion led by Kirsten Ellenbogen, Ph.D.
It can be difficult to evaluate the brief, episodic, STEM learning experiences that are characteristic of cart demonstrations, science blogs, and exhibits in informal science education. Five minutes of a volunteer-led activity at a demonstration cart may seem like an eternity when compared to a forty-second interaction with an exhibit, and neither may appear to have measurable learning outcomes. This presentation will build upon the Learning Science in Informal Environments report from the National Research Council and use examples from the Nanoscience Informal Science Education Network (NISE Net) to offer well-tested approaches to identifying realistic outcomes, evaluating across interconnected, brief experiences, and embedding assessments into program design.
12:00 – 1:15pm: Working/Networking Lunch; Nomination and Election of Vice Chair, MRSEC Education Directors
1:15 – 3:00pm: Presentation, Discussion and Workshop led by Robert Tai, Ph.D.
Workshop Outcomes:
Participants will gain an increased knowledge of:
1. Existing and validated instruments for assessment of
Participant Interest and Engagement
2. Limitations of existing instruments
3. Revisiting the FOCIS framework for the last MRSEC meeting
4. Examination of the FOCIS survey instrument
5. Methods of data analysis and implementation
6. Examples of Data Analysis: Examination of the analysis of
FOCIS survey pilot data
Expected long-term outcomes:
The MRSEC Educators Network will:
1. Establish a plan for continued cross-site evaluation efforts
2. Will start a list of useful and validated instruments to be shared with network members
3. Will initiate the development [and sharing of] resources for data analysis and reporting
3:00 – 3:15pm: Coffee Break and Networking
3:15 – 3:30pm: Wrap-up, post survey and feedback
Bios for speakers:
Robert Tai Ed.D., is currently Assoc. Prof. of Education at the Univ. of Virginia. His background includes undergraduate studies in physics and math at the Univ. of Florida, graduate work in physics at the Univ. of Illinois, Urbana-Champaign. He taught high school physics for three years before pursuing graduate studies in science education at Harvard. Upon graduation in 1998 he took a position as assistant professor of science education at the College of Staten Island City University of New York. After three years, he moved to the Curry School of Education at the University of Virginia, where he remains, earning tenure in 2007. His research agenda combines qualitative research techniques, such as clinical interviews and on-site observation studies, with the development and implementation of large-scale surveys. He learned this approach from his dissertation advisor, Philip M. Sadler, with whom he collaborated on Project FICSS and continues to collaborate, serving as a senior research consultant on several of Sadler’s subsequent projects. Tai has embarked on his own research agenda that includes Project Cross-Over, Impact of Specialized Public High Schools of Science, Math and Technology. Tai has 39 data-based peer-reviewed research journal articles to his credit with two appearing in Science.
Patricia Campbell, Ph. D.: President of Campbell-Kibler Associates, Inc, has been involved in educational research and evaluation with a focus on science and mathematics education and issues of race/ethnicity, gender and disability since the mid 1970's. Dr. Campbell is the co-author, with Beatriz Chu Clewell, of two practical guides on Building Evaluation Capacity (available from the Urban Institute): Guide 1: Designing a Cross-Project Evaluation and Guide 2: Collecting and Using Data in Cross-Project Evaluations. She earned her BS, from LeMoyne College (in Mathematics), her MS, from Syracuse University (in Instructional Technology) and her PhD, also from Syracuse University ( in Teacher Education). Dr. Campbell, formerly a professor of research, measurement and statistics at Georgia State University, has authored more than 90 publications co-authoring Engagement, Capacity and Continuity: A Trilogy for Student Success and Upping the Numbers: Using Research-Based Decision Making to Increase Diversity in the Quantitative Sciences with Eric Jolly and Lesley Perlman. She also is a co-author, with Beatriz Chu Clewell, of Good Schools in Poor Neighborhoods: Defying Demographics, Achieving Success.
Dr. Campbell’s professional activities have been diverse ranging from doing training in educational evaluation and research in South Africa and Uganda to serving as an expert witness in the sex discrimination case brought against the Citadel. She also was one of the co-authors of The AAUW Report: How Schools Shortchange Girls. Dr. Campbell was a member of the US Department of Education’s Impact Review Panel and was part of the team involved in the development of the National Science Foundation publications A Framework for Evaluating Impacts of Informal Science Education Projects and Infusing Equity in Systemic Reform: An Implementation Scheme.
Kirsten Ellenbogen, Ph.D.: As Senior Director for Lifelong Learning at the Science Museum of Minnesota, Dr. Ellenbogen leads the departments responsible for evaluation, learning research, field trip programming, school outreach, and programming for adults, families, and youth. In this role she fosters a strategic vision and culture of innovation around lifelong, lifewide, and lifedeep learning. She began her career as a demonstrator at the Detroit Science Center in 1987 and has worked in informal learning environments and academic settings as an exhibit and program developer, evaluator, and learning researcher. Leadership activities include serving as a founding officer of the Informal Learning Environments Research SIG-American Education Research Association, affiliated researcher of the Museum Learning Collaborative, project director for the Center for Informal Learning & Schools, King’s College London, and Noyce Leadership Fellow. Dr. Ellenbogen is currently the director of learning research for the Nanoscale Informal Science Education Network (NISE Net) and past-president of the Visitor Studies Association. She serves as co-principal investigator of the Center for Advancement of Informal Science Education (CAISE), a National Science Foundation-funded resource for organizations, programs, projects and individuals engaged in providing informal STEM learning experiences. In addition to her numerous publication, she was appointed to the National Academies of Science’s committee that produced Learning Science in Informal Environments. Dr. Ellenbogen holds a Ph.D. in Science Education from Vanderbilt University and a B.A. from University of Chicago.
Gil Noam, Ed.D., PhD (Habil), is the founder and director of the Program in Education, Afterschool & Resiliency (PEAR) at Harvard University. A full-time Associate Professor at Harvard Medical School and McLean Hospital focussing on prevention and resilience, Dr. Noam trained as a clinical and developmental psychologist and psychoanalyst in both Europe and the United States. Dr. Noam has a strong interest in translating research and innovation to support resilience in youth in educational settings. He served as the Director of the Risk and Prevention program at Harvard, and is the founder of the RALLY Prevention Program, an intervention that combines early detection of health, mental health and learning problems in middle school youth, and pioneers a new professional role -- "prevention practitioner". Dr. Noam has published over 200 papers, articles, and books on topics related to child and adolescent development, and risk and resiliency. He is the editor-in-chief of the award-winning journal New Directions in Youth Development: Theory, Practice and Research and consults nationally and internationally to youth development, education and child mental health organizations, foundations and agencies.
Cary Sneider, Ph.D. is Associate Research Professor at Portland State University in Portland, Oregon, where he teaches courses in research methodology in a Masters of Science Teaching Degree program. He also consults on diverse issues in STEM education, such as youth programs at science centers, educational standards, and assessment. He contributed to A Framework for K-12 Science Education (National Research Council 2012, which laid the groundwork for new science standards, and is currently a member of the writing team for Achieve, Inc., which is implementing the Framework under the title Next Generation Science Standards. In 2011 he joined the National Assessment Governing Board, which sets policy for the National Assessment of Educational Progress (NAEP), also known as “The Nation’s Report Card.” Until 2007 Dr. Sneider served as Vice President for Educator Programs at the Museum of Science in Boston, and prior to that he served as Director of Astronomy and Physics Education at the Lawrence Hall of Science at the University of California. Dr. Sneider’s curriculum development and research interests have focused on helping students unravel their misconceptions in science, on new ways to link science centers and schools to promote student inquiry, and on integrating engineering and technology education into the K-12 curriculum. Dr. Sneider earned a B.A. in Astronomy from Harvard College (1969), and a Secondary Teaching (1971), M.A. (1976) and Ph.D. in Science Education (1982) from the University of California at Berkeley. In 1997 he received the Distinguished Informal Science Education award from NSTA and in 2003 was named National Associate of the National Academy of Sciences.
Tim Weston is a research associate for the University of Colorado’s Alliance for Technology, Learning and Society (ATLAS) where he has worked on NSF, Department of Education, NASA and private foundation funded projects for 12 years. Weston specializes in the evaluation of programs with educational technology interventions, as well as assessing new STEM curricula. Currently he is the lead evaluator on the project Improving Science Learning through Dialogues with Virtual Tutors, a large-scale study of the use of automated tutors linked to the Full Option Science System (FOSS) curriculum for elementary schools. Weston also designs and develops assessment instruments and surveys for evaluation and testing. In this capacity he has worked to develop and improve the Student Assessment of Learning Gains (SALG) survey and website, the Undergraduate Research Student Self Assessment (URSSA), and ICARE Reading Assessment. Weston holds a Ph.D. in Research and Evaluation Methods in Education from the University of Colorado, has taught graduate statistics and research methods, and has served on advisory boards for science education projects and other non-profit organizations.
