.

Measurement

How we make a difference

Our Methodology

We approach measurement holistically, measuring not only the outcomes and impact of our programs, but also how we offer our programs using a program quality lens. Together, these data provide a deeper understanding of what we do and how we impact youth in our communities.

We collect a mix of qualitative (i.e., internal and external program observations, interviews, focus groups) and quantitative data (i.e., survey scores) which provide exploratory insights and useful context to build compelling narratives about our programs. We track growth and change in each outcome through pre and post surveys and use secondary research as a comparative benchmark to approximate what it would have been like otherwise for each youth by comparing their experience and results with similar youth who did not participate in our programs. We use industry leading frameworks and tools to help us understand how our programs are facilitated, continually identify areas for improvement, and provide a high-quality experience for all youth. Our measurement efforts are all-encompassing as we connect with campers, the Youth Advisory Council, alumni, and staff to fully understand the camp experience.

We firmly believe that measurement is continual. We don’t just measure once, but instead we measure multiple times using a cycle. Through continual measurement we identify what we’re doing well, determine where we can improve, and implement key changes to our programs and staff training.

Program Quality Measurement

Program quality is key to understanding how Tims Camps programs are facilitated for our youth. Aligned with many other educational settings and out-of-school-time programs, the fields of positive youth development and camp are prioritizing a program quality lens to ensure that every youth has a quality experience at camp. Through participation in high-quality settings and programs, youth develop greater positive outcomes that are associated with wellbeing and thriving in young adulthood. We seek to measure program quality across all our programs and use program quality data to inform changes and improvements to our programs.

Program quality refers to how programs are facilitated at camp through staff practices that create four environments. The David P. Weikart Centre for Youth Program Quality (part of the Forum for Youth Investment) structures these environments in a pyramid, similar to Maslow’s hierarchy of needs. The lower levels of the pyramid must be satisfied (i.e., the environments must be created) before higher levels of the pyramid can be reached. Health and safety standards are addressed in accreditation models, and therefore program quality focuses primarily on social emotional aspects of the program.

The four environments are:

Pyramid showing pillars of program quality

Copyright 2021 The Forum for Youth Investment

  • Safe: The program is a psychologically and emotionally safe environment for youth. The environment is free from bias and staff actively seek to include all young people and create a space where every youth feels they belong and people with different identities are celebrated.
  • Supportive: Staff create a learning environment in which young people are supported to try different things, develop new skills, and understand and manage their emotions and social interactions.
  • Interactive: The environment is rich in opportunities for youth to work together to achieve goals, independently succeed, and understand others’ strengths, areas for improvement, individual perspectives, and experiences.
  • Engaging: Program participants are encouraged to more deeply engage in learning and learning transfer beyond the program. Participants are also supported in taking agency, ownership, and initiative over their learning through planning and goal setting.

At Tims Camps, we began assessing program quality through our participation in the American Camp Association’s Camp Program Quality Initiative (CPQI). This was a pilot program to understand and develop sustainable systems of program quality in the camp space. We were honoured to be selected to participate in the CPQI pilot.

Through our participation in the CPQI and our initial assessment, we realized the need to prioritize program quality in all that we do and have launched a full-scale continuous improvement cycle focused on program quality assessment. We are using a research-validated measurement tool for internal and external assessments, engaging multiple stakeholders in the assessment process (i.e., staff across the organization, campers, Youth Advisory Council members, school partners), and creating new and unique ways of understanding program quality from the youth perspective. We are using program quality data to reflect on our programs and make changes to better serve youth.

Summer Program Outcome Measurement

Tims Camps Summer Program is a strength-based, high-impact leadership development program with a rigorous learning model based on research. Our program frameworks are grounded in evidence that focuses on developing tangible skills to help youth succeed and thrive in young adulthood. We intentionally design our programs to meet the short, medium, and long-term needs of the youth we serve, and we rigorously measure these outcomes with our youth to identify program areas to target.

Our Summer Program methodology assesses each individual youth’s progress towards demonstrating the priority outcomes of social and emotional skills, learning and innovation skills, post-secondary access and independent living capability. See the program outcome definitions listed below.

In creating this framework, we mapped our outcomes against similar organizations to answer two primary questions:

  • Are these outcomes what our youth need?
  • Are these outcomes realistic and attributable to our programs?
Outcomes Framework Summer Program
Expand for plain text version

The impacts of our summer program build on each other as participants grow through the multi-year program, starting with short-term outcomes that lead to intermediate outcomes and create long-term outcomes.

Starting in the short term, youth at Tims Camps develop foundational skills they need to become resourceful, responsible, and empowered individuals. These skills fall into two categories:

Social and Emotional Skills

  • Positive identity
  • Positive behaviors
  • Positive connections

Learning and Innovation Skills

  • Cognitive skills
  • Organizational skills
  • Interpersonal skills

Once mastered, those short term skills enable youth at Tims Camps to be prepared for life after high school. Post high school readiness is marked by two outcomes:

  • Postsecondary access
  • Independent living capability

Having gained access to postsecondary education or joined the workforce, graduates of Tims Camps continue to develop along the long-term trajectory of the program as they successfully transition to adulthood. This is marked by three outcomes:

 

  • Postsecondary completion
  • Meaningful employment
  • Supportive relationships
Outcome Landscape Analysis Summer
Expand for plain text version

Among 16 organizations serving youth, we found universal alignment on the focus on social and emotional skills. Likewise, there was widespread agreement on importance of learning and innovation skills (12 out of 16 organizations). Where Tims Camps is unique is in the emphasis on job readiness (only two of 16 organizations focused on this), postsecondary access and financial knowledge and skills—both of which are singularly emphasized at Tims Camps. There are other areas that organizations emphasize not currently found at Tims Camps summer program, including environmental awareness, healthy behaviors, academic achievement and civic engagement.

School Program Outcome Measurement

Working with schools in priority neighbourhoods for day or overnight camp sessions per school year, our School Program inspires students to become visible, positive leaders in their schools and make their communities more positive places. This model allows us to leverage the capacity of our camps and serve more youth in our communities in a meaningful way.

In addition to the focus on social and emotional skills and learning and innovation skills, the School Program focuses on positive contributions to the school environment including motivation to learn, volunteerism and classroom behaviour. The long-term goal is to strengthen communities with productive avenues for family involvement, supportive neighbourhoods and empowered youth. This program truly relies on collective impact - working hand-in-hand with educators, schools and school boards to move the needle on these key indicators. See the program outcome definitions listed below.

School Program evaluation includes a pre- and post-youth survey, evaluating changes to key indicators using report card data (where applicable) and a retrospective pre- and post-educator survey.

School program outcome framework
Expand for plain text version

Similar to the impacts of our summer program, youth in Tims Camps school program begin with foundational skills and behaviours in the short term that lead to intermediate and long-term outcomes.

Starting in the short term, youth in the Tims Camps school program establish skills and behaviours they need to become resourceful, responsible, and empowered individuals. These fall into three categories:

Social and Emotional Skills

  • Positive identity
  • Positive behaviors
  • Positive connections

Learning and Innovation Skills

  • Cognitive skills
  • Organizational skills
  • Interpersonal skills

Positive Contributions

  • Motivation to learn
  • Classroom behaviour
  • Volunteerism

Once mastered, those short-term skills enable youth in Tims Camps school program to invest in their academic success and the future of their community. Youth contribute to a positive school climate as marked by two outcomes:

  • Academic growth
  • Civic responsibility

Having invested themselves in their own growth and civic responsibility, youth in Tims Camps school program contribute to stronger communities in the long term. This is marked by three outcomes:

 

  • Empowered youth
  • Supportive neighbourhoods
  • Avenues for family involvement
Program Outcome Definitions
Social & Emotional Skills
  • Positive Identity - Youth develop self-awareness, emotional intelligence, self-confidence, self-efficacy and a sense of empowerment
  • Positive Behaviors - Youth develop the self-management, impulse control, stress-management, self-motivation, and perseverance necessary to behave responsibly and positively
  • Positive Connections - Youth develop healthy and rewarding relationships with peers, adults, and the community
Learning & Innovation Skills
  • Cognitive Skills - Youth develop skills in critical and analytical thinking, innovation, and intellectual curiosity
  • Organizational Skills - Youth develop time management, planning, resourcefulness, and goal-setting skills
  • Interpersonal Skills - Youth develop skills in communication, collaboration, problem-solving, conflict-resolution, teamwork, leadership and cultural sensitivity
Post-High School Readiness
  • Postsecondary Access - Youth obtain their high school diploma, gain access to financial support and gain knowledge of post-secondary education opportunities
  • Independent Living Capability - Youth demonstrate job readiness skills and financial knowledge skills
Successful Transition to Adulthood
  • Postsecondary Completion - Tims Camps alumni complete post-secondary education degree or certification 
  • Meaningful Employment - Tims Camps alumni obtain a job in their desired career path
  • Supportive Relationships - Tims Camps alumni feel a sense of belonging with peers and the community
Positive Contributions
  • Classroom Behaviour - Youth demonstrate active participation, self-management and respectful and caring attitudes in the classroom
  • Motivation to Learn - Youth demonstrate positive attitudes towards school and learning, a growth mindset, and commitment to academics and good attendance
  • Volunteerism - Youth volunteer through community service and social activism to bring about social change
Positive School Climate
  • Academic Growth - Youth achieve a higher letter grade on school marks or standardized tests
  • Civic Responsibility - Youth demonstrate civic efficacy, civic interest and awareness, civic attitude and civic initiative
Stronger Communities
  • Empowered Youth - Tims Camps alumni demonstrate civic leadership and academic success
  • Supportive Neighbourhoods - Residents believe in the future of their community, participate in the community, and maintain a sense of belonging
  • Avenues for Family Involvement - Parents/guardians are provided with a welcoming school environment and have opportunities to be involved in youth’s education and success
Footnote
  1. Waves of the Future: The First Five Years of the 4-H Study of PYD. 2009.
  2. Identifying 4-H Camping Outcomes Using a Standardized Evaluation Process Across Multiple 4-H Educational Centers, 2003.
  3. Addressing (Nature-Deficit Disorder): A Mixed Methods Pilot Study of Young Adults Attending a Wilderness Camp, 2015.
  4. Exploring the Impact of a Wilderness-Based Positive Youth Development Program for Urban Youth
  5. The Art & Science of Creating Effective Youth Programs. Algorhythm.
  6. A Multi-Method Approach to Understanding Empowerment Processes and Outcomes of Adventure Education Program Experiences, 2009.
  7. Outward Bound Program Outcomes: http://www.outwardbound.org/about-outward-bound/philosophy/outcomes/
  8. Research and Evaluation Projects Update. Foundation for Youth Development. 2014.
  9. Project K in Black & White: A theory-driven & randomised controlled trial evaluation of a youth development programme. 2012.
  10. Youth Development and the Camp Experience. New Directions in Youth Development. 2011.
  11. Youth Development Outcomes of the Camp Experience: Evidence for Multidimensional Growth. 2006.
  12. Components of Camp Experiences for Positive Youth Development, 2006.
  13. Wilderness challenge programs for delinquent youth: a meta analysis of outcome evaluations. 1999.
  14. Adventure Camp Programs, Self-Concept, and Their Effects on Behavioral Problem Adolescents. 2007.
  15. Factors associated with positive youth development outcomes of youth summer camps. 2013.
  16. The Effects of a Summer Camp Experience on Factors of Resilience in AtRisk Youth. 2012.
  17. Wilderness challenge programs for delinquent youth: a meta analysis of outcome evaluations. 1999.
  18. Influences of Resident Camp Experiences on Career Choice: A Case Study of Female Alumnae
  19. Transforming Leadership: Focusing on Outcomes of the New Girl Scout Leadership Experience. 2008
  20. Boy Scouts of America Summer Camp Outcomes Study. 2001.