Smart Cities Challenge Final Proposal

Chapter 2 – Performance measurement

March 5, 2019

 

Chapter 2 – Performance measurement
 
Winners will receive their prize money through an outcomes-based contribution agreement with Infrastructure Canada. Payments will be triggered by the successful achievement of progress toward outcomes, tailored to the requirements of each winning project. In other words, payments will not be linked to the reimbursement of eligible costs. This chapter serves as the foundation of this outcomes-based contribution agreement.
 

Appendix 2 (page 24) provides a detailed guidance on outcomes-based performance measurement.
 

Provide an outcomes-based performance measurement plan for the implementation phase, including:

  • Project activities and their links to outputs and outcomes
  • Project timelines, deliverables, and milestones that are ambitious yet attainable
  • Payment schedule with amounts that are reasonable and rational in the context of project timelines, deliverables, milestones, and prize category
  • Qualitative and quantitative performance indicators and related data sources that are meaningful and can measure short, medium, and long-term progress towards outcomes
  • Monitoring, reporting, and evaluation strategies and checkpoints for contingencies and course corrections, if necessary
  • Identification of risks and development of appropriate mitigation strategies
  • Other details, as required

 

Evaluation criteria

  • Plan is detailed, complete, feasible, and well-suited to achieve the outcomes
  • Plan forms a solid basis for development of outcomes-based contribution agreement
  • Risk strategy is thorough and adequately addresses key risk

 

Chapter 2 – Performance measurement

 

Strategy

An outcome-based performance approach is a results-based management tool that can be used to guide the selection, development and ongoing use of performance measures. For this project, this strategy will be adopted, which will enable the project team to continuously monitor and assess the results of programs as well as make informed decisions and take appropriate mitigation actions should they be required. A success outcome-based performance measurement strategy will also provide effective and relevant departmental reporting on the program and ensure that credible and reliable performance data is being collected to effectively support evaluation.
 

Approach

The key to achieving the desired outcomes of this project is to set clearly defined goals which are achievable for the timeline of this project. This will involve defining metrics that can be set to clearly measure the project’s success, as well as to identifying any deviations which can then be quickly remedied. The metrics to be used for this project will follow the SMART methodology for goal setting, as defined below:
 

  1. The set outcomes of this project are Specific
  2.  
    The desired outcomes for this project are clear, well defined and based upon the specific requirements for the completion of this project.
     

  3. The set outcomes of this project are Measurable
  4.  
    Metrics have been developed to measure the progress of this project. The project team will use these metrics to assess levels of project completion and identify any deviations.
     

  5. The set outcomes of this project are Achievable
  6.  
    The set outcomes of this project were developed by assessing and interpreting relevant data and by receiving input from various experts in relevant fields.
     

  7. The set outcomes of this project are Relevant
  8.  
    The defined outcomes of this project are essential for the overall success of the project and were developed by pinpointing the specific needs of the project and by having discussions and input from stakeholders.
     

  9. The set outcomes of this project are Time-Bound
  10.  
    The outcomes of the project have set target dates for completion based upon required inputs, activities and outputs.

 

Logic Model for Performance Measurement

A basic logic model will also be used to understand the process of measuring each metric’s impact throughout the project. As shown below performance measures can be depicted linearly, however, it is important to note that for each input and relative activity there will be short, medium and long-term results.
 

As part of developing our measurement approach, the project team completed a resource assessment which has resulted in the identification and classification of the project’s inputs. The project team has also developed an implementation plan for the activities required to achieve the desired outcomes and impacts. Each of these activities has an associated immediate output which will then result in the desired outcomes and impacts that are essentially the goals of our project. Further details on the aforementioned activities (resource assessment, implementation planning, and objective/outcome setting) are included in the Project Management Plan, presented in Chapter 3. The project specific components of the above logic model are summarized in the following sections:
 

Inputs:

Project Inputs can be defined as financial and non-financial resources used to complete project activities, produce outputs and achieve outcomes. Our project inputs include, but are not limited to, the following:

  • Project team members (Chief and Council, Elders, Subject Matter Experts, Project Manager)
  • Teaching Staff
  • Procured software and technology (i.e. code.org curriculum, Open edX platform, etc.)
  • Financial Contributions

 

Activities:

Project activities can be defined as the planned work, actions, and events that our project team will undertake to produce one or more outputs. The activities to be undertaken as part of this project include:

  • Record written aadsookaanan (sacred stories)
  • Produce Waawaa video
  • Overdub videos in our Nishnaabe language
  • Produce STEM videos
  • Implement K-12 curriculum augmentation
  • Upload all data and materials to Open edX

 

Outputs:

Outputs can be described as the direct result of the planned activities.

  • Recordings of aadsookaanan (sacred stories)
  • 2,000 hrs of immersion video
  • 1,000 hrs of STEM video
  • STEM curriculum framework and materials
  • Digitized publicly accessible version of the above

 

Outcomes:

Outcomes can be described as the medium-term impact of the project.

  • Increase in number of K-12 STEM courses successfully completed by K-12 students
  • Decrease in percentage of K-12 students needing remedial help in STEM subject areas
  • Increase in number of K-12 students who have at least a basic ability to code
  • Increase in number of K-12 students who know the key concepts expressed in each of our core aadsookaanan
  • Increased annual participation rate in number of traditional Nishnaabe activities for K-12 students
  • Minimum 2,000 hours of Nishnaabe language acquired by graduating eighth-graders
  • Minimum 1,000 hours of Nishnaabe-language STEM video successfully completed by graduating high school seniors

 

Impact:

Impacts can be described as the long-term outcome of the project. For the Biigtigong Nishnaabeg Smart Cities project this includes impacts to our community as well as to society as whole, as described below:
 

Community Impact:

  • Youth who are “better-educated”
  • Youth who are “more-employable”
  • Youth who are “better-grounded” in their Nishnaabe identity
  • Youth who are “more holistically Nishnaabe”

 

Social Impact:

  • Preserved language
  • Preserved cultural teachings
  • Model for connecting STEM, and traditional/cultural learning
  • Onboarding approach to creating Smart Cities regardless of current resources or capacity.

 

Indicators and Milestones:

Key Performance Indicators (KPIs) allow project managers to gauge whether the planned activities have been successful. This is measured by how well they have been able to meet the objectives of the project. The four (4) main objectives for the project have been outlined in Chapter 3. The completion of the activities outlined in the implementation plan for this project should satisfy the objectives of the project and should bring tangible value added to the community and society as a whole.
 

It is important to have both qualitative and quantitative indicators for tracking, measuring and evaluating performance, to capture the holistic perspective of the project goals. It is also important to select a reasonable number of indicators to ensure that performance measurements are met and accomplished, while not over specifying and removing opportunities that may evolve throughout the project lifecycle.
 

Quantitative Key Performance Indicators (KPI):

The following section outlines some of the KPIs identified for the performance measurement process of our project. This section details the KPI, the data source and associated responsible resource. The table following this section summarizes this information in the context of associated outputs, outcomes, and target completion dates.

  1. Number of aadsookaanan lessons created
  2. Number of hours of video produced
  3. Number of courses for which curriculum is implemented

 
The above KPIs are designed to track performance with respect to the outputs for this project. Data will be provided in the form of a summary report to be submitted by the SMEs to the project manager who will then report to the advisory committee on a biannual basis.

  1. Number of K-12 STEM courses successfully completed by K-12 students
  2. Percentage of K-12 students needing remedial help in STEM subject areas
  3. Number of K-12 students who have at least a basic ability to code
  4. Number of K-12 students who know the key concepts expressed in each of our core
    aadsookaanan
  5. Annual participation rate in number of traditional Nishnaabe activities for K-12 students

 

Qualitative Key Performance Indicators:

  1. Teacher feedback and opinions
  2. Student feedback and opinions

 

Other Key Performance Indicators

In addition to the specific KPIs that the project team has identified for tracking performance on specific project outputs and outcome, all outputs will also be evaluated through implementation of financial- and schedule-based KPIs. Financial- and Schedule-based KPIs for reporting on outcomes are less reliable and therefore these KPIs will only be associated with the following project outputs:

  1. Estimated Budget vs. Actual Expenditures (Financial)
  2. Target Completion Date vs. Actual Completion Date (Schedule)

 

Source of Data and Reporting Role and Responsibilities

With respect to tracking the Key Performance Indicators listed above, much of the data required including; participation rates, hours of aadsookaanan and STEM video developed, student grades etc. can be collected directly from the open-source online Open edX platform. clear and concise reporting of the Key Performance Indicators will also be a vital component when measuring the progress made throughout this project.
 

Since teaching staff will be responsible for the direct oversight of students’ engagement with the curriculum (i.e. grading), they will play a large role in assessing and reporting on the performance of students and the quality of educational materials developed. Reporting of these performance indicators will then be communicated to the project team for further analysis. The project team will be responsible for tracking their internal performance, regarding budgeting and scheduling.
 

 

Payment Schedule

Understanding that the intent is to develop an outcome-based contribution agreement with Infrastructure Canada, where payments will be triggered by the successful achievement of progress toward outcomes, we have developed a payment schedule that is simple, manageable and associated with the completion of milestones as outlined in detail in Chapter 8.
 

Risk Management

Identifying risks that may be presented throughout the project lifecycle is an essential component of project management as it allows the project team to proactively establish mitigation strategies to effectively minimize the impact of the risk on the project before it occurs. The table below summarizes a list of potential risks related to the above Key Performance Indicators, as well as the associated mitigation strategies that will be implemented to ensure that the delivery of this content remains effective, up to date, and on schedule.