Home / Climate / Climate Prosperity / Measuring Up: Benchmarking Canada’s Competitiveness in a Low-Carbon World / Measuring Up: Benchmarking Canada’s Competitiveness in a Low-Carbon World – Advise / Benchmarking – Chapter 5.3

Benchmarking – Chapter 5.3

5.3 Detailed Methodology The approach to conducting the benchmark analysis following the initial literature review, stakeholder consultation, and data feasibility study, consisted of five steps, as described below: STEP 1 // PLAN BENCHMARKING AND INDEX Deloitte was initially provided a draft benchmarking framework and list of 25 performance indicators developed by the Conference Board of Canada and tested out. The framework and indicators were developed as an output of a multi-stakeholder consultation exercise facilitated by the NRTEE. A draft benchmark framework was prepared by Deloitte & Touche LLP designed to measure a country’s ”carbon competitiveness” as expressed by overall level of country low-carbon performance. Focusing on energy production and use was deemed a useful, and realistic means of considering comparative low-carbon performance. Deloitte and the NRTEE project team subsequently agreed that the purpose of the exercise was to assess Canada’s position relative to other comparator countries in a variety of low-carbon performance categories beyond energy and emissions deemed important to Canada’s ability to reduce carbon emissions while fostering economic growth and prosperity. Two different types of indicators were initially identified: 1 // STATUS INDICATORS: ENERGY, EMISSIONS AND OTHER RELATED OUTPUTS 2 // CAPACITY INDICATORS: INNOVATION, INFRASTRUCTURE, INVESTMENT, SKILLS, INSTITUTIONS, AND POLICY It was acknowledged that differing stages of economic development, physical geography, and natural resource endowments make comparing countries difficult, because countries require different efforts to reach their specific objectives in relation to their current status and needs. However, it was felt that the indicator categories identified above had common importance for all countries and were therefore appropriate for the purpose of this exercise. It was agreed that given the project execution timelines, selection of indicators was to be based on the following criteria: // Relevance to emerging definitions of country low-carbon performance // Public availability of indicator data (e.g., free or nominal fee) from credible sources // Broad geographical scope (in order to capture all comparator countries) and regularly produced // Quality of the data (e.g., completeness, reliability of data) // Relative ease of data manipulation (e.g. in complete format within existing datasets, and / or in correct format within multiple data sources that could be quickly researched and manipulated) After application of these decision criteria, an initial list of 33 potential indicators with perceived high importance (which comprised the status and capacity indicators) was refined into a list of 15 feasible indicators. These were selected to provide a comprehensive and meaningful analysis, eliminate duplication and overlap among indicators, and create a basis for an equally weighted composite index. The benchmarking framework selected consisted of a single composite index comprising 15 metrics across the G8.[i] For classification purposes, the indicators were grouped into five equally balanced categories: // Emissions and Energy // Innovation // Skills // Investment // Policy and Institutions STEP 2 // RESEARCH AND GATHER METRIC DATA For each of the 15 metrics, we validated the existence and accessibility of the identified information sources. Where multiple data points were combined to create a metric, we sought to identify additional valid information sources (and additional indicators) to reinforce the benchmark data sets. Our review focused on secondary sources, including the following: // Organisation for Economic Co-operation and Development (OECD) // International Energy Agency (IEA) // Energy Information Administration (EIA) // Various published reports / studies. Where possible we sought to collect time series data and sought information for a common base year (e.g., 2006 for energy use and CO2 emissions data; 2007/08 for institutions and policy data). Time series data, while available, does not factor into the index or any of the weightings. It is used to provide context to the performance of particular countries. Through this process the final list of 15 indicators were gathered and populated to conduct the country benchmark analysis. STEP 3 // CLEANSE AND VALIDATE DATA This step involved validating, organizing, and “cleansing” data (i.e., some interpretation was required) in order to present the data in a uniform manner so that it could be accurately analyzed. This included: // Documenting sources of data and data definitions, // Applying a naming scheme to identify data set / type of data, and // Scanning the data for truncated / repeating / nonsense / missing / zero values / outliers / duplicates. The indicators differed in terms of completeness, magnitudes, trends, and volatility. We had to normalize, standardize, and restructure the data in a manner that allowed it to be properly analyzed. Sub-tasks within this step included the following: // Consolidation of fragmented data or data set components // Imputation of missing data (this was a key issue with countries such as Russia) – depending on the indicator under consideration, we either defaulted to the most recent year with data available // Standardization of data (to enable direct comparisons) Where possible and relevant, we adjusted the metrics for size by dividing each indicator by a denominator (such as GDP, population) in order to control for the size of a country. This approach was undertaken to avoid rewarding (or penalizing, depending on the indicator) large countries simply for being large. STEP 4 // COMPILE INDEX Based on the 15 metrics selected, their categories, and the data sources, the index was compiled and metrics populated for each of the G8 countries. Where data was available, metrics were compiled for the three additional countries as well (Norway, China and Australia), but were not included in the Low-Carbon Performance Index (LCPI). To calculate the actual composite index, a country’s raw value for each indicator was first normalized between 0 and 100 based on relative position in the G8. Each indicator was equally weighted within its category, and the normalized scores were then summed up by category. For example, the country with the best ranking was given 100, and the lowest was given 0; all others received a score in between 0 and 100 based on their relative position. Weightings were applied at the category level—this exercise applied an equal weighting. The sum of the equally weighted categories produces the index. This is a standard practice in the benchmarking of performance where no rationale can be applied for differentiated weightings. The framework was designed to be replicable in order that improvements over time can be monitored and Canada’s relative position reassessed. The formula above calculates a country’s normalized score on a scale between 0 and 100 while maintaining the relative differences between countries’ raw scores. For example, in the carbon productivity indicator, Canada ranks in seventh position, scoring a raw value of 1.78 and a normalized value of 28.07. The normalized value for Canada reflects its relative position between the metric leader (France, with a raw value of 4.83 and normalized value of a hundred) and the metric laggard (Russia, with a raw value of 0.59 and normalized value of zero).[j] The normalized scores are summed up by category equally. Weightings are applied at the category level. The sum of the weighted categories produces the LCPI. The sensitivity of the weightings is discussed further in the next section. As outlined above, the selection of the indicators, building block categories, and weightings was strongly influenced by the pre-project scoping exercise. No rank-correlationtype exercise was conducted to determine the relationship between the Index and potential indicators in order to screen for their inclusion, and as a result no such analysis was available as input to the relative importance/attributed weightings for each of the final selected indicators. STEP 5 // PERFORM ANALYSIS AND SENSITIVITY TESTING In the absence of such analysis to inform the relative weighting of indicators, building block and support categories, we defaulted to an equal weighting approach in the initial development of the LCPI. While there are strengths and weaknesses associated with adopting an equal weighting approach, we felt that in this situation where we are conveying not just the results of an LCPI but the importance of developing one, that such an approach was prudent and defensible. Benefits of this approach include avoiding the introduction of deliberate bias of the index results against any particular country or group of countries, and enabling the reproduction and enhancement of the index in subsequent iterations (equal weighting of the building block categories means that as further research is conducted, and as data improves, indicators can be tweaked and/ or replaced without affecting the overall integrity of the index). We then ran multiple scenarios to test the relative weighting assumptions to the overall index results. These scenarios were selected through discussion with the NRTEE. The advantages of selecting different scenarios include helping to gauge the robustness of the results, increase the index transparency, and identify the countries whose performance can improve or deteriorate under certain assumptions. The following scenarios were selected: // EQUAL WEIGHTING OF EACH OF THE FIVE CATEGORIES; // TWO-THIRDS WEIGHTING ON CANADA’S TWO STRONGEST CATEGORIES (INNOVATION, SKILLS); 1/3 WEIGHTING ON ALL OTHER CATEGORIES; AND // TWO-THIRDS WEIGHTING ON CURRENT CAPACITY AND GOVERNANCE (EMISSIONS & ENERGY, POLICY & INSTITUTIONS); 1/3 WEIGHTING ON ALL OTHER CATEGORIES. The relative rankings of countries remained relatively static across the three selected scenarios, as outlined below: