Foundations for Evidence-Based Policymaking Act of 2018 - Program Evaluation Competency Model

UNITED STATES OFFICE OF PERSONNEL MANAGEMENT Washington, DC 20415 November 8, 2023 The Director Memorandum for Chief Executive Officer Councils From: Kiran A. Ahuja Director Subject: Foundations for Evidence-Based Policymaking Act of 2018 – Program Evaluation Competency Model

The U.S. Office of Personnel Management (OPM) is working in consultation with the Office of Management and Budget (OMB) to implement specific requirements of Public Law 115-435, Foundations for Evidence-Based Policymaking Act of 2018. In accordance with the Act, OPM is required to identify key skills and competencies needed for program evaluation in an agency; establish a new occupational series, or update and improve an existing occupational series, for program evaluation within an agency; and establish a new career path for program evaluation within an agency. To fulfill these requirements, OPM conducted an occupational study on program evaluation work. Additionally, this study supports OMB’s guidance, Phase 4 Implementation of the Foundations for Evidence-Based Policymaking Act of 2018: Program Evaluation Standards and Practices (March 10, 2020) by informing agency efforts to better build the program evaluation workforce. OPM is pleased to provide the attached program evaluation competencies and competency model. OPM will release classification policy on the program evaluation series and career path guidance through separate issuances. In support of this effort, OPM consulted with subject matter experts (SMEs) identified by OMB, as well as additional SMEs collectively covering the 24 Chief Financial Officers (CFO) Act of 1990 (31 U.S.C. § 901(b)) agencies, to identify key skills and competencies needed to perform program evaluation work. Feedback was solicited via focus groups and Governmentwide surveys. The qualitative and quantitative data collected through the focus groups and surveys was used to paint a comprehensive picture of program evaluation work throughout the Federal government and to validate the program evaluation competency model. The competencies identified may be used by agencies in the workforce planning, recruitment, employee selection, training and development, and performance

management for their program evaluation positions. The competency model includes the following: • the program evaluation competencies, • the program evaluation general and technical competencies for each applicable grade level, • the program evaluation general and technical competency definitions, and • an addendum to support agency efforts to enhance their program evaluation workforce research capacity. We appreciate your assistance with developing policy in line with the requirements of the Evidence Act. If you have any questions regarding the competency model, please contact Classification and Assessment Policy at competency@opm.gov. Attachment: Program Evaluation Competency Model (see 508-compliant PDF below) cc: Chief Human Capital Officers (CHCOs), Deputy CHCOs, Human Resources Directors, Evaluation Officers, Chief Information Officers, and Chief Data Officers Table of Contents Program Evaluation Competencies......................................................................... 4 Program Evaluation General and Technical Competencies by Grade ........................ 5 Program Evaluation General Competencies by Grade .............................................. 6 Program Evaluation Technical Competencies by Grade ........................................... 7 Program Evaluation Competency Model General Competency Definitions ............... 8 Program Evaluation Competency Model Technical Competency Definitions .......... 11 Program Evaluation Competency Model Research Addendum ............................... 13

Program Evaluation Competencies The table below presents the 33 competencies found to be important for performing program evaluation work. Through job analysis (5 CFR § 300.103), agencies must determine the applicability of these competencies to positions within their agency. Please refer to OPM’s Delegated Examining Operations Handbook for more information on conducting a job analysis. General Competencies Technical Competencies • Accountability • Conflict Management • Continual Learning • Creative Thinking • Customer Service • Decision Making • Flexibility • Influencing/Negotiating • Integrity/Honesty • Interpersonal Skills • Leadership • Legal, Government and Jurisprudence • Manages Human Resources • Oral Communication • Organizational Awareness • Partnering • Planning and Evaluating • Political Savvy • Problem Solving • Reasoning • Teamwork • Technical Competence • Technical Credibility • Writing • Data Analysis • Evaluation • Knowledge Management • Organizational Performance Analysis • Performance Measurement • Project Management • Quality Management • Research • Stakeholder Management Program Evaluation General and Technical Competencies by Grade The following tables list the 24 general and eight technical competencies that have been confirmed as appropriate for employee selection on a governmentwide basis for program evaluation work at the grades indicated. Note that while the competency “Research” is identified as important for performing program evaluation work, it was not confirmed as appropriate for selection governmentwide at the grade levels indicated. See the addendum on research for more details. Program Evaluation General Competencies by Grade Grade 9 Grade 11 Grade 12 Grade 13 Grade 14 Grade 15

• Accountability • Conflict Management • Continual Learning • Creative Thinking • Customer Service • Decision Making • Flexibility • Integrity/Honesty • Interpersonal Skills • Oral Communication • Planning and Evaluating • Problem Solving • Reasoning • Teamwork • Technical Competence • Technical Credibility • Writing • Accountability • Conflict Management • Continual Learning • Creative Thinking • Customer Service • Decision Making • Flexibility • Integrity/Honesty • Interpersonal Skills • Oral Communication • Planning and Evaluating • Problem Solving • Reasoning • Teamwork • Technical Competence • Technical Credibility • Writing • Accountability • Conflict Management • Continual Learning • Creative Thinking • Customer Service • Decision Making • Flexibility • Integrity/Honesty • Interpersonal Skills • Oral Communication • Planning and Evaluating • Problem Solving • Reasoning • Teamwork • Technical Competence • Technical Credibility • Writing • Accountability • Conflict Management • Continual Learning • Creative Thinking • Customer Service • Decision Making • Flexibility • Influencing/Negotiating • Integrity/Honesty • Interpersonal Skills • Leadership • Oral Communication • Organizational Awareness • Partnering • Planning and Evaluating • Problem Solving • Reasoning • Teamwork • Technical Competence • Technical Credibility • Writing • Accountability • Conflict Management • Continual Learning • Creative Thinking • Customer Service • Decision Making • Flexibility • Influencing/Negotiating • Integrity/Honesty • Interpersonal Skills • Leadership • Oral Communication • Organizational Awareness • Partnering • Planning and Evaluating • Problem Solving • Reasoning • Teamwork • Technical Competence • Technical Credibility • Writing

• Accountability • Conflict Management • Continual Learning • Creative Thinking • Customer Service • Decision Making • Flexibility • Influencing/Negotiating • Integrity/Honesty • Interpersonal Skills • Leadership • Legal, Government and Jurisprudence • Manages Human Resources • Oral Communication • Organizational Awareness • Partnering • Planning and Evaluating • Political Savvy • Problem Solving • Reasoning • Teamwork • Technical Competence • Technical Credibility • Writing Program Evaluation Technical Competencies by Grade Grade 9 Grade 11 Grade 12 Grade 13 Grade 14 Grade 15 • Data Analysis • Evaluation • Knowledge Management • Data Analysis • Evaluation • Knowledge Management • Organizational Performance Analysis • Data Analysis • Evaluation • Knowledge Management • Organizational Performance Analysis • Performance Measurement • Data Analysis • Evaluation • Knowledge Management • Organizational Performance Analysis • Performance Measurement • Project Management • Stakeholder Management • Data Analysis • Evaluation • Knowledge Management • Organizational Performance Analysis • Performance Measurement • Project Management • Quality Management • Stakeholder Management • Data Analysis • Evaluation • Knowledge Management • Organizational Performance Analysis • Performance Measurement • Project Management • Quality Management • Stakeholder Management

Program Evaluation Competency Model General Competency Definitions This section presents definitions for the general competencies found within the Program Evaluation Competency Model. Accountability - Holds self and others accountable for measurable highquality, timely, and cost-effective results. Determines objectives, sets priorities, and delegates work. Accepts responsibility for mistakes. Complies with established control systems and rules. Conflict Management - Manages and resolves conflicts, grievances, confrontations, or disagreements in a constructive manner to minimize negative personal impact. resolve their problems, or satisfy their expectations; knows about available products and services; is committed to providing quality products and services. Decision Making - Makes sound, wellinformed, and objective decisions; perceives the impact and implications of decisions; commits to action, even in uncertain situations, to accomplish organizational goals; causes change. Continual Learning - Assesses and recognizes own strengths and weaknesses; pursues self-development. Flexibility - Is open to change and new information; adapts behavior or work methods in response to new information, changing conditions, or unexpected obstacles; effectively deals with ambiguity. Creative Thinking - Uses imagination to develop new insights into situations and applies innovative solutions to problems; designs new methods where established methods and procedures are inapplicable or are unavailable. Influencing/Negotiating - Persuades others to accept recommendations, cooperate, or change their behavior; works with others towards an agreement; negotiates to find mutually acceptable solutions.

Customer Service - Works with clients and customers (that is, any individuals who use or receive the services or products that your work unit produces, including the general public, individuals who work in the agency, other agencies, or organizations outside the Government) to assess their needs, provide information or assistance, Integrity/Honesty - Contributes to maintaining the integrity of the organization; displays high standards of ethical conduct and understands the impact of violating these standards on an organization, self, and others; is trustworthy. Interpersonal Skills - Shows understanding, friendliness, courtesy, (for example, technical, sensitive, controversial); makes clear and tact, empathy, concern, and politeness to others; develops and maintains effective relationships with others; may convincing oral presentations; listens to others, attends to nonverbal cues, and responds appropriately. include effectively dealing with individuals who are difficult, hostile, or Organizational Awareness - Knows the distressed; relates well to people from varied backgrounds and different situations; is sensitive to cultural diversity, race, gender, disabilities, and other individual differences. Leadership - Influences, motivates, and challenges others; adapts leadership styles to a variety of situations. Legal, Government and Jurisprudence - Knowledge of laws, legal codes, court procedures, precedents, legal practices and documents, Government regulations, Executive orders, agency rules, Government organization and functions, and the democratic political

organization's mission and functions, and how its social, political, and technological systems work and operates effectively within them; this includes the programs, policies, procedures, rules, and regulations of the organization. Partnering - Develops networks and builds alliances; collaborates across boundaries to build strategic relationships and achieve common goals. Planning and Evaluating - Organizes work, sets priorities, and determines resource requirements; determines Manages Human Resources - Plans, short- or long-term goals and strategies to achieve them; coordinates with other organizations or parts of the distributes, and monitors work assignments; evaluates work performance and provides feedback to organization to accomplish goals; monitors progress and evaluates outcomes. process. others on their performance. Oral Communication - Expresses Political Savvy - Identifies the internal and external politics that impact the information (for example, ideas or facts) to individuals or groups work of the organization. Perceives organizational and political reality and effectively, taking into account the audience and nature of the information acts accordingly. Problem Solving - Identifies problems; determines accuracy and relevance of produces written information, which may include technical material, that is information; uses sound judgment to generate and evaluate alternatives, and to make recommendations. appropriate for the intended audience.

Reasoning - Identifies rules, principles, or relationships that explain facts, data, or other information; analyzes information and makes correct inferences or draws accurate conclusions. Teamwork - Encourages and facilitates cooperation, pride, trust, and group identity; fosters commitment and team spirit; works with others to achieve goals. Technical Competence - Uses knowledge that is acquired through formal training or extensive on-the-job experience to perform one's job; works with, understands, and evaluates technical information related to the job; advises others on technical issues. Technical Credibility - Understands and appropriately applies principles, procedures, requirements, regulations, and policies related to specialized expertise. Writing - Recognizes or uses correct English grammar, punctuation, and spelling; communicates information (for example, facts, ideas, or messages) in a succinct and organized manner; 10 Program Evaluation Competency Model Technical Competency Definitions This section presents definitions for the technical competencies found within the Program Evaluation Competency Model. *Note the competency “Research” is included; see the addendum on research for more details. Data Analysis - Knowledge of the Performance Measurement - principles, methods, and processes Knowledge of the principles and used to analyze data and datasets; methods for evaluating program or including investigating trends or organizational performance using anomalies, establishing relationships, financial and nonfinancial measures, and identifying patterns. including identification of evaluation Evaluation - Knowledge of evaluation theory, concepts, standards, methods, and tools to assess the effectiveness and

factors (for example, workload, personnel requirements), metrics, and outcomes. efficiency of programs, policies, and Project Management - Knowledge of organizations. the principles, methods, or tools for Knowledge Management - Knowledge of the value of collected information and the methods of sharing that information throughout an organization. Organizational Performance Analysis Knowledge of the methods, techniques, and tools used to analyze program, organizational, and mission performance; includes methods that deliver key performance information developing, scheduling, coordinating, and managing projects and resources, including monitoring and inspecting costs, work, and contractor performance. Quality Management - Knowledge of the principles, methods, and tools of quality assurance, quality control, and reliability used to ensure that a project, system, or product fulfills requirements and standards. (for example, comparative, trend, *Research - Knowledge of the scientific diagnostic, root cause, predictive) used principles, methods, and processes to inform decisions, actions, used to conduct a systematic and communications, and accountability objective inquiry; including study systems. design, collection, analysis, and 11 interpretation of data; and the reporting of results. Stakeholder Management - Knowledge of the concepts, practices, and techniques used to identify, engage, influence, and monitor relationships with individuals and groups connected to a work effort; including those actively involved, those who exert influence over the process and its results, and those who have a vested interest in the outcome (positive or negative).

Program Evaluation Competency Model Research Addendum In accordance with Public Law 115-435, Foundations for Evidence-Based Policymaking Act of 2018, the Evidence Act, OPM identified key skills and competencies needed for program evaluation in an agency. A key component of OPM’s governmentwide occupational study was a qualitative and quantitative analysis of program evaluation work. A review of legislation and policy guidance, along with an environmental scan, focus groups, and survey analysis, identified research as an important technical skill for program evaluation positions. While Research was confirmed as an important competency for program evaluation work governmentwide, the study did not confirm Research as an appropriate competency for employee selection on a governmentwide basis. Therefore, Research is not displayed in the Program Evaluation Technical Competencies by Grade Table on page 7. Several factors may contribute to this finding, to include agency differences in mission and in the current application of research methods to perform program evaluation work, as well as variations by agency in when research is needed on the job (e.g., on day one upon entering into a position versus developed over time on-the-job). Agencies may conduct a job analysis to determine whether or not Research is an appropriate competency for selection for their program evaluation work. Please refer to OPM’s Delegated Examining Operations Handbook for more information on conducting a job analysis. Research is presented within the Program Evaluation Technical Competency Definitions Table beginning on page 11. OMB M-19-23, Phase 1 Implementation of the Foundations for Evidence-Based Policymaking Act of 2018: Learning Agendas, Personnel, and Planning Guidance, describes an emphasis on collaboration and coordination to advance data and evidence-building functions in the Federal Government; the Evidence Act statutorily mandates Federal evidence-building activities, open government data, confidential information protection, and statistical efficiency. In addition, the Evidence Act mandates a systematic rethinking of government data management to better facilitate access for evidence-building activities and public consumption. The purpose of program evaluation is to assess the effectiveness or efficiency of a program, policy, organization or aspect of these. Evaluation can look beyond the program, policy, or organizational level to include assessment of particular projects or interventions within a program, for example, or particular aspects of a policy or functions or units within an organization. Evaluations may address questions related to

the implementation or institution of a program, policy, or organization; the effectiveness or impact of specific strategies related to or used by a program, policy, or organization; and/or factors that relate to variability in the effectiveness of a program, policy, or organization or strategies of these. Evaluations can also examine questions related to understanding the contextual factors surrounding a program, as well as how to effectively target specific populations or groups for a particular intervention. They can provide critical information to inform decisions about current and future programming, policies, and organizational operations. Importantly, evaluation can be used for learning and improvement, as well as for accountability purposes. Evidence obtained through program evaluation and the knowledge and skills helpful to program evaluators are described in more detail in OMB M-20-12, Phase 4 Implementation of the Foundations for Evidence-Based Policymaking Act of 2018: Program Evaluation Standards and Practices. In OMB M-20-12, Appendix C (p.16), a list of Evaluation Practices is provided to assist agencies in successfully building and maintaining evaluation capacity, emphasizing that effective program evaluation requires agency personnel with evaluation expertise. In program evaluation, depending on the questions being asked, different types of evaluation can be used such as Formative, Impact, and Process/Implementation Evaluation. Table A.1 in OMB Memorandum, M-21-27: Evidence-Based Policymaking: Learning Agendas and Annual Evaluation Plans lays out a series of evidence-building questions, indicates what type of evaluation may be most appropriate, and suggests methodological approaches. Program evaluators use their expertise to refine evidencebuilding questions, identify the appropriate evaluation type, and select the scientifically-sound method that aligns to that evaluation type and question. To do so, they bring understanding and awareness of a range of methodological approaches, including, but not limited to: pilot projects, ethnography, structured observations, qualitative interviews and focus groups, time studies, statistical analysis of program or participant data, community-based participatory research, randomized controlled trials, regression discontinuity design, propensity score matching, and difference-indifference approaches. As footnoted in M-20-12, p. 4, and in Appendix A: Definitions, “the term “design and methods” is used to collectively address the structure of an evaluation, inclusive of the evaluation approach; variables for, conditions under, timing of, and sources from which data are used or collected; and quantitative and qualitative data collection and analysis methods”. The terms and description of technical knowledge and skills used in 14

Program Evaluation, such as the methods, types of studies, and data generated, align with the competency Research (defined as “Knowledge of the scientific principles, methods, and processes used to conduct a systematic and objective inquiry; including study design, collection, analysis, and interpretation of data; and the reporting of results.”). For example, studies can be quantitative or qualitative in nature and the design and methods address the approach and structure of a study, such as the variables, methods, and sources of the data collected, whether quantitative or qualitative. Such components of research are an important part of building rigor and objectivity, as well as addressing the ethics and utility of a study. Prioritizing employee development in research skills will be integral in strengthening the program evaluation workforce.