Cyclical Program Reviews - Graduate Programs

Western’s protocol for Cyclical Program Reviews has five principal components:

1. Self-Study;
2. Evaluations (including site-visit) with a report, and internal responses from the academic unit and Dean to the report;
3. Institutional Evaluation;
4. Recommendations;
5. Ongoing Monitoring - Annual Planning Process.

Schedule of Reviews

The schedule is designed to allow the undergraduate and graduate programs within an academic unit to be reviewed concurrently; however, although the reviews will occur concurrently, they will normally undergo separate review processes with different external consultants.

The review schedule includes all collaborative, joint, and interdisciplinary programs. In addition, the programs offered by Western’s affiliated university colleges are included in the schedule. Joint programs that involve more than one institution will identify a lead institution to prepare the self-study document, consulting and obtaining relevant input from all participating institutions.

Protocol for Cyclical Program Reviews

The Provost and Vice-President (Academic) is responsible for Cyclical Program Reviews and for reporting their outcomes to the QC. In the review of graduate programs, the Provost is supported by the Vice-Provost (Graduate and Postdoctoral Studies). The Committee on Program Review advises the Provost on all matters related to graduate program review.

Steps / Timelines

1. The self-study brief is developed by the program with support from the School of Graduate and Postdoctoral Studies (for graduate programs).
2. The brief is received by SUPR-G; SUPR-G appoints internal reviewers and external consultant(s) to review the brief and conduct a site visit. The external consultant(s) submits a written report of the review; the internal reviewers prepare a summary report of the review for SUPR-G.
3. On the basis of the external consultants’ report, the academic unit’s response to the report, and the internal reviewers’ summary, SUPR-G submits a final assessment report to SCAPA (and shares this report with the program and Dean). This report includes acknowledgement of program innovations and recommendations for program improvements.
4. SCAPA reviews the report of SUPR-G and makes an executive report to Senate.
5. The Provost, through the Vice-Provosts, ensures that recommendations for improving the program and a plan for their implementation are shared with the Dean of the program’s Faculty.
6. Provost’s Office includes the outcome of the cyclical review in the annual report to the Quality Council.
7. Implementation of the recommended improvements is monitored by the University through the Annual Planning Process.

Self-Study

The self-study will comprise a broad, reflective, critical and forward-looking analysis of the program. It will reflect the involvement and consultation of faculty, staff and students of the program being reviewed and it will include data on university recognized indicators. In large part, these data will be provided by, or corroborated by, the School of Graduate and Postdoctoral Studies.

The self-study document will address:

• Objectives of the program;
• Program regulations;
• Consistency of the program’s learning outcomes with the University’s mission and with degree level expectations, and how the program’s graduates achieve those outcomes;
• Fields of specialization (for graduate programs with fields);
• Special matters and/or innovative features of the program;
• Concerns or matters raised in the previous review of the program;
• Program-related data and measures of performance, where applicable and available;
• Financial support for students (as applicable);
• Areas for improvement identified through the self-study;
• Opportunities for enhancement;
• Academic services and resources that contribute to the academic quality of the program, including library resources and support;
• Enrolments, graduations, and withdrawals;
• Employment or subsequent academic pursuits of graduates;
• Publications of current students and recent graduates;
• How faculty, staff, and students were included in the self-study;
• Indicators relevant to the evaluation criteria;
• The integrity of the data included.

Where appropriate, input of others deemed to be relevant may be included in the self-study brief. For example, input from graduates of the program, professionals, industry representatives, and employers may be included. The Vice-Provost (Graduate and Postdoctoral Studies), or his/her delegate, will review and approve the self-study report for graduate programs undergoing cyclical review.

Evaluation

The Review Team

The evaluation will include internal and external reviewers. The review team will normally include:

• one faculty member internal to Western, but not a member of the academic unit under review;
• one graduate student who is not from the program being reviewed;
• two faculty members external to Western.

The faculty member internal to Western and the student comprise the internal reviewers. The Chair of SUPR-G may invite additional discretionary members of the Review Team if circumstances warrant.

All members of the review team will be at “arm’s length” from the program under review. Internal reviewers will not be from the program being reviewed. Additional conflicts of interest may include family ties, partnership ties, supervisory relations or other types of relationships with individuals in the program being reviewed. Any such relationships must be declared to determine the potential for conflict of interest. The Chair of SUPR-G, in consultation with the Provost, will evaluate the potential for conflict of interest. External consultants will normally be associate or full professors, preferably with some program administration experience, and must be at “arms length” from the program under review. “Arms length” reviewers have no family ties, partnership links, supervisory relationships or other relationships with anyone in the program being reviewed. A conflict of interest would exist in cases where the proposed consultant has collaborated or published with a member of the program within the past 7 years, has an administrative or family link with a member of the program being reviewed, has been a supervisor or supervisee (graduate or postdoctoral) of a member of the program being reviewed within the past 7 years, is a former member of the program being reviewed, is a friend of a member of the program being reviewed, or has been a recent (within the past 5 years) visiting professor in the program being reviewed.

The Chair of SUPR-G will appoint the internal reviewers. The faculty member internal reviewer will be a member of SUPR-G. Student members of the review teams will be selected from a list of student volunteers and student members of SUPR-G.

The Chair of SUPR-G, in consultation with the Vice-Provost (Graduate and Postdoctoral Studies), will select the external consultants from the list of potential consultants provided by the program.

All members of the Review Team will receive the program’s self-study. In addition, they will be provided with a volume containing the CVs of all of the full-time faculty members in the program under review.

The Chair of SUPR-G has the responsibility to ensure that the Review Team will:

• understand it role and obligations;
• identify and commend the program’s notably strong and creative attributes;
• describe the program’s strengths, areas for improvement, and opportunities for enhancement;
• recommend specific steps to be taken to improve the program, distinguishing between those that the program can itself take and those that require external action;
• recognize the University’s autonomy to determine priorities for funding, space, and faculty allocation;
• respect the confidentiality required for all aspects of the review process.

These expectations will be shared with the Review Team in the form of written instructions and through face-to-face meetings.

The Site Visit

The site visit will be arranged by the Office of the Vice-Provost (Graduate and Postdoctoral Studies) in collaboration with the program. The visit will normally be for two days and the internal reviewers will participate with the external consultants in all aspects of the site visit. The visit will include meetings with:

• the Vice-Provost (Graduate & Postdoctoral Studies) and/or the Associate Vice-Provost (Graduate & Postdoctoral Studies), at the beginning of the site visit and again at the end of the site visit
• the Vice-Provost (Academic Planning, Policy and Faculty)
• the Vice-President (Research & International Relations)
• the University Librarian and/or Assistant/Associate University Librarian
• the Dean and/or Associate Dean of the program undergoing review
• the Graduate Chair of the program undergoing review
• the Department/School/Centre Chair or Director of the program undergoing review
• faculty members of the program undergoing review
• graduate students of the program undergoing review
• support staff of the program undergoing review.

The review team will be free to seek information from other sources and to suggest other individuals and groups with whom to meet during the site visit.

The Report of the External Consultant and the Internal Reviewers’ Summary

The external consultants will normally provide a joint report appraising the standards and quality of the program and addressing the evaluation criteria in Section 4.3. The consultants will be instructed to submit their report to the Chair of SUPR-G within two weeks of the on-site visit.

The report of the external consultant(s) will be shared with the relevant Dean(s) and unit/program Chair(s) or Director(s) and their response to the report will be requested. In addition, the report will be shared with the Vice-Provost (Graduate and Postdoctoral Studies), who may provide a written response. The response of the Dean(s) and/or Chair(s)/Director(s) will comment on:

• the plans and recommendations proposed in the self-study;
• the recommendations advanced in the report of the external consultant(s);
• the program’s response to the report of the external consultant(s).

and will describe:

• any changes in organization, policy or governance that would be necessary to
  meet the recommendations;
• the resources, financial and otherwise, that would be provided in supporting the
  implementation of selected recommendations; and
• a proposed timeline for the implementation of any of those recommendations.

The internal reviewers will prepare a summary of the onsite visit, the report of the external consultant(s), and the response(s) to the report; this summary will constitute a draft of the final assessment report that SUPR-G will submit to SCAPA. SUPR-G will receive the summary, in addition to the report of the external consultant(s) and the response(s) to the report. The internal reviewers’ summary will:

• identify significant strengths of the program;
• identify opportunities for program improvement and enhancement;
• prioritize recommendations for implementation; and
• include an executive summary suitable for publication on the University’s website (the report may also contain a confidential section).

Report to SCAPA and Senate

SUPR-G are subcommittees of Western’s Senate; they report directly to the Senate Committee on Academic Policy and Awards (SCAPA). SUPR-G will review the report of the external consultant(s), the response(s) to the report, and the summary of the internal reviewers. SUPR-U / SUPR-G may consult with the Vice-Provost (Graduate and Postdoctoral Studies), or the Provost in its evaluation of a program’s review. SUPR-G will recommend to SCAPA a final assessment report that identifies:

• recommendations to be addressed;
• who is responsible for acting on the recommendations;
• what resources are implicated in the recommendations and who has responsibility for these resources; and
• the timeline for implementing recommendations.

SCAPA will review the final assessment report from SUPR-G. SCAPA may seek clarification or additional information from SUPR-G prior to acceptance of the report. The final assessment report, exclusive of any confidential information, will be provided to the program and to the Dean(s) responsible for the program. A copy of the final assessment report will also be sent to the Quality Council. Implementation of the recommendation included in the report will be monitored through the Faculty Annual Planning Process where Deans will be required to report on steps taken to address the recommendations in the final report. SCAPA will report an executive summary of the final assessment and recommendations to Senate.

Following Senate’s receipt of the executive summary of the final assessment, the University Secretatiat’s office will post the executive summary of the review on the University’s webpage. Implementation of the recommendations resulting from the review will be monitored through the Annual Planning Process.

The Provost, in consultation with the University Secretariat, the Vice-Provost (Graduate and Postdoctoral Studies), and the Faculty Deans, will determine to what extent the public will have access to:

• the information made available for the self-study;
• the self-study report;
• the report of the external consultant(s);
• the responses to the report of the external consultant(s); and
• the summary of the internal reviewers.

Annual Report to the Quality Council

Western will provide an annual report to the QC that includes the executive summary of the final assessment for all cyclical program reviews conducted during the year, as well as all major modifications approved by Senate during the year.

Accreditation Reviews

Cyclical program reviews may be scheduled to coincide with accreditation reviews. The normal period between reviews may be shortened to allow a program’s cyclical review to coincide with an accreditation review; however, synchronization of the cyclical review and accreditation review will only be permitted in cases where the maximum period between cyclical reviews does not exceed 8 years.

Although cyclical program reviews may be scheduled to coincide with accreditation reviews, accreditation reviews will not take the place of cyclical reviews. A cyclical program review will normally be conducted in addition to the accreditation review to ensure full consideration of all aspects of the cyclical review.

Evaluation Criteria

Objectives

• consistency of the program with Western’s mission, values, strategic priorities, and
  academic plans;
• clarity and appropriateness of the program’s requirements and associated learning outcomes in relation to the undergraduate degree level expectations or the graduate level degree expectations.

Admission Requirements

• admission requirements are appropriately aligned with the learning outcomes established for completion of the program.

Program Structure and Curriculum

• how the curriculum addresses the current state of the discipline or field of study;
• identification of any unique curriculum or program innovations or creative components;
• mode(s) of delivery to meet the program’s identified learning outcomes are appropriate and effective;
• for research-focused graduate programs, evidence that the nature of the major research requirement is appropriate;
• evidence that at least two thirds of the course requirements are graduate level;
• evidence that the program length ensures that the program requirements are reasonably met within the expected time period (with a maximum of 6 terms for master’s programs and 12 terms for doctoral programs).

Assessment of Teaching and Learning

• evidence that the methods for assessing student achievement of the learning outcomes are appropriate and effective;
• evidence of the appropriateness and effectiveness of the methods of teaching and assessment in demonstrating achievement of the program learning objectives and the degree level expectations.

Resources for All Programs

• adequacy of the academic unit’s human, physical, and financial resources to the support the program;
• participation of a sufficient number and quality of faculty who are competent to teach and/or supervise in the program;
• evidence that resources adequately support the quality of scholarship and research activity expected of the undergraduate or graduate students, including:
  • library resources and support,
  • information technology,
  • laboratory resources and access.

Resources for Graduate Programs

• evidence that faculty have the scholarly/research or professional/clinical expertise needed to sustain the program, promote innovation, and foster an intellectual climate;
• for research-based graduate programs, evidence that financial support for students is sufficient to ensure adequate quality and numbers of students;
• evidence of appropriate instruction and supervisory qualifications and capacity;
• evidence that the program structure and curriculum supports timely completion.

Quality and Other Indicators

In addition to the evaluation criteria above, the reviews should include relevant information (as available) regarding:

• Faculty: qualifications; research and scholarly record; honours and awards; class sizes; proportion of classes taught by full-time faculty; commitment to student mentoring (graduate programs);
• Program: evidence of a program structure and faculty research that will ensure the intellectual quality of the student experience;
• Students: applications and registrations; success rates in provincial and national scholarship competitions and awards; academic awards; rates and timing of attrition; final-year academic achievement; time-to-completion; graduation rates; scholarly output (graduate programs); time to completion (graduate programs); student in-course reports on teaching; and
• Graduates: graduation rates; employment and post-graduate study; “skills match” and alumni reports on program quality when available and when permitted by FIPPA.

Quality Enhancement

Initiatives that have been implemented to improve the quality of the program and the associated learning outcomes and teaching environment.