EMERGING DIALOGUES IN ASSESSMENT

Building Sustainable Program Assessment Capacity

Academic programs often struggle to establish assessment strategies that produce actionable evidence of student learning while remaining manageable for faculty (Banta et al., 2009; Allen, 2004; Fulcher & Prendergast, 2021). For example, programs may rely on repetitive assessments that fail to generate meaningful data to support faculty in implementing continuous improvement interventions at the frequency required by accreditation and university guidelines. These repetitive structures may also neglect to rotate program learning outcomes (PLOs) regularly and thus not meet institutional and accreditation standards. Ultimately, student needs are unaddressed due to lack of compliance with institutional and accreditation standards.

Another common issue presents when programs frequently update their curricula without changing their assessment strategies to align with programmatic alterations. This can cause confusion for program leadership who are asked consistently to determine continuous improvement actions but may be given assessment data that does not align with newly designed program requirements. Alternatively, program redesigns often invite overly ambitious overhauls of PLO assessment that ultimately fail due to lack of time, faculty cooperation, and resources.

Context

To address common issues in program assessment implementation, the Office of Institutional Effectiveness & Evaluation (OIEE) designed a scalable assessment strategy that could be feasibly adapted to any program. The proposed plan sought to address both systemic challenges in program assessment and limited resources. OIEE manages university-wide assessments at a large, public, decentralized university with the goal of assisting in continuous improvement efforts (Office of Institutional Effectiveness & Evaluation, n.d.). OIEE also provides administrative support for the cloud-based assessment management system used by the university for processes such as annual program assessment documentation, student course evaluations (SCEs), and juried assessment. Given the large number of programs requiring annual program assessment documentation, the strategy prioritized sustainability since OIEE is comprised of a relatively small number of full-time staff members and graduate student assistants. 

Intervention

To mitigate systemic issues in program assessment, OIEE piloted an assessment intervention with an undergraduate program undergoing a formal curriculum redesign with the university’s Center for Teaching Excellence (CTE). This process includes clearly identifying PLOs (Center for Teaching Excellence, n.d.). OIEE collaborated with CTE in developing a transition process to help program leadership—a cohort of administrators and course coordinators directly responsible for the oversight and delivery of the courses included in this assessment—move seamlessly from reshaping curriculum to revising their assessment strategy. Since the curricular changes made the previous assessment plan obsolete, the goal was to integrate new assessment practices into the redesigned curriculum without overwhelming faculty as they implemented program changes. OIEE proposed an assessment structure and designed a process to ease program leadership into completing the process independently.

Proposed Structure
As the program finalized the redesign, OIEE met with program leadership to propose an assessment strategy appropriate for the new curriculum represented in Figure 1. The scaffolded and scalable approach constructed a robust assessment foundation that could expand over time to include additional outcomes. Consistent with best practices, the plan identified both direct and indirect measures of student learning data for one of the program’s new PLOs (Allen, 2004, p. 6), further strengthened by identifying multiple assessment points in the curriculum. Specifically, data was gathered from a newly required 100-level course for incoming students as well as a 200- and 300-level course. This approach capitalized on the program’s new curriculum map indicating that students should achieve higher levels of performance on PLOs as they progress through coursework. Beginning after the redesign, the assessment established two baselines: one from incoming students enrolled in the 100-level course and another from upper-division students who had not taken the introductory course. As a result, the program could compare the PLO performance between the two student groups while building an evidence base for the new curriculum.

Figure 1:  Assessment Structure

Proposed Measures
When program leadership agreed to the proposed structure, OIEE provided additional support by conducting a formal review to identify evidence-based instruments. OIEE proposed two primary measures—SCE questions tailored to the PLO and juried assessment of artifacts (student-produced work). For juried assessment, student-written essays were identified as a direct measure for the PLO. After course coordinators located relevant assignments, OIEE suggested assessing essays with a rubric, as they are widely recognized as a best practice for transparent and consistent assessment, demonstrating value beyond the classroom by helping facilitate ongoing program improvement (Wolf & Stevens, 2007). OIEE conducted a benchmark review of over 15 rubrics from peer institutions and professional organizations. These were synthesized into a comparative table, allowing program leadership to see the landscape of criteria and select the rubric that best fit their learning objectives. In a face-to-face meeting with the program leadership, OIEE presented the researched table and available rubric options. Per OIEE’s recommendation, the program selected the AAC&U VALUE Rubric (n.d). OIEE then collected student essays and facilitated juried assessment through the assessment management system.

OIEE presented SCE questions as a complementary indirect measure to offer insight into students’ overall course satisfaction, course organization, and perceptions of teaching (Centoni & Maruotti, 2021; Stanford University, n.d.). OIEE compiled a list of questions appropriate for the PLO based on a literature review of scholarly articles and peer institutions’ course evaluations; the program ultimately selected five questions addressing the PLO that were added to SCEs at the course level by OIEE. For both measures, OIEE served as an internal consultant, providing evidence-based recommendations that empowered the undergraduate program to make an informed final decision.

Handoff Process  
To support implementation of the new assessment strategy, OIEE developed a scaffolded handoff over a three-year cycle. During the first cycle, OIEE provided hands-on support for the program, including leading assessment meetings, establishing the juried assessment process, scoring student artifacts, reporting findings, and suggesting potential curricular adjustments. Program leadership, including coordinators for the targeted courses, participated in rubric calibrations with OIEE. While OIEE provided assessment scores in the first cycle, the program leadership guided application of the rubric with discipline-specific feedback. Because the program would ultimately be responsible for scoring, the calibrations served both to norm the rubric and to prepare program leadership to conduct future sessions. At the time of this article, the first cycle has been completed.

During the second cycle, OIEE will collaborate with program leadership to build internal capacity and support a sustainable assessment process. In this phase, the undergraduate program will take a more active role, including participating in artifact scoring and incorporating faculty in data analysis. By the third cycle, the program will assume primary responsibility for the assessment strategy, with OIEE providing backup support as needed. This gradual transition allows faculty to become familiar with the assessment structure while minimizing the immediate administrative burden of implementation. As responsibility for the process shifts to the program, the handoff approach supports the long-term sustainability of assessment practices. Unlike one-off consultations with programs, OIEE will be able to monitor whether the plan is successful and sustained since the office administers SCEs, reviews annual program assessment documentation, and sets up juried assessments.

Lessons Learned

Completion of the first cycle yielded lessons that may be useful to other institutions considering a similar approach. OIEE coordinated the implementation by managing logistics and monitoring deadlines, enabling program leadership to focus on content. For example, staff held ad hoc meetings with individual program leaders to answer questions quickly and prevent delays. The program’s dedicated OIEE contact also assisted program leadership in updating their assessment documentation to ensure support from assessment design to reporting. To avoid unnecessary setbacks, future iterations of this pilot will ensure that decisions are made, as much as possible, during meetings rather than over email, which tend to languish in inboxes due to competing priorities.

By starting with one PLO, the process was manageable. Program leadership had time to decide on course-level measures and organize the assignments for OIEE’s collection. Concurrently, OIEE had the capacity to complete the bulk of the assessment in-house. The simple structure (one PLO, one direct measure, one indirect measure) allows program leadership to expand the scope in terms of number of courses, additional PLOs, etc. The structure models an exemplary assessment foundation for the program’s new curriculum that will be expanded over time.

Due to time constraints, OIEE consciously curated assessment recommendations to program leadership based on their updated curriculum map rather than overwhelming them with all possible assessment strategy options. This approach helped program leadership transition into assessment conversations and facilitated a smoother integration of a new assessment strategy. By relying on OIEE staff’s assessment expertise, designing a new assessment strategy only took a few weeks to construct, significantly speeding up implementation. It also supplied the program with an assessment foundation that easily fit into the new curriculum and could be expanded to meet guidelines and, most importantly, student learning needs.

Conclusion

The completion of the first cycle reflects the value of approaching program assessment as a collaborative, iterative process rather than an administrative obligation. By starting with a single PLO, identifying complementary direct and indirect measures, and managing the logistical demands of implementation, OIEE and program leadership were able to establish a meaningful assessment foundation during an already complex period of curricular transition. The lessons learned throughout this first cycle—the importance of disciplined scope, real-time decision-making, and curated evidence-based recommendations—will continue to shape how OIEE supports the program as responsibility gradually shifts through cycles two and three. What began as a pilot intervention has evolved into a sustainable partnership, one that prioritizes both program ownership and long-term assessment capacity. As this work continues, this structured handoff model and lessons learned from the first cycle may serve as a useful framework for other institutional assessment offices and programs seeking a manageable, scalable approach to assessment.

REFERENCES

Allen, M. J. (2004). Assessing academic programs in higher education. Anker Publishing Company.

American Association of Colleges and Universities. (n.d.). VALUE rubrics. https://www.aacu.org/value/rubrics

Banta, T. W., Jones, E. A., & Black, K. E. (2009). Designing effective assessment: Principles and profiles of good practice. Jossey-Bass Publishers.

Center for Teaching Excellence. (n.d.). Program (re)design. https://cte.tamu.edu/services-and-impact/program-redesign.html

Centoni, M., & Maruotti, A. (2021). Students’ evaluation of academic courses: An exploratory analysis to an Italian case study. Studies in Educational Evaluation, 70. https://doi.org/10.1016/j.stueduc.2021.101054

Fulcher, K. H., & Prendergast, C. O. (2021). Improving student learning at scale: A how-to guide for higher education. Stylus Publishing.

Office of Institutional Effectiveness & Evaluation. (n.d.). Office of institutional effectiveness & evaluation. https://assessment.tamu.edu/

Stanford University. (n.d.). Course feedback as a measure of teaching effectiveness. https://evals.stanford.edu/evaluating-teaching/course-feedback-measure-teaching-effectiveness

Wolf, K., & Stevens, E. (2007). The role of rubrics in advancing and assessing student learning. The Journal of Effective Teaching, 7(1), 3-14. https://files.eric.ed.gov/fulltext/EJ1055646.pdf