EMERGING DIALOGUES IN ASSESSMENT

• Erica Bender, University of the Pacific

Abstract: Assessment professionals are increasingly using AI in their work and are navigating the AI landscape largely through self-taught experiences. Assessment professionals are self-directed, curious, and looking for frameworks to help them navigate the complexities of AI within the specific demands of assessment. This paper introduces the AI Literacy Compass, a four-direction framework for exploring and developing different dimensions of AI literacy: awareness, application, accountability, and agency.  The AI Literacy Compass can support ongoing, self-directed learning with AI, offering a flexible and non-prescriptive approach to developing AI literacy. Applications of the compass to assessment-specific contexts and questions are also discussed. 

A Compass for Navigating AI Literacy

Recent surveys of assessment professionals show that most respondents identify as regular or occasional users of AI (Slotnick et al., n.d.). The respondents also specify a need for AI training, identify as “self-taught,” and hold a variety of concerns about AI. These survey trends indicate that many assessment professionals are navigating AI as self-directed learners within complex professional and institutional contexts. To continually develop AI literacy, assessment professionals need a framework for orienting their AI-related learning activities. Such a framework would honor the realities of professional self-directed learning, which individuals pursue from different starting points and in response to different circumstances.

Background

The proliferation of AI tools with ever-increasing functionalities has presented assessment professionals with a legitimate concern for assessment validity, while simultaneously enhancing certain aspects of teaching, learning, and assessment. Because there is no universally correct answer for how, when, and why to use AI in assessment, the current AI landscape can constitute a “disorienting dilemma,” a situation that cannot be resolved through simple knowledge acquisition or improving technical competencies (Mezirow, 1991, as cited in DeAngelis, 2022). When viewed from the lens of transformational learning, the disorienting dilemma of AI in assessment can only be resolved through self-examination, encountering new perspectives, exploration, building confidence, and perspective shifts that reorient the assessment professional’s understanding of their work for the AI age.

Despite assessment professionals’ desire for AI training, the dilemma of making contextually appropriate decisions about AI is unlikely to be resolved through formal training alone. While training can support learning how to use AI tools, they are insufficient for learning how to think about AI in unique professional contexts. The dilemma of AI requires a literacy-oriented approach to professional development.

Existing frameworks provide helpful grounding for AI literacy, which entails progressively complex thinking about AI (Ng et al., 2023; Hibbert et al., 2024; Hervieux & Wheatley, 2024) as well as technical proficiencies (e.g., prompting), critical reflection, metacognition, and ethical reasoning (Biagini et al., 2024; Cardon et al., 2023; Chiu et al., 2024; Kassorla et al., 2024). Table 1 synthesizes different AI literacy frameworks into seven prevailing competencies.

Table 1: Prevailing AI Literacy Competencies

Together, these competencies reveal that AI literacy extends well beyond using tools; it also entails perspective shifts in awareness, responsibility, and choosing how we use AI.

How might assessment professionals develop these skills? Like with earlier educational technology dilemmas, professional development in AI must occur in a continuous, situated, and self-directed manner (Kukulska-Hulme, 2012; Webster-Wright, 2010). Indeed, the prevalence of survey respondents who describe themselves as “self-taught” suggests that many assessment professionals already recognize this need (Slotnick et al., n.d.). Adult education practitioners have long emphasized self-directed learning (SDL) as a framework where learners take initiative to understand their needs, develop goals, identify resources, and self-assess (Loeng, 2020). In contrast to learning driven by external authority, SDL emphasizes developing and exercising control over one’s own learning process (Morris, 2024). As higher education professionals collectively adapt to rapidly expanding AI tools, self-directed learning is a necessary strategy.

The AI Literacy Compass

Assessment professionals need a framework that reflects emergent AI literacy competencies while also empowering self-directed learning. The AI Literacy Compass is a metaphor for orienting many different learning activities within emerging AI literacy competencies. The AI Literacy Compass (Figure 1) draws on four interconnected directions that each represent a dimension of AI literacy.

Figure 1: The AI Literacy Compass

The four cardinal directions of the Compass are Awareness, Application, Accountability, and Agency. Awareness entails knowing what AI is, how it works, its limitations, as well as broader critical awareness of the impacts of AI on humans and the environment, the human actors behind AI, and the forces that influence the AI industry. “Application” entails developing techniques for effectively using AI tools to achieve desired outputs through iterative experimentation, as well as recognizing and critically engaging with AI-generated output. Accountability means approaching AI in a way that centers human values and relationships. In addition to properly disclosing AI use, accountability requires awareness of how AI can impact our communities and acting in alignment with the humans we serve. Finally, agency entails developing our own perspectives and retaining control over how and when we use AI. Agency requires maintaining evaluative judgement, recognizing the line between helpful and harmful offloading, avoiding reflexive adoption or avoidance, acting with intention, and practicing self-scrutiny.

As with an actual compass, the cardinal directions of the AI Literacy Compass are interconnected but not hierarchical. No aspect of AI literacy is more “essential” than another and the directions are mutually reinforcing. However, unlike an actual compass, where moving toward one direction also entails moving away from another, the directions of the AI Literacy Compass are not mutually exclusive. The same activity can support learning in multiple dimensions. For example, learning about AI biases and limitations (awareness) can shape how one prompts an AI tool for desired output (application) and surface questions about ethical impacts (accountability). Similarly, cultivating a philosophy of how one wants to use AI (agency) can also sharpen application strategies and invite further learning to deepen awareness.  

Visualizing AI literacy as a compass allows learning to be non-linear and self-directed; developing AI literacy is a process that can be started from any direction and does not follow a singular, prescribed path. Practitioners can also use the Compass to generate coherence from prior learning activities. Those who have already engaged in AI-related professional development can use the compass to find unexplored directions, and those who feel uneasy about AI literacy can use the compass to “begin anywhere” in their self-directed journey.

The AI Literacy Compass and Assessment

While the Compass can be used for general AI literacy, it can also support exploration of AI in assessment. In the awareness dimension, professionals can explore how AI impacts the validity of different assessment techniques, the methodologies behind still-imperfect AI detectors, and how AI disrupts assessment design at the course- and program-level. In the application dimension, professionals can continue to develop their understanding of how to use AI tools in different aspects of the assessment cycle, including developing SLOs and rubrics, assisting with data analysis, communicating results, and facilitating on-demand support through AI agents or custom chatbots.

The application of AI throughout the assessment cycle will likely catalyze necessary and complicated conversations about accountability, especially the relational accountability between faculty and students. How does using AI to create SLOs and rubrics support and/or undermine the teacher-student relationship? To what extent should the assessment office model AI disclosure in their communications? How can the individual assessment professional center their human relationships during AI integration? Developing this dimension of AI literacy will require reflection and relational work to understand and align with the diverse communities we serve.

These activities help to support a sense of agency with AI, shaping not only how we use AI tools in our own work, but how we advocate for AI integration within our institutions. If assessment professionals can think deeply about how AI supports and undermines our assessment philosophies, we will be well positioned to advocate for realistic and balanced AI integration.

Conclusion

Like others in higher education, assessment professionals are navigating AI through self-directed learning. While formal training and resources are available, they are unlikely to provide the well-rounded and assessment-specific AI literacy competencies that we seek. We should feel empowered to self-direct our own learning. The AI Literacy Compass supports this learning and helps to mitigate some common SDL challenges: managing information overload, creating structure, and sustaining motivation.

Communities of practice can also leverage the Compass to shape and support their collective learning about AI. Transformational learning rarely happens in isolation; peer engagement is key for sustaining and validating self-directed learning. One of the survey respondents spotlighted by Slotnick and colleagues (n.d.) emphasized the need for “a community of others” in this work. A shared framework for AI literacy can help assessment professionals transform our individual wayfinding into shared navigation, allowing us to collectively shape how the assessment profession copes with the disorienting dilemma of AI. In a moment where assessment professionals feel pressure from multiple directions, the AI Literacy Compass offers a simple invitation: begin anywhere, orient often, go toward the unfamiliar, and trust the journey.

REFERENCES

Biagini, G., Cuomo, S., & Renieri, M. (2024). Less knowledge, more trust? Exploring potentially uncritical attitudes toward AI in higher education. Italian Journal of Educational Technology, 32(1), 37-52. https://doi.org/10.17471/2499-4324/1337 

Cardon, P. Fleischmann, C., Aritz, J., Logemann, M., & Heidewald, J. (2023). The challenges and opportunities of AI-assisted writing: Developing AI literacy for the AI age. Business and Professional Communication Quarterly, 86(3), 257-295. https://doi.org/10.1177/23294906231176517  

DeAngelis, L. (2022). Enabling the exploration of disorienting dilemma in the classroom. Journal of Education, 202(4), 585-600. https://doi.org/10.1177/0022057421991865

Hervieux, S., & Wheatley, A. (2024). Choice White Paper: Building an AI Literacy Framework: Perspectives from Instruction Librarians and Current Information Literacy Tools. https://www.choice360.org/wp-content/uploads/2024/08/TaylorFrancis_whitepaper_08.28.24_final.pdf 

Kassorla, M., Georgieva, M., & Papini, A. (2024). AI Literacy in teaching and learning: A durable framework for higher education. EDUCAUSE https://www.educause.edu/content/2024/ai-literacy-in-teaching-and-learning/defining-ai-literacy-for-higher-education  

Kukulska-Hulme, A. (2012). How should the higher education workforce adapt to advancements in technology for teaching and learning? The Internet and Higher Education, 15(4), 247-254. https://doi.org/10.1016/j.iheduc.2011.12.002

Loeng, S. (2020). Self-directed learning: A core concept in adult education. Education Research International, 2020, 3816132. https://doi.org/10.1155/2020/3816132

Morris, T.H. (2024). Four dimensions of self-directed learning: A fundamental meta-competence in a changing world. Adult Education Quarterly, 74(3), 236-254. https://doi.org/10.1177/07417136231217453

Ng, D.T.K., Wu, W., Leung, J.K.L., Chiu, T.K.F., & Chu, S.K.W. (2023). Design and validation of the AI literacy questionnaire: The affective, behavioral, cognitive and ethical approach. British Journal of Educational Technology, 55, 1082-1104. https://doi.org/10.1111/bjet.13411 

Slotnick, R., Boeing, J., Pinnelli, B.G., Bao, Y., Hathcoat, J., Miller, W., & Wells, N. (2025). GenAI Survey for Higher Education Assessment Professionals – Fall 2025 Survey Results. Bridgewater State University. https://genaiassessment.neocities.org/Fall2025_Results%20Snapshot

Webster-Wright, A. (2010). Authentic Professional Learning: Making a Difference through Learning at Work. Springer. https://doi.org/10.1007/978-90-481-3947-7