In the We’re Only Human (WOH) project, we are interested in how well learning materials that faculty incorporated into their courses helped students to engage ethically, safely, and critically with generative AI (e.g., ChatGPT, Google Gemini, etc.). We are also interested if students gained a better understanding about if/when generative AI can be used to support or enhance their academic writing.
When incorporating new learning materials into a course, as with other pedagogically informed assessments, it is good practice to evaluate whether or not the assignment was successful in achieving desired learning outcomes. The information gained through this evaluation can then aid in evidence based changes to the assignment, course components, or pedagogical approach one is taking. In this project, we used a mixed methods approach that incoproated both qualitative and quantitatve data collection strategies to provide a deeper understanding of students’ experiences with generative AI.
Evaluation Tools
The student survey took approximately 10 minutes to complete and collects the following information:
- Demographic information.
- How students are using AI use in their personal, professional, and academic life.
- How learning materials have helped students to more ethically, safely, and critically engage with generative AI.
- How learning materials can be improved.
Please visit our sample survey for a complete list of questions.
Focus groups of 3-5 students were recruited from each course taking part in this study, and lasted approximately 45 minutes. The following questions were asked of students:
Prior Use and Understanding
- Can you describe your level of familiarity with AI prior to this course? How did you use it and in what contexts?
- Prior to this course, did you have a clear understanding of when it was or was not ok to use AI to support your academic writing at UBC?
Awareness and Understanding
- Did the learning activities in this course help you to better understand when and how it is appropriate to use AI in your academic writing? How so?
- Follow-Up Question: Did you gain a better understanding of how AI is used, or is not used, in your field?
- Follow-Up Question: Did you gain a better understanding of how to disclose your use of AI through notes or citations?
- Follow-Up Question: Did you learn techniques to help you discern the accuracy of AI-generated content or how to consider the bias present in such text?
Enagement and Learning Environment
- Did you feel like you could talk to your instructor or TA about your use of AI more openly after these learning activities?
- Did the AI activity that you took part in during this course result in you being more interested in Generative AI and how it can be used to help support your academic writing?
Persistence and Scaffolding
- Do you think the lessons learned through these AI activities will help you better engage with such technology in your future courses?
- Follow-Up Question: Do you think you’ll be more likely to discuss your use of AI with your instructors or ask questions about acceptable use in your other courses?
Learning Activity Design
- Was there anything we could do to improve the learning activities that you took part in? Was there anything we could do to improve the learning activities that you took part in?
- Do you have any recommendations about how we could help other students gain a better understanding of Generative AI and writing, beyond the activities you took part in during this class?
To gain a broader understanding of the use of AI learning materials in each course, and how these might fit into the pedagogical frameworks being implemented in UBC classrooms, we also conducted one-hour interviews with each faculty member taking part in this project. The following questions were asked of instructors:
Course Information and Context
- Please briefly summarize the course: topic, level, number of students, place in unit curriculum etc.
- What role does writing play in this course?
- What course policies do you have in place related to Generative AI?
Pedagogical Model/Approach
- What motivated you to incorporate Generative Ai-focused learning activities into your course?
- How do you see Generative AI fitting into your overall teaching philosophy, especially when it comes to teaching writing and critical thinking in Arts courses?
- How did you integrate these activities into the overall structure of your course?
Student Experience and Impact
- How do you think students responded to these learning activities? Were there any noticeable changes in their engagement, attitudes, or perceptions of AI? What specifically did you notice that suggested that change?
- Did students express any concerns or misconceptions about Generative AI and academic writing? How did you address those?
- Did you find students were more open to discussing Generative AI with you following the activities?
Learning Activity Design and Reflection
- Would you make any changes to the activities you trialed with your class? If so, what and why?
- Are there other kinds of activities you are now thinking about to extend or support students’ learning about GAI and/in relation to writing?
- What advice would you offer to other instructors considering incorporating Generative AI-focused activities into their writing assignments?
Please note that when assessing pedagogy in one’s own course for the purpose of improving course design and learning outcomes, you do not need behavioral ethics approval to collect information at UBC. However, one must still collect, store, and use this information in an ethical manner!