Rethinking data science education in the age of genAI

Jenny Richmond Ph.D

How do we know what students know when chatGPT can do many of the tasks we have traditionally relied on to assess learning?

Thanks everyone for coming. The elephant in the room in terms of education and generative AI is how do we know what students know when the ChatGPT can do their homework? I’m not promising to answer this question today but I would like to talk about what we as a community are doing about the problem. A recent survey out of the UK showed that 92% of uni students are using genAI (up from 66% in 2024) and 88% admit to using to for assessment. Today I want to focus on what we as educators are doing (and can do) to in response to what is a wicked problem and give you some examples of what I did in a course i used to teach for psychology students about computationl reproducibility.

about me

For those of you who don’t know me my name is Jen Richmond until earlier in this year. I was an academic in school of psychology at UNSW. I am a developmental psychologist by training and my area of expertise learning and memory. And in my last several years I was also Director of Academic Programs in the school which meant that for the last few years wicked problems like COVID and generative AI were a big part of my job.

I’m also the cofounder of RLadies Sydney and the developer are you with Me which is a set of online modules for beginners that have helped 40,000 people get into R.

I actually resigned from my position earlier in the year and I’m currently working out what the next half of my career is gonna look like. I’m doing a little consulting about academic program and assessment reform but I’m looking for a job where I can use my data skills for good so if you know anyone let me know

• ex-academic UNSW Psychology
• co-founder R-Ladies Sydney
  • developer #RYouWithMe
• currently… working out next steps

what are educators doing?

warning students against short cutting the process of learning

Educator is doing. lots of educators are warning students about the risks and then hoping for the best I liked this comic graphic by Nick Susanna from San fran State where the course outline the syllabus is a graphic comic and it will students through the idea that by using it their shortcutting themselves learning is all about mistakes and struggles and surprises and by outsourcing that process to generative AI they’re only losing out on what they’re here for which is learning in class. Data science community have also written interesting things about what they’re doing in their database courses essentially trying to warn students about the dangers or and require them to acknowledge whether they have used it or not.

• learning is about mistakes, struggles, and surprises… own it

See also Prof Andrew Heiss

thinking about assessment change

Preprint from Tiffany Timbers (UBC) and Rohan Alexander (UToronto) interviews with 42 data science educators, questions about attitudes toward LLMs in general and in the context of teaching data science, questions about impact on students attitudes motivations, benefits and risk to learning, changes they have made in response to LLM availability.

preprint

Interviews with 42 data science educators (33 institutions, 10 countries)

Most educators are most concerned about students shortcutting the process of learning not acquiring the skills that are in the curriculum and getting a false sense of how much they actually know.

Lopez-Miranda, Timbers, & Alexander

I thought these ideas about again when asked about how students attitudes towards learning and motivation these educated to say that students are most motivated to speed up the process of finishing their tasks and they are concerned about the value of learning the skills that AI can do relatively easily

Lopez-Miranda, Timbers, & Alexander

the most interesting thing to come out of this paper I thought was answers to the questions of what are you doing about assessment in your course and I have seen These responses in my colleagues as well. The automatic reaction is alright. We’ll just have to assess them in class we’re gonna go back to supervised exams. They will write things on paper. We will reduce the weight of tasks that they do in the presence of AI. Relatively few respondents talked about designing assessments in a way that integrated AI or involved students learnin how to use it responsibly.

Lopez-Miranda, Timbers, & Alexander

why are educators not adapting more quickly?

There seems to be a fair amount of status quo bias going on and this paper by Corbin and colleagues out of the Deacon cradle group which is just across the road is a really interesting read. They also interviewed 20 Deakin academics across faculties about how they were responding to AI and coded the data in terms of Rittel and Webbers 75 wicked problem characteristics. They found that the problem of generative AI and assessment met all 10 characteristics of a “wicked problem”. It is hard to agree on what the problem is, there are an infinite number of responses, no real solution, lots of trade offs, potential responses are all context specific, you don’t know whether changes you made “work”, the stakes are really high. The problem isn’t really “solvable”

This paper made me feel better because it concluded that for all wicked problems, change involves experimenting and being ok with trade offs, it is iterative, and context specific.

• interviews with 20 Deakin academics responsible for unit coordination late 2024
• coded responses in terms of Rittel & Webber (1975) wicked problem characteristics

The challenge of generative AI and assessment meets all 10 of Rittel & Webber’s (1973) criteria for “wickedness” Corbin et al., 2025

my approach to the wicked problem of generative AI

disclaimer: this is example is probably only applicable to my context

So what do you wanna do is tell you about a course that I used to run at Unsw and how I approached starting to make change in light of this wicked problem that we have with generative AI now acknowledge that because this is a wicked problem my response is specific to my context And it’s only one of any number of possible choices I might have made. I don’t know whether it worked I will never know whether it worked but what I want to do today is give you some ideas for how you might integrate the idea of learning to learn into the assessment in your courses in a way that counters some of the risks of generative AI.

learning to learn

No I wanna stop by just talking a little bit about what learning looks like because I think that we assume that students arrive at university knowing how to regulate their own learning and I don’t think that’s true so the Zimmerman model of self regulated learning is one that I really like Because it lays out this process whereby learners set goals and plan what they need to learn and how they might go about that they implement effective strategies given the context of the problem that they’re dealing with. They monitor how things are going and adjust those strategies or ask for help if they need they are reflecting on both the process and the product evaluating their own work and then they set more goals and the process continues and I think as educators This processes Justin our DNA and I think it’s an example of where we might suffer from the curse of knowledge and it is helpful for our students if we embed components of this planning execution metacognition reflection process in learning activities and assessment in a way that models what good learning looks like

Self regulated learners succeed because they

• set goals and plan
• implement effective strategies
• monitor how things are going and seek help if needed
• evaluate their own work
• set more goals…

Zimmerman & Moylan 2009 Self-regulated learning model

Learning (re)design

So the context I wanna tell you about today is a third year research internship course that I used to run at Unsw. This is a course that runs for about 40 students in the last year of the undergraduate program before honours. This is an invite only course and so the students are not average. The course was originally designed to get students hands-on research experience in a lab so students would be assigned a supervisor they would write a research proposal, run a small project, collect data, analyse it, and write it up. That was great until 2020 when students couldn’t be physically in the lab any more so we redesigned the course to centre around compututational reproducibility, giving the students the opporunity to gain similar data skills by working with open data.

And thinking about the changes that we needed to make to assessment in that context it’s important to us two questions first. What do you want Students to learn? What do you want them to be able to do by the end of their time with you? In this context, We wanted students to come to understand the challenge of computational reproducibility, to gain practical skills in visualisation and reproducible reporting and to learn how teamwork works in the real world

• Who: N = ~40 3rd year Psych students, WAM > 75% invited, tracking into honours

WHAT do I want them to learn?

I want students to be able to…

• understand the challenge of computational reproducibility
• visualise data using code
• document their process in a reproducible report
• work in a team

HOW do I want them to learn it?

how might they learn that?

I wanted them to learn together. I wanted to embed those self regulated learning skills of metacognition and self reflection in the process and critically for today I wanted them to learn how to leverage generative responsibly.

I want students to …

• learn together
• engage in metacognition and self reflection
• leverage generative AI responsibly

reproducibility project

Okay so the major assessment in this course is a reproduceability project that is modelled on a new type of registered report article that psychology journals have begun to adopt in response to issues with computational reproducibility - verification report. I think Cortex might’ve been the 1st to start to offer the opportunity for researcher is to write a registered report outlining how they would like to re-examine some already published data.

For our students we leveraged the fact that psychological science which is the flagship APA journal has for a few years now required that all authors make their data open and we assigned each group of students a psych science paper and challenged them to use the open data that was available and their relatively new R skills to reproduce the descriptive statistics and the plots in the paper.

The assessment involved working as a group to produce the code that reproduce the descriptives and plots and then individually submitting a verification report that documented that process in a reproducible way using RMarkdown.

Before they started these projects they got some basic dyplr and ggplot skills under their belt in the first 3 to 4 weeks of the course. In week four they took a Foundations test. And then from week 5 to 8 they worked in groups on their reproducibility project.

In week eight they presented their project to the class and then in week nine and 10 they worked individually on their verification submissions.

Now I think what is key here is that when i say document the process, I really mean the process. Yes it’s important to document code on what it’s doing but in this case we really emphasised the value of writing documentation that outlines your thinking process. And by putting most of the marks for the final submission into our students documented the process of reproducing these plots we were able to make ChatGPT less useful because of course the process that you went through in your head

Task modelled on verification report process being adopted by Psychology journals

• each group assigned a Psychological Science paper
• task to use open data and code to reproduce descriptives and plots

• document the process using RMarkdown

assessment guidelines

course structure

Fortnightly workshops + 2 hour lab each week

Week 1-4 - video modules re data cleaning and visualisation in R & RMarkdown - formative self test exercises

Week 5-8 - group work, reproducibility project

Week 9-10 - individual report write up

can we use GenAI?

yes… but, lets play with it first

So we allow students to use generative AI but to set them up for success and to hopefully minimise the likelihood that they would outsource their learning completely in week five we run a workshop where we gave students the opportunity to play with the tools and just test out how good they were at the kinds of things that they were going to need to do as part of the projects in the coming weeks. Learning how to do new things, documenting your code, debugging errors and adhering to tidyverse style.

In the workshop we assigned each group one of these four tasks And gave them example code and prompts that were modelling how you might use ChatGPT in a way that was gonna enhance your learning. Each group was given one of these tasks and there was us to compare how well two of the AIs did at that task.

They had 45 minutes class in class to play and then each group presented back to class what they had learned about the usefulness of each of these tools for each of these tasks. What was really interesting is that because they were all working together Using the same prompts in the same tour at the same time they saw her in real time the variability in ChatGPT gives you and they very quickly learned that the quality of the prompts you give ChatGPT makes all the difference to the quality of the response you get. The students concluded most of these tools were somewhat useful in learning new things or debugging errors, helping you style your code but for the kind of documentation we were are looking for they were useless (because the thinking process is in your head.)

Week 5 workshop

• learning new things in R
• documenting your code
• debugging errors
• adhering to tidyverse style

Each group is assigned a task and 2 AI tools (chatGPT, Claude, Copilot, Perplexity) to compare. They get 45 min class time to “play” and then report back what they learned.

can we use GenAI?

So in addition to running this workshop as they were embarking on their group work we also required that students acknowledge how they had used generative AI in their final submission. With their final reports they submitted an acknowledgement that owned up to the tools that they had used, the tasks that they had used them for and gave examples of the prompts that they have used.

yes… but acknowledge how in your final submission

AI assistance acknowledgement form

what tasks did they use GenAI for?

Data from this form told us that chatGPT was their favourite ai tool, and most students were using AI to help them debug code, write code and explain code. I was happy that relatively few were using it for writing documentation, which was where I see the evidence of learning and most of the emphasis was in the marking criteria.

Take home message

So my take home message for you today is that generative AI and assessment in education is a wicked problem. If it feels like you’re walking through mud, that’s just the nature of wicked problems and I encourage you to be kind to yourself in this process. Change is gonna be slow and iterative and context specific. There is no solution to this problem. So be kind to yourself in the process.

Generative AI had exposed problems that were always there, and forced us to sit back and ask what do we really want our students to be able to do, and how do we want to them to learn to do that. The answers to these questions might have changed now that GenAI is on the scene.

Once you have a clear idea on what you want students to learn and how, you can more easily see how assessment might provide you evidence of that.

We are dealing with a wicked problem

• it is not solvable
• be kind to yourself

designing learning

• What do you want students to be able to do?
• How do you want them to learn to do that?

• Design assessment that provides you evidence of learning

one more thing…

Read Olivia Guest’s position paper

what are students doing?

This headline is kind of scary 97% of Genzie students cheating their way into university but it’s actually not that far off. What daughter is telling us about the extent of students AI use this plot comes from a piece of research by

guardian

what are students using it for?

the higher education policy Institute in the UK and it’s a survey of more than 1000 undergraduates across disciplines and what’s interesting about the starter is that they have run the same survey over the last couple of years and the increase in the percentage of students that have used ChatGPT is a lot so up in general up from 66% in 2024 to 92 and 2025 in the most latest survey 88% so they have used it for assessment and the percentage of students were saying that they are using to write code is up to 15% from 6%.

HEPI UK survey Feb 2025

why are students using it?

This is not at all surprising to me and it won’t be the educated in the room but what I thought was interesting out of the starter. Is why students are using it And somewhat concerning the top reason that students are using it is to save them time and reasons relating to learning and support and feedback are much further down the list which to the educators in the room is a little bit discouraging

HEPI UK survey Feb 2025

what are universities doing?

some universities are trying to control it

In terms of policy and practice what our university is doing about this problem this is spectrum at my old institution at Unsw. They are still thinking that they can control it. This graphic shows the levels of AI assistance that coordinator course coordinators needed to decide for each piece of assessment which of these six levels they will allow students to use AI and That appears in the course outline for the assessment of course it doesn’t really make any sense to use AI only for planning but not for producing the document and it’s all a bit of a muse because we don’t. These models are getting to the point where we can’t really detect and even if we could detect do we want to spend our time policing AI use

AI@UNSW

others are acknowledging that it is ubiquitous

in contrast the university of Sydney this semester changed their policy such that coordinators are no longer allowed to say that you can’t use it and or penalise you for using it and so Sydney has taken what they’re calling a two lane approach. Assessment is either in lane one where it’s supervised Under conditions so much I cannot be used or it’s in line to in which case all bets are off. This is only come into effect this semester so we’ll see how that goes for them.

AI@USydney