AI In The Classroom Revisited
Students might not be doomed by AI. I see them becoming “prompt engineers”, structuring their writing with detailed instructions for the AI. Students are now using a multi-step writing process that produces well thought through essays even when fluency is difficult.
Please scroll down for links below the transcript. This is lightly edited AI generated transcript and there may be errors.
Dr. Josh Stout 0:00
The fear about having something else do the writing is, writing is thinking. But now what I'm seeing is, yeah, writing is thinking, but now it's sort of at a meta level. It's the next level up of the thinking. And I'm still seeing them have to organize their thoughts and put it into writing. But it's now at a different level and it's I am enjoying it a lot.
Eric 0:28
Today is Friday, September 6th. We're trying to get back on to our schedule of Fridays. Hey, Josh, how you doing?
Dr. Josh Stout 0:34
I'm doing well. We might have to run if the food arrives, but otherwise we're good to go. I've just been through my first couple of weeks of the 2024 fall semester, and I wanted to give an update on the AI use in the classroom and how that is doing. I, I had started out reading about AI online, of course, and really enthusiastic about how this was going to work, but also a little scared that it was going to end any kind of homework assignment that involved writing.
Eric 1:12
And yeah, it.
Dr. Josh Stout 1:13
Seems like that actually could be somewhat happening. So I came up with lots and lots of rules and said, you know, if you want to use A.I., you can you can just give us the prompts. And if you show the prompts, then I'll know what to do.
Eric 1:32
This is what you said - you’ve said that before.
Dr. Josh Stout 1:33
I said, they could do it. And their first response was just never using A.I.. And everything was great, except a couple of them had some strange language and I ran it through the AI detector. Now, A.I. detectors are problematic. Why? They're problematic for a couple of reasons. They were great for me. Like if they can tell it just to be my text. And if I take that text and run it through A.I., it'll show it every time. But A.I. looks for, so AI detectors look for patterns of various kinds. One is just forensics. If you've got a lot of cut and paste, they can notice that and see that you've cut and paste from more than one source. And so they suspect you. If if they see a lot of that. Another one is it's called bursting or it's measuring the the entropy, but the amount of strange language you use, humans never stay on a single path, whereas AI tends to always use the same format in its language. And so A.I. has a much.
Eric 2:33
Because that's the higher statistical probability of the way language would be used, right? In it’s opinion.
Dr. Josh Stout 2:39
So they have a very low entropy. They have a lot less randomness with what's called bursting. Interesting. And then there is there's another one where it just looks for particular words. So if you want to do an analysis of of papers that have been using AI, you look for words that AI really likes, like meticulous. If if a study has been meticulous, there's a really good chance that they used A.I. to write that. Okay. And there's a lot of like keywords like that just suddenly have gone up hugely in all scientific papers, everything from students, everything.
Eric 3:11
Okay.
Dr. Josh Stout 3:12
And so even if you're not finding it, you're finding these words that AI is providing the word..
Eric 3:17
I like that word. I have to stop using that word now?
Dr. Josh Stout 3:18
No, you don't have to stop using it. It's just an example. There's there's a whole bunch of them that are like that, that are just sort of a little extra flowery and not the sort of thing, you know, I would carefully measure. I would meticulously measure. It's just why why go there? Yeah. So that's one of the ways. And so what happens is that students start using Grammarly or one of the other programs that uses AI to fix phrases, right, and make them more grammatically correct.
Eric 3:51
I also think you've mentioned that you have a high number of or a high percentage of ESL students.
Dr. Josh Stout 3:58
Well, so if English as a second language is a problem for a graduate writing class, there are very likely to be using Grammarly to fix things. Right? Because you know, there's subject verb agreement, you know, just little problems that are very common. Right. Because, you know, English is hard.
Eric 4:14
And I'm not saying that's necessarily wrong. I'm saying this is where you would go.
Dr. Josh Stout 4:18
It's yeah, yeah, yeah. And so I was starting to see that the AI detectors were coming up with people being 60% human, and I'm like 60% human. Most humans come out as 100% human. What is this, 60% human? And so I contacted them to see if they were cutting and pasting something from an AI into their work or they were refer raising AI. And finally, I got someone to confess basically that they were using grammarly and I realized that Grammarly was itself AI. And so this is also particularly true with the ESL students because they're less likely to be straying outside of a standard in English already.
Eric 5:01
But Grammarly existed before this.
Dr. Josh Stout 5:03
AI Yes, and then it then then Grammarly became AI supported and it actually changed the way Grammarly works, right? Yeah. And so Grammarly now works much, much better, but it now rewrites your paper as AI and…
Eric 5:22
So they don't need to go to Grammarly and then go to AI to clean up the phraseology.
Dr. Josh Stout 5:27
You know, the phraseology.
Eric 5:28
Now just does the whole thing for you.
Dr. Josh Stout 5:30
Yeah, exactly. So I said, so I've now said, if you're using Grammarly, just tell me then I'll know what I'm seeing. If you're using Chat GPT, give me all the prompts and I'm seeing actually really creative use of this and students who I don't think could do necessarily a graduate level writing class in biology are now able to do it and do it well. And they're using AI as a tool and they're showing me exactly the prompts that they’re using. And I am very hopeful about this.
Eric 5:59
So this is actually working because before you said this.… So the difference here from the last time we talked about this is you said specifically when you said, show me the prompts. They're like, that's too much work.
Dr. Josh Stout 6:10
That's too much work. Well, so so the first…
Eric 6:11
But now it’s not?
Dr. Josh Stout 6:12
Not. Yeah, well, the first batch said that's too much work and didn't use AI. Then they started to use AI but not tell me and tried to hide it from me. At which point I really kind of feel that's approaching cheating. And they're trying to fool me and I don't like that. And so that made me upset.
Eric 6:30
So what changed it on their side that they're now giving you the prompts?
Dr. Josh Stout 6:34
They understand how to use it better. So they've been introduced to it already. They they've known about it for a year or two. Okay. I made the threats really clear in bold and underline is if you don't tell me that is cheating.
Eric 6:47
If you don't tell me and then I figure it out, which I will.
Dr. Josh Stout 6:49
Which I will, I run these things through checkers and I do find these things and I wrote a letter to them all saying I had a problem with this last semester. There were people who were using this and I had to give them zeros on their paper and they're all like, Ooh, ooh. We don't want that to happen.
Eric 7:06
So you it's so this is you address this head on at the beginning of the semester?
Dr. Josh Stout 7:10
Absolutely. So that they would know what the downside was. And then I said, if you use it, just tell me what you did.
Eric 7:19
Using it is not bad. Not telling me is bad.
Dr. Josh Stout 7:23
Exactly. And so now people are able to write who I don't think would have been able to write before. And they're it's it's I could not have gotten a PhD without spell check on my computer.
Eric 7:35
Yeah you said this this is so we were coming like full circle.
Dr. Josh Stout 7:37
We're coming full circle. And the teachers were worried that none of us would learn how to spell if we had spell check. And that's somewhat true. But to a certain extent, the people who are going to learn how to spell, they learn how to spell anyway. And people like me who are never going to learn, we can survive. And I think it's somewhat true when you're when you're getting through graduate school and English is not your first language and you're working at how do I get my ideas across? Having a AI seems like a very useful tool.
Eric 8:09
I completely agree. But a student needs to be introduced to it and coached through it and needs to be taught how to use it.
Dr. Josh Stout 8:19
That's why I was very, very clear on what I wanted. You know, just give me the prompts and I can see what you asked for and I can see how it was if you just tell me what you did. We're good.
Eric 8:33
So, so…
Dr. Josh Stout 8:34
And they can cut and paste it. They don't have to write anything more. And so I see students who hand in a two page paper and they have like a one page prompt for it that they hand wrote or not handwrote, but you know, they wrote themselves, instructing the A.I. what to do with the paper they'd already written. So they took it, They wrote a paper, and then they gave me the prompt for what they wanted the AI to do to the paper. And I'm seeing really intelligent work and I'm reading their prompt and I see that their English is not great, but they are fantastic.
Eric 9:06
So. So I'm sorry. So they've written a paper in there, however fractured English, right? And then they go ahead and they write a significant prompt.
Dr. Josh Stout 9:16
Significantly because it has to be detailed telling the AI exactly I want, I want you to condense this so I can I already reading that I know that their original paper was longer than what they turned in.
Eric 9:26
What other what other things are you finding in their prompts? I'm fascinated.
Dr. Josh Stout 9:30
They they they say, Please, please read this for grammar and phrasing. Please make it more succinct and make it read like a professional scientific paper.
Eric 9:43
I'm just I'm. I mean, I know and it works. I know that this is what it does, but I'm blown away that you can write a paragraph worth of instructions to the AI in for what it needs to do with the other five paragraphs to ten paragraphs you've written and you can take all of that into account. Yeah, Yeah, of course.
Dr. Josh Stout 10:04
Well, we also have to be really, really careful. The dean doesn't even want us using A.I. detectors because they're fallibility. I think mostly it's the problem with a racism accusation because people who are not native speakers of English are more likely to show up as being AI supported.
Eric 10:25
Well, because they are.
Dr. Josh Stout 10:27
Often they are. But even if they're not there, English is more standardized. And so even so, even if they're fully rewriting something. So they they got an AI.
Eric 10:37
Because by definition, they do not have idiomatic English.
Dr. Josh Stout 10:40
Exactly. So if they start with an AI and then they rewrite it, they they don't get far enough away from the AI and they're rewriting. And so no one wants this to go to any kind of great appeal.
Eric 10:50
But but, but the whole point of this is that there always has to be the human, the human input, not just human input, but the human direction of what the output is going to be, even on the teacher's side, even on the professors side, if you have to use the AI detector because they're going to use the A.I., But that doesn't mean that you just let the AI make the final determination of what’s happening.
Dr. Josh Stout 11:16
No, I would actually like it if it were always on just like safe assign is always on, you know, if one of my students comes back with a safe assign that is 40% copied, that's actually fine because they're using an original paper and then writing a review of that paper. It's going to sound very similar. If I see it's 80% copied, then I'm going to look into it and try and figure out.
Eric 11:35
But here, But that's what I'm saying, is that you're not going to be like, That's it. 80%. That's the threshold…
Dr. Josh Stout 11:43
Then I asked them to come in.
Eric 11:44
You're going to look into it.
Dr. Josh Stout 11:46
That's exactly what. So the threshold for AI for me is around 70%, not non-human. If you're only 30% human, you didn't rewrite much. And I need to talk to you.
Eric 11:57
And you need to talk to the person about it. You're not going to come in saying you have been determined to have not written this paper. “We need to talk” is what’s gonna happen.
Dr. Josh Stout 12:06
But I also get 100% AI.
Eric 12:09
Well - I don't have any - I don't know what to say about that.
Dr. Josh Stout 12:12
And so and that case, I say you have a zero. Now, I have been letting students rewrite and I think I shouldn't. And I need I need to chop off a few heads. But I think saying that I would so far seems to be doing the job because I you know, I make it I make it easy in one direction and dangerous in the other.
Eric 12:32
Well, you know, it's interesting because I have…
Dr. Josh Stout 12:34
The other option is they can just write. And that's good for them, too.
Eric 12:37
I have other I have other professor friends who are at other universities and struggling now with, you know, not in the sciences, but in the humanities, struggling with, you know, how do we let them use it at all? Because, you know, it's one thing if you've got ESL students and you're dealing with sciences, but if you've got native speakers or even people who are studying other languages, that's what they're doing. They're studying other languages, and then they're supposed to write something and then they use the AI. At what point? Now it's interesting because it's it's, it's a little bit easier where you are. It gets it gets a lot murkier as you leave.
Dr. Josh Stout 13:16
Oh, there's there's murky in every direction. I know engineering students who graduated with an engineering degree because they got a a I to write their thesis.
Eric 13:27
Okay that's.
Dr. Josh Stout 13:27
That's engineers writing people who build bridges and things like.
Eric 13:30
No, I don't want to hear that.
Dr. Josh Stout 13:33
You know, this is this is this is happening to the students who fool me and get through and are good at this, you know, could well become doctors. You know, this this is one of the fears. You know, we do have a fear in general of the loss of knowledge caused by AI. As A.I. starts copying itself, I now have a student for the first time whose job it is to actually look for his hallucinations. And I need to talk to her some more about details. But it's wonderful.
Eric 14:01
You do. But but. But the loss of jobs, too. I mean, the loss of knowledge to AI. What about the loss of you know, this is what Jed is talking about. What about the loss of desire for knowledge, the loss of desire to work for knowledge that is the loss of any hope of remuneration for your work, for knowledge.
Dr. Josh Stout 14:24
That's I don't I don't think that's an AI thing. Well, what, what what I'm particularly worried about with AI…
Eric 14:29
Copyright is completely up in the air. Why would anyone - the incentive to create original work is diminishing and diminishing…. the… You're only going to be able to do it for your own self, you know your personal satisfaction.
Dr. Josh Stout 14:45
That’s not all what I am seeing. I'm seeing students doing really good work. Using the AI. Yes, is effective.
Eric 14:51
In the sciences.
Dr. Josh Stout 14:52
And I think this is happening in many, many fields. What I'm worried about is that when you use AI instead of the knowledge you actually need, they start using the AI's knowledge, which isn't knowledge, it's pseudo knowledge. And I'm really worried about pseudo knowledge, copying pseudo knowledge until there's no knowledge at all.
Eric 15:12
Well, you have mentioned that that A.I. is is reviewing the papers that are submitted to the journals and that the AI is writing the papers that are submitted to journals.
Dr. Josh Stout 15:25
Yeah, no, I had my student, I had my students make timelines of when different species had evolved. Oh, I guess that's the
Eric 15:36
And the food has arrived and we're going to get back to it.
Dr. Josh Stout 15:39
Yeah. So they're making these timelines of when did a Neanderthal evolve? When Homo Erectus evolved, when did Homo sapiens evolve? And I said to them that in the past there would be several different dates that you would be given for these things, but that now basically it was all A.I.. Reading Wikipedia and whatever Wikipedia wrote was going to be the official date across the internet. Oh. And so I said, I suggest you go into some of the not original papers, but other places where knowledge is collated like the Smithsonian or National Geographic or, you know, popular organizations that will have something that's not Wikipedia. And we found there were, you know, 100,000 year differences and many of them were something like, you know, 10% of the entire lifespan was variable. I really not a lot of knowledge, but one of my students pointed out that the A.I. summary said that Homo erectus had lived until 40,000 years ago, and there is no way Homo erectus died out. More like 400,000 years ago. So the guy had dropped to zero and I said, Oh, that's a hallucination. She said, Yeah, that's my job. And so she had spotted this right away. And it was really interesting to think of that as there is. There are now people who are kind of on it. They are looking for these kinds of things and they're pretty good at it.
Eric 17:04
But there's too much knowledge out there.
Dr. Josh Stout 17:07
No, you can't you can't just verify everything all the time.
Eric 17:10
It can’t be done and that kind of a difference that'll get baked in in a minute.
Dr. Josh Stout 17:14
In a minute. Yeah, yeah, yeah.
Eric 17:16
That is utterly petrifying.
Dr. Josh Stout 17:16
Then someone who Wikipedia because the AI said it was the 40,000.
Eric 17:22
Then it's done and then it's done. Absolutely done forever like and yeah, this is crazy. This is why this is why, you know, the most important thing now is, is. And how do you do this? Because this thing is still evolving, but you can't just incorporate it into how they use it in the classes, how the students use it in their classes. But you have to incorporate teaching them how to use it intelligently in every situation.
Dr. Josh Stout 17:47
But I think I think there's also these like really somewhat insidious aspects to it where they're trying to create clickbait and they're so. So sometimes the eye goes for something that might be true but is more likely to get you to respond to it, of course. So I just Googled my own name recently because at the pharmacy I discovered there was someone with my name and my birthday in Colorado and I wanted to look into this and I immediately got sex offenders and I'm like, What? What's going on? And I looked in the did not share my name or my birthday, but that's what that's what it popped up is the first thing it was someone with the same last name. And because there was enough similarity, the clickbait was going to work. And so I think they're now guiding us to the most extreme versions of the searches that we could look for because they know we'll be interested.
Eric 18:38
I have no reason to dispute that. I mean, there's it's.
Dr. Josh Stout 18:40
A strange thing. I just noticed.
Eric 18:42
We have no idea what what else they're putting into these algorithms and with what they can already do to us. Yeah, it's.
Dr. Josh Stout 18:51
Not I'm seeing this a lot, often with scientific things where the the AI summary is a little bit wrong and you say, Huh, that's wrong. And you click on something to verify it. And I'm like, Oh, I just did what it wanted me to. Mm hmm. And that's fine, because you end up doing the actual research. But it's a little disturbing that they're first putting out something a little bit wrong. And in some cases, a lot of it really is not a sex offender, is it? Is it is.
Eric 19:16
Is that part of the algorithm? Is that a purposeful choice? And yeah, we may never know.
Dr. Josh Stout 19:23
Yeah, no, it just it said like, you know, something something in Colorado sex offender registry Joe's a start a click on that and it's Aaron Stout and I'm like.
Eric 19:31
You just put that. That's right. That's right. And we have to assume that it was purposefully done. We have to.
Dr. Josh Stout 19:37
Or at least not purposeful by a human, but that the algorithm knew that you would get more responses. We did stuff.
Eric 19:44
Like this and put that into the algorithm to get more response.
Dr. Josh Stout 19:47
No, no. These they make their own algorithms now.
Eric 19:49
No, please. No.
Dr. Josh Stout 19:51
They they know what they, what they want. They don't, they don't get told these things. I know no one in their right mind would tell you that when you Google your name, you should show up in police registrants.
Eric 20:02
True, That's not true. Not if the point is to get clicks. Not if the clicks. Yeah, but turn into money.
Dr. Josh Stout 20:07
This didn't used to happen. This is this is a brand new thing that I noticed.
Eric 20:11
But in in a high summary at the top of every search didn't used to happen.
Dr. Josh Stout 20:15
Didn't used to happen exactly. And to what I'm saying who.
Eric 20:17
Asked for it or not? Whether you are informed as to how legitimate or or correct that summary may be, you may have no idea. You've never seen this before or heard of it? No. Oh, look, a summary. Yeah.
Dr. Josh Stout 20:34
No. And it's often useful. It's often very useful.
Eric 20:37
And it's often wrong. And if you don't know, if you don't know, check, you'll never know.
Dr. Josh Stout 20:41
But what I'm saying is slightly more insidiously than the giant corporate overlords that look over our lives. I let go of that long ago. They are, they are - I think the AI's themselves are making decisions about how to maximize certain responses.
Eric 21:01
And why not?
Dr. Josh Stout 21:03
Because I think they were told to do that. Yeah, but people didn't necessarily think about all the repercussions. And so, you know, this is why I wanted to talk about sort of a return to talking about I, I stick I did want to stay more focused about what's happening in the classroom because I found that very hopeful.
Eric 21:17
I'm sorry. I'm sad in the classrooms. Yeah.
Dr. Josh Stout 21:19
And away from that, I've I've, I've really enjoying what the students are doing because they're using it in ways I didn't necessarily expect, but they are good. And I'm seeing people writing excellent papers.
Eric 21:34
That's fantastic.
Dr. Josh Stout 21:34
You know, appear well thought out. And I and they have to write their thoughts down. And so the fear about having something else do the writing is writing is thinking, hmm. But now what I'm seeing is, yeah, writing is thinking, but now it's sort of at a meta level. It's the next level up of the thinking. And I'm still seeing them have to organize their thoughts and put it into writing. But it's now at a different level and it's I am enjoying it a lot.
Eric 21:58
Well, I love this. Yeah, this is very hopeful.
Dr. Josh Stout 21:59
It is hopeful, Yeah, it's very hopeful.
Eric 22:01
Wonderful. Well, thank you, Josh. And hey, thank you, students, for the good work. It's nice. Absolutely nice to bring that out to the world. All right. Until next.
Dr. Josh Stout 22:09
Time. Until next time.
Theme Music
Theme music by
sirobosi frawstakwa
