Arthur C. Clarke has a famous quote, so famous that it gets attributed to other people, especially Neil Postman: “Any sufficiently advanced technology is indistinguishable from magic.”
I’ve been using AI fairly often to assist my work for over a year now. It has proven very useful, sometimes incredibly so. Sometimes, not so much. I don’t know if I can go back, however, to the days before it. But my first truly magical moment with AI did not come until last week, when a new AI tool called SermonScore graded my real-life sermon.
#NotSponsored
This video is not sponsored by SermonScore, and I don’t think I would have them sponsor it even if they offered, because I’m just not sure what to make of it yet. I am a little shaken, to be honest. This experience has left me not only a little shaken but also a little tempted, and I’ll explain that in a bit.
I want to talk about what SermonScore told me—both the positive and the negative things about my sermon—and how I am processing it.
Homiletical positives
First, let me take you through a little of what SermonScore said about my recent sermon. I’m not going to show you the nice things it said—let another bot praise you and not your own bot. I’m only going to tell you about those positive things more generally and then show you some of the negative things it said.
SermonScore did tell me that there were various rhetorical, theological, and exegetical elements of the sermon that it regarded as strong. And all I can say there is: I did what I was taught. The things that SermonScore named were things that I did on purpose.
So those praises—though my wife feared they would go to my head—just felt like my old teachers from seminary telling me, “Yes, you did what we taught you to do.”
Homiletical constructive criticisms
The critical comments, however, included a “Primary Growth Edge” involving the rhetorical restructuring of a key story illustration that I told in the sermon. SermonScore also critiqued me at various other theological and homiletical points, including that I ought to have used more illustrations that were not just taken from my life; that I ought to have made applications to cultural things; and that my relatively narrow gamut of personal experience within the kinds of ministry and Christian circumstances that I’ve mostly been in throughout my life ended up artificially limiting the extent of the application.
It also caught me in what I think I would call a minor exegetical error that I committed concerning the phrase “you who are spiritual” in Galatians 6. It also felt that I went too quickly through some cross-references—that if I’m going to use a supporting passage, I ought to pause long enough to show the congregation why that text says the same thing differently.
And, a little humblingly, it told me that I failed to ground my application truly in the enabling power of the Spirit through the gospel. I’ll have some reflections on that in a little bit, but I think it was right.
I included everything from my sermon—even the prayer—in the manuscript that I gave to the software. And it noticed this. It felt that I ought to have ended with the final declarative sentence rather than with a prayer.
Reflections
Now let me offer some reflections on this experience.
I assume that what I’m dealing with here is a standard LLM that’s been given a web skin, along with some direction and skills. It looks to me—I think they said this—that it’s been trained on a certain set of excellent sermons from each major tradition or denomination that it serves.
Indeed, I chose (or I think I chose) the Baptist tradition, or maybe I chose the Reformed tradition. I can’t remember which one. And this, presumably, is why I was getting feedback on the gospel-centeredness, or lack thereof, of my sermon.
Evaluation quality
But here’s what shook me: The evaluation that it provided to me was a great deal like the evaluations that I have provided to my students in seminary, only better. I haven’t graded a lot of sermons per se, but I’ve graded a lot of papers, and those papers are often very much homiletical. I’m using similar principles to grade those papers and offer feedback.
But this set of reflections from an AI bot gave me far more than anything I could really keep up with as a teacher—far more than anything I received as a student. I got page after page of detailed prose evaluation.
It was all broken up neatly with nice headings, so that I knew exactly what the software was focusing on. It wasn’t just overwhelming me with one big block of text.
But I found it very interesting that for an almost 7,000-word sermon, I got about 4,300 words of evaluation. There’s no way that’s happening very often in any seminary. Maybe that’s the equivalent of what you’d get in the British teaching system, the tutorial system. But I’ve never been through that system. I’ve never gotten feedback quite like this, with the possible exception of my dissertation.
Also, the feedback here was all written in prose. It was not a set of cryptic comments written in pen. It was all very easy to understand. I was pretty impressed with that fact.
Again, I would really struggle to provide this for my students.
An illuminating (?) parallel
And that’s a segue to a little parallel I’d like to make. Everybody, I remember, at the beginning of online seminary education was saying that brick-and-mortar, face-to-face instruction is better than online instruction. And I don’t know that I’ve ever heard anybody contradict that. That really is the obvious ideal.
But I would say that the vast majority of students I’ve taught—and I’ve probably taught something like 200 of them in various seminaries over the last decade or so—were adults who were unlikely to be able to step away from ministry, professional, and family obligations to attend a seminary.
So the trade-off is, okay, your seminary education might not be quite as good as it would be if you could sit in a classroom as your forebears did, but at least you’re getting one instead of not getting one.
My question
However, this sharpens a question I have about the SermonScore tool.
Somehow, it feels like I ought to be cultivating my relationships with various other humans who are able to give me feedback. I’ve never really done that, despite being pulled in that direction by Sean DeMars’s Room for Nuance episode about his formal sermon-evaluation process. I was pretty impressed by that.
If I’m going to be preaching to a large group of people in a church, isn’t it worth some kind of editorial process in advance? I definitely prefer to be edited when I’m writing. Why not when I’m writing a sermon?
Ping-pong
But now the ping-pong in my mind sends the ball back over the net.
Because I will note that, though I have cultivated skills to evaluate sermons, and though I use those skills “professionally” to evaluate student sermons, I’ve never been asked to provide anything like a formal evaluation of any pastor’s sermon.
I did do it one time without being asked, although it wasn’t actually a pastor. It was a layperson in the church who preached a sermon. He was a friend of mine, and I knew that he could handle it and would appreciate it.
I was pretty detailed, though not nearly as detailed as SermonScore. I won’t give any identifying details beyond that, of course. I had overwhelmingly positive things to say, but there were some definite points of growth to point out.
I sat through that sermon taking notes on the sermon—notes about the way it was delivered and about its structure. I noted the illustrations. I praised multiple specific rhetorical and theological and exegetical elements. I was again pretty detailed, but nothing like the AI in sheer word count.
I have to say, it took me quite a lot of years of education and practice, and of listening to many, many expository sermons, to develop the skills that I used in that sermon evaluation.
And maybe, in an ideal world, pastors in churches would feel the freedom to ask people who have that capacity to give them a formal evaluation every once in a while. But I very rarely hear of this happening, and I don’t exactly blame any of the pastors I’ve sat under. It would be very awkward for both of us. I’m not entirely sure why.
So maybe, although I ought to be cultivating a relationship with some other human—even beyond my wife, whose sermon evaluations are always helpful and encouraging and occasionally necessarily incisive—maybe AI sermon evaluation is like online seminary education: better than nothing.
Bizarre accuracy
The AI SermonScore tool accurately picked up not only the moments of humor that I planned but also their connection to the theme and their rhetorical purpose within the sermon. That was pretty amazing.
There were times when the AI tool articulated what I was trying to do better than I had ever articulated it to myself. This was a bizarre experience.
Receiving criticism
I have to note also that one nice thing about the criticisms—or is it nice?—is that they were delivered by a computer. Just as one of the most inhuman things I had ever seen AI do was respond with a chipper lack of self-defense when corrected, now it was offering me the opportunity to return that favor. Because I did find it’s really hard to be offended by a computer.
You preachers out there know what I’m talking about when someone says, “Hey, about your sermon . . .”
I’m 46, almost, and I’ve preached quite a few times, and still my heart skips a little beat when someone approaches me that way. I’m a fairly self-confident guy. I don’t go around moping and mooning about other people’s criticisms of me. But when another person goes to the trouble of saying something, and I don’t know whether it’s going to be positive or negative, it comes with a real weight.
Maybe that’s a problem with AI. Its criticisms were excellent, but, in essence, they were floating in space because they weren’t tethered to another human heart.
Prior to the AI era, the most incisive critiques any computers had ever offered me were things like Grammarly telling me how I’m supposed to write a sentence. But I know how to write a sentence. I don’t need a stupid computer to tell me.
It never gave me any advice that I wanted to take, and I always turned off grammar checks as soon as I could whenever I got a new computer or new installation of Word or whatever. I don’t mind spell-checking because every so often I have typos, but I hate grammar checks.
This SermonScore evaluation was different. It was insightful and incisive.
Is it cheating?
Could I, through this tool, become a much better preacher? Or is it cheating? Am I missing out on building a relationship with a brother in the trenches, or a long-term relationship—investing time with another trained pastor?
Am I missing out on opportunities to implicitly train other young preachers in good preaching by failing to hold the jam sessions that my friend Sean DeMars does?
You know, I don’t think I have time. But should I? Should I make time?
I really had a decisive moment in my life as a media ecologist at the BibleTech conference a number of years ago. A media ecologist, like Neil Postman and Marshall McLuhan, is someone who observes the overall effect of communications and other media on the culture.
Bob Pritchett, the founder and CEO of Faithlife, the company behind Logos Bible Software, really changed my mind in one conversation. He’s a very thoughtful guy. I had, and have, a lot of respect for him—not just because he used to pay my bills with his business.
He told me, as I was preparing the next day to deliver a talk on what Bible technology was doing to Bible study, that he was skeptical of a lot of that kind of talk.
He made it so that I was just not as skeptical about technology as I had been before that, because here’s what I noticed: I would issue all of these dark warnings about what technology might be doing to our Bible study, but then I would turn right around and use a bunch of technological tools.
I am more skeptical about AI, however, because I see it as having a greater potential for cheating. I do believe that wrestling with a sentence, a paragraph, or a concept—those are necessary things for my development as a writer and an exegete and a preacher.
The future
I have not fully decided what to do with SermonScore, but as of this moment, I just can’t imagine that I’m going to use it in the future.
It does feel too much like cheating. I don’t think it would be helpful for me. I’m not saying that they’ve done wrong to make this amazing tool or that you would do wrong to use it. I do not want to legislate this aspect of morality for you. But this experience had me reaching out to a fellow elder at my church to talk about the possibility of human evaluation of my sermons.
The chatterati
I usually try to come to YouTube videos with a little more definitive perspective, but I think everybody is working through this AI revolution. You know, we all land in a certain part of the chat-osphere.
We tend to listen to certain segments of the chatterati. People watch particular YouTubers—I know I do. We listen to particular podcasts. We read particular books. We subscribe to particular magazines and newspapers, and those tend to cluster around a certain viewpoint that we hold, whether conservative, Christian, or liberal, whatever.
Those subscriptions show something about our allegiances.
I don’t know about you, but I was pretty disappointed in my own chatterati—our thought leaders—during COVID. I thought, surely somebody is going to get this right. But I think we suffered as conservative Christians from pretty much the same confusions suffered by non-Christians.
I fear the same is happening with AI, but I don’t know. I wish I had greater wisdom to contribute to this conversation.
Strong and weak
I can tell you who I’m listening to. I really like Andy Crouch. I think he’s very good at making evaluations of technology. He’s a user of technology, and he does use AI, but for many years, before the advent of AI, he warned—in his book The Tech-Wise Family, and in Strong and Weak in particular—about people who want to be all strength and no vulnerability.
You don’t want to be all vulnerability and no strength, either. Humans, because we’re finite and made in the image of God, have always had a combination of power and dependence on others.
I sense in myself that I need to reach for dependence on others when it comes to any needed evaluations of my sermons. I need to be vulnerable to them. I need to risk the relationship with some people by asking them to critique me.
This is much easier said than done. It’s easiest with my wife because I trust her so much. She’s proven her love for me, so any criticisms she needs to give concerning any of my sermons come wrapped, if not in sugar and kisses, at least in nice down pillows.
For now
For now, I think that’s the best thing I can do while I look out for opportunities for others who might be able to help evaluate my sermons.
I really do just want to do a good job in order to serve whatever Christians end up having to sit before me as I open God’s Word. I also, of course, want to serve the Lord faithfully and represent him accurately.
Not that long ago, I think I would have been willing to use whatever tools were available to help me make my sermons better. There’s something in my conscience telling me that this particular tool is not really good for me.