Recently, I got a taste of my own medicine—and I loved it. A fellow Christian, a pastor, respectfully asked me to answer a question I’ve put to his “side” in a debate—though I wish I had a better word for it, because we share a ton of agreement on far more important matters. Now I pose the question to you: how many readability difficulties need to appear in a given document before it no longer counts as “intelligible”?
I spent a long time asking KJV-Onlyists this question, and a few months ago I asked the same question to those who, like this pastor I just mentioned, resist officially updating the language of the Westminster Confession.
In other words, algorithm—if you’re listening—CONTROVERSY! People love listening to other people disagree about stuff. And in this video I’m going to give you some disagreement. But, alas, algorithm, it’s going to be patient and constructive, not combative and nasty.
Because my opponent in this debate, ordained OPC pastor Taylor Sexton, showed me one of the supreme acts of charity in debate by asking me a good faith question. I really love this. So let me raise it again: what kind of objective measure can we use to determine whether a given text is “readable”? Or, I could say, “sufficiently intelligible.”
Hu and Nation
I actually did what YouTubers are not supposed to do. We’re supposed to go off one half of one percent cocked. There’s no time for research; all that matters is views. But no, I waited a few months and did some research in the meantime. And I was fascinated by what I found. Bible translators and confessional Presbyterians aren’t the only ones facing the question of what makes a text intelligible. Language researchers are also interested.
I’m going to focus on one article that was particularly apropos, which is a Frenchified vocabulary word meaning, “it said what I hoped it would.”
But I didn’t make it say these things. It said them before I finished college, and on an island far from my influence. The article is titled, “Unknown Vocabulary Density and Reading Comprehension,” and it was written by Marcella Hu and Paul Nation, who were at the time in the school of Linguistics and Applied Language Studies at the Victoria University of Wellington, New Zealand. Marcella Hu continued to work in this field, as her Google Scholar profile shows. She’s written journal articles about subjects such as “Word Frequency Levels and L2 Learners’ Vocabulary Knowledge Level.”
Paul Nation was, I believe, Marcella Hu’s professor at the time of the article we’re going to look at, and he has been especially prolific in this field—up to the present day. His ResearchGate profile has dozens of articles on vocabulary, reading, language fluency, second-language acquisition, and related topics. Nation is an expert in the field of reading and comprehension. That’s all I’m trying to establish here.
It’s the job of scholars such as Nation and Hu to probe the common-sensical, to try to help people like Taylor Sexton and me find some reasonably objective ground on which to claim that the Westminster Confession does or doesn’t need to be translated into contemporary English.
And they start out with what sure sounds like common sense to me:
There may be a threshold where vocabulary knowledge becomes sufficient for adequate comprehension. If the learner is on one side of the threshold, vocabulary knowledge is not sufficient for adequate comprehension. If the learner is on the other side, then the learner knows enough vocabulary, other things being equal, to gain adequate comprehension of the text.
Three major views that boil down to one
As scholars are wont to do, Hu and Nation lay out major views. They give three. There’s the “instrumentalist” view which “sees vocabulary knowledge as being a major prerequisite and causative factor in comprehension.” But then I’ve read some very difficult writers that, even as an adult with lots of reading under my belt, I found nearly impossible to understand—even though I knew all the words they were using.
So there’s also an “aptitude” view, which “sees vocabulary knowledge as one of many outcomes of having a good general cognitive ability.” Trying to read David Bentley Hart’s The Beauty of the Infinite persuaded me that I do not have the cognitive ability I always imagined I did. Ephraim Radner’s Time and the Word did the same—even though I’ve read Radner’s columns in First Things many times with full comprehension. I do apparently lack the aptitude for some reading of some writers, even theological ones. There’s just more I need to know before I can tackle these books.
So there’s a third view, a “knowledge” view in which “vocabulary [i]s an indicator of good world knowledge.” This view argues that “the reader must bring as much information to the text as the reader expects to get from it.” Hu and Nation give a short example: “It is difficult to read about astrophysics if you know nothing about it.”
I’m not sure I see a big difference between view 2 and view 3. But no matter: which view is right?
Hu and Nation say, citing other scholars, that “no serious scholar holds any of these positions to the exclusion of the others, and different relationships exist at different stages of vocabulary growth and skill development.” Good reading requires knowledge of vocabulary, general cognitive ability, and knowledge specific to the topic about which the reader is reading. That’s why it’s so hard to answer the apparently simple question, “Is the Westminster Confession sufficiently intelligible?”
Hu and Nation cite other scholars, Laufer and Sim, who “determined that the most pressing need of the foreign language learner was vocabulary, then subject matter knowledge, and then syntactic structure.” That last one is maybe part of general knowledge? In other words, good reading demands a fair bit from people. Humans are like the proverbial inexplicable bumblebee. Scientists looked at the bumblebee’s girth and its small wing size and determined that it’s probable that bumblebees cannot read at all.
I can’t help but bring in what I’ve been told by KJV-Onlyists since time out of mimes: We just need to teach people how to read the KJV. I’m being told the very same thing by Taylor Sexton and others who don’t wish to see the Westminster Confession translated into contemporary English. They’ll say to me, “That’s what pastors are for—explaining things to Christian people.”
And I think what’s happening with a response like that is taking one of these three elements of good reading and forgetting that all are necessary. I have seen people with good general cognitive ability who bring a fair bit of theological knowledge to the Confession who nonetheless simply don’t realize that they’re misunderstanding words in that Confession.
Authentical
The supreme example is the word “authentical” in WCF 1.8. The line reads that “the Old Testament in Hebrew … and the New Testament in Greek … are … authentical.”
I’ve heard people say that the confession here is saying that the Westminster divines were calling the Hebrew Old Testament and Greek New Testament then in their possession “authentic” or “genuine.”
But “authentical” simply didn’t mean “authentic” or “genuine” in the 1600s in a context like that one; it meant “authoritative.” I talked about this in my previous video. This is clear in the OED.
Nonetheless, the meaning “authentic” works well enough in context that people don’t realize they’re misunderstanding.
I have seen at least one Reformed minister with advanced seminary training who didn’t even seem to grasp what I was saying when I pointed this out. He is no dummy, I can assure you. But it’s really hard for a lot of people to imagine a time when familiar English words meant something different than they do today. And it’s effectively impossible—no, it is actually impossible—to read a long text like the Westminster Confession with that fact kept in mind at every point.
As viewers of my channel will know, I call words like “authentical” in WCF a “false friend.”
Other archaisms
And then, of course, there are words like “vouchsafeth” and “succor” which are somewhere between dead and doornail. And odd, archaic collocations which are unnecessarily difficult, such as “any wise opposing God’s truth” or “answerable conversation.” Even if you picked up from context that “any wise” has to mean “any way” you still might stumble for a beat. And even if you knew that “conversation” meant “conduct” or “lifestyle,” you’d probably not know what an “answerable” lifestyle is.
So we’re back to the question: how many of these unfamiliar lexical items—both words we today know we don’t know and words we don’t know we don’t know—can appear in a text before it must be called “not fully intelligible”?
The general cognitive ability point and the background knowledge point suggest that there is no one answer to this question. There are as many degrees of reading skill as there are readers. There is no clear and definite line between a readable text and an unreadable one. But there’s no clear line between blue and green or between red and orange, and that doesn’t mean those are a mirage. Useful generalizations can be made.
Putting a number on it
My gut feeling as a philologist, not a social scientist or someone with access to grant funding for studies, has long been that the KJV is about 5% unintelligible due to language change. I’d say the same, roughly, for the Westminster Confession—only its content is a little more difficult, so though the vocabulary presents a problem similar to that of the KJV, the concepts are probably more demanding. And those do indeed need to be taught, in part through the confession itself. You don’t want to change the content, just the vocabulary and syntax, and only where needed. And my gut sense as a teacher of the Bible for many years is that where we draw the line between readable and unreadable has a lot to do with two little words: for whom? For whom was the Bible created? For whom was the confession created? Who is the intended contemporary readership of each text? I assume the answer is not “clergy” but “laity,” regular Christians.
Does my longtime very rough guess of 5% unintelligible place the KJV and the confession across the readability line? Are these texts blue or green, or perhaps orange?
And this brings us back to the paper that we’ve been examining. I am not an expert in second-language acquisition. But at least these two experts in applied linguistics, citing numerous other experts, did come up with something definite in answer to our question. They put a number on it:
Laufer found that the group scoring 95% and above on the vocabulary measure had a significantly higher number of successful readers (scores of 55% and above on a reading test) than those scoring below 95%. The 90% level did not result in significant differences between those above and below. A comparison of the 95%+ group with the 90–94% group revealed a significant difference in comprehension scores.
When I read this, I did not just a double take or a triple take. I did 1,611 takes. The very number I had used as a rough guess was the line these experts used to divide intelligible texts from unintelligible ones. I’m not a math guy, so please have mercy on my interpretation here, but that study also made getting 55% of the comprehension questions right as the definition of “passing,” because that was at the time the lowest passing grade allowed in the Haifa University system (presumably the system where Laufer and Sim taught?). Even if only 5% of the words in a text or unfamiliar—whether you know it or not—readers will find themselves in a comprehension drop-off.
Hu and Nation provide a little chart:
| % text coverage | Density of unfamiliar and familiar tokens | Number of text lines per 1 unfamiliar word |
|---|---|---|
| 99 | 1 in 100 | 10 |
| 98 | 1 in 50 | 5 |
| 97 | 1 in 33 | 3.3 |
| 96 | 1 in 25 | 2.5 |
| 95 | 1 in 20 | 2 |
| 90 | 1 in 10 | 1 |
| 80 | 1 in 5 | 0.5 |
If only 1% of words are unfamiliar, that is, of course, 1 in 100. You’ll encounter an unfamiliar word approximately every ten lines. Raise the percentage of unfamiliar words just one point to 2%, and you’ll encounter an unknown word in around every five lines. By the time you get to 5% unfamiliar “tokens,” they call them—units of meaning, words—you’re seeing an unfamiliar words in every other line. That’s why there’s a drop-off.
Hu and Nation did their own study in which they introduced certain percentages of nonsense words into an English text and asked ESL students to answer questions about that text after reading it. You can see the comprehension scores drop the more nonsense words are added:
They noted that the majority of readers who had texts that contained 5% unintelligible words did not demonstrate adequate comprehension of the text. Hu and Nation actually conclude:
This research does not support the idea of a 95% vocabulary knowledge threshold for comprehension of narrative text. Although adequate comprehension is possible with this coverage by known words, only a small proportion of the learners achieved it. It seems that around 98% coverage may be needed for most learners to gain adequate comprehension.
Getting a 95–98% score on a difficult history exam is a very good thing. You remembered a lot. Good for you. Getting a 95–98% score on vocabulary knowledge is a bad thing. Especially if you don’t always know you’re missing words because of language change.
Hu and Nation think that the absolute bare minimum vocabulary threshold is somewhere between 80%–90%. That is if a reader lacks knowledge of between 10% and 20% of the words in a given text, that reader will not be able to understand.
Hu and Nation in their paper point out what is obviously commonsensical:
Studies show that the number of word families needed to cover a set percentage (say 95%) of the tokens in a text depends on: the type of text (novel, newspaper, academic text, spoken informal conversation, etc.), the length of the text, and the homogeneity of the text (same topic and same writer).
They did their test with a narrative text, a story. And that story was ordered chronologically. Highly abstract texts like confessions of faith would, presumably, be generally more difficult for readers.
Hu and Nation say that, before their study, research had not “been able to provide a clear guideline about the optimal density of unknown words.” At least they gave clarity. A number. 98%. I’ll stick with that, too. If there is an average of more than lexical difficulty per five lines for the intended readership of a text meant for the average person, that text can fairly be called “not sufficiently intelligible.” According to Hu and Nation.
Putting a number on the KJV and WCF
I am a tired YouTuber right now. I cannot honestly say that I have done a thorough count of readability difficulties in the Westminster Confession. I did have ChatGPT give me ten random paragraphs from it, and I marked them up for words and phrases and collocations that I thought would be difficult for contemporary readers, generally because of language change. I found approximately thirty-eight words in that category. That’s close to 4%.
But I’ll be honest: I am simply not smart enough to count or weigh the clear difficulties in syntax or the use of words like “thereby” and “unto” that are comparatively uncommon today. It was the sentence length and construction that most struck me as difficult in these paragraphs.
How do I count this?
They who are effectually called and regenerated, having a new heart and a new spirit created in them, are farther sanctified really and personally, through the virtue of Christ’s death and resurrection, by his word and Spirit dwelling in them; the dominion of the whole body of sin is destroyed, and the several lusts thereof are more and more weakened and mortified, and they more and more quickened and strengthened in all saving graces, to the practice of true holiness, without which no man shall see the Lord.
And after all this talk about concrete numbers, and after insisting (and believing) that it was fair for my Christian opponents in a serious debate to ask me for concrete numbers, and after going ahead and providing one—98%—I find myself resisting the need for numbers.
A story and an observation
So let me conclude with a story and an observation.
I’ll never forget handing my precocious nine-year-old an article about electric cars, because he was very interested in the topic. He was and is quite the reader. But he stumbled hard. He simply didn’t have the background knowledge to read a newspaper article written for a college-educated audience.
The first line read, “With a new decade starting, it’s time for conscientious columnists to undergo self-administered decennial performance reviews.”
Next paragraph:
“For the past 10 years, I’ve waged a quixotic counteroffensive against electric-car boosterism, raining skepticism on the vehicles’ potential to cure climate change, much less to be the clean green wave of the transportation future.”
He understandably lacked both the vocabulary knowledge and the real-world knowledge he needed to read this well. He’d never been through anything that was decennial, because he was only nine. (I wrote about this at TGC right before COVID hit, about a hundred years ago now.)
That child turns sixteen next week. I just re-read those words to him from that piece, and he got the message completely, almost perfectly. He just didn’t know the word “quixotic,” though now he does because I explained it to him. Out of 51 words he was ignorant of one. But he still got the message, because it was a not-strictly-necessary adjective, he knew 98% of the words in total, and he has now gained sufficient background knowledge of columnists and cars.
I didn’t do a study. But am I justified in learning anything from my many, many personal experiences like this? Experiences with my own kids, experiences with other people’s kids, experiences with YouTube commenters, experiences in many evangelistic conversations, experiences as a reader myself? Or do I have to have a scientifically rigorous study to justify my conclusion that the Westminster Confession of Faith needs to be translated into contemporary English?
It’s nice to have studies; don’t get me wrong. They can correct common sense when common sense is wrong. But did people know anything before the scientific method? I think they did. And I am willing to make what in Latin they call an “argumentum ad verecundiam,” an argument from authority. Who’s authority? My own.
My dad is a natural Socratic teacher. He was an English major at the University of Virginia. He is the reason I learned to read at four. He is, presumably, the key reason why I was a five-time spelling bee champion in elementary school. (And, like him, my superior spelling did not impress girls in the way I hoped.) When the kids in my second-grade class had seat work time while Mrs. Page taught the third graders on the other side of the room, they came to me with their questions. I specifically recall trying to guide them to the answer rather than giving it to them. I have to imagine I learned that skill from my dad. I have been teaching others since at least age seven. When I left the weekly evangelistic ministry I gave my heart to for six years, I got some of the most meaningful praise of my life from my mentor, whom we called PV: he told the boys, “Have you noticed that Mr. Mark is always, always talking to you and explaining your memory verses?” I hadn’t noticed it, but I did then. It was and is so natural to me. I just didn’t see the point of teaching the urban boys to repeat syllables for points for the green team if they didn’t understand the syllables. I am nothing—nothing—if not a teacher. I magnify mine office.
And I say: the Westminster Confession of Faith, like the King James Version, is an inestimably precious document that is simply too difficult to read for too many of the people who are supposed to read it. The Christian faith is meant for the plow boy, or it is meant for no one.
I’m with Erasmus, who said,
I disagree very much with those who are unwilling that Holy Scripture, translated into the vulgar tongue, be read by the uneducated as if Christ taught such intricate doctrines that they could scarcely be understood by very few theologians, or as if the strength of the Christian religion consisted in men’s ignorance of it.
But if normal people don’t know what a “chief end” is anymore, it’s not the people who are broken; it’s the tradition. The tradition will be tradited to smaller and smaller groups of people if those people refuse to translate it into words regular people can understand. Good Christian men who disagree with my assessment can make appeals to their authority, too, I know. But I’d like to think that years of careful attention to archaic words and their effect on intelligibility counts for something.