If a quick brown fox jumping over a lazy dog is the only thing that comes to mind when you hear Pangram, perhaps it’s time to come out from under the rock you’ve been living (no judgment, I like rocks too).
The Pangram I’m writing about is an AI detection tool that has gained a lot of media attention since the New York Times reported on the alleged AI use in Mia Ballard’s book Shy Girl in March this year. In the reporting, they cited that a whopping 78% of the book’s content was suspected to be AI-generated, a number produced by Pangram.1 It’s touted as the new cutting edge tool for AI detection, especially since the release of its newest model version, v4. The release of that version coincided with Substack’s announcement that all articles published on or after July 21, 2026 can be scanned for AI involvement using none other than Pangram itself. Unless writers opt out, every Note, every article on Substack can now be scanned for AI use. And in late October, Pangram already released its updated model version 4.1.
I appreciate that the numbers in Pangram’s latest technical report are really impressive. With a false positive rate of 0.0041% and false negative rate of 0.3396% it seems almost impossible to fool. At the same time, there is a seemingly endless stream of essays on Substack and elsewhere claiming to have gamed the system and tricked Pangram into thinking human-written text was AI-generated (inducing a false positive) or conversely that AI-generated text was human-written (inducing a false negative).
If you’re chronically onstack,2 none of this will be news to you. At least if you’re on an algorithm similar to mine. However you feel about Pangram personally, you’ll likely find yourself exposed to numbers such as 62% AI, segments, confidence labels, and other Pangram-related lingo. But as popular as Pangram has become, I don’t think everyone understands what a Pangram result actually means. At a time in which the rise in AI-generated content is being rivaled only by the growing prominence of tools created to detect it, it’s important that we have a decent understanding of the technology we seem to be putting our collective trust in. Conveying that understanding is what this article will try to achieve.
This is Part 1 of (probably) 3
Long story short: This was originally going to be a single article about evaluating the accuracy of the Pangram scan I ran on a play I wrote with ChatGPT nearly four years ago. But what started as one story on one topic branched into multiple related-but-distinct sub-topics that I simply couldn’t fit into a single, coherent article. So consider this Part 1 in a series of (likely) 3 articles on Pangram. If articles are ice cream flavors, this one’s vanilla. A straight-forward yet detailed description of how to interpret the different levels of a Pangram scan.
Look, obviously I want you to read the whole article because I think reading it in its entirety will give you the best overall understanding of how to interpret a Pangram scan. But if you’re the type of person not to heed such advice, I suppose, if you must, if there’s nothing I can say to convince you otherwise... you can skip ahead to the recap sections here and here.
Disclaimer: I have no affiliation with Pangram, and the text I’ll use to demonstrate most of the following results was scanned using a free trial of Pangram version 4.1 on October 24, 2026. Scans I ran using Pangram’s API I paid for myself. Given the topic, I want to emphasize that I did not use LLMs in any part of the writing process of this article. Finally, not everything I write about in this article is explicitly documented in Pangram’s communications. Whenever I am making conjectures or offering best guesses, I will transparently say so. If you have some expertise in this area and notice a mistake, please let me know in the comments.
Interpreting a Pangram report
To demonstrate what a Pangram result means, I’ll be using one of my own texts from 2022. I wrote about this in my last article. It’s a play I wrote with ChatGPT about an entity like ChatGPT because I wanted to test what such a technology was capable of. For this article I’ll be using the first draft of the play, since that was the one I used ChatGPT most on. That draft is eye-gougingly horrible, but its content is irrelevant to this article.
The reasons I’m using a draft of that play for this demonstration is that (1) I know the ground truth of its production process, (2) it might more accurately reflect people’s hybrid approaches when using LLMs than using a text generated with a single prompt, (3) I wrote this before I ever learned about AI detectors, meaning I wrote this without trying to evade detection, and (4) it’s a longer piece of writing containing 9,000+ words, which should make the Pangram results more useful. If you’re interested, you can access that text’s Pangram report here.
The structure of a Pangram report
In case you’ve never seen a Pangram report, this is what the newest version (4.1) looks like when using the online dashboard:
The results themselves constitute a bit of a nesting doll situation.
What we first see on the dashboard are the global, or document-level results. For instance, two of the most frequently-cited elements are the predicted AI fraction (here 46%) and a label (here “Mixed”). But these document-level predictions reflect the more fine-grained results at the segment level. So to better understand what we’re looking at, that’s where we’ll start.
Segment-level results
Segment-level results are visible in the “Details” tab:
As you can see, Pangram identified 425 segments in this document. Segments are sections of the whole document that the model attributes to different origins. They can be of different lengths; in this case a segment ranged from 0 (simply a “?”; though that is an anomaly) to 229 words. The figure in the screenshot above visualizes authorship progression across all text segments. Orange indicates segments the model considers AI-generated, green indicates human-written segments.
If you’re familiar with previous versions of Pangram, you’ll notice that 425 segments is a lot more than what we’re used to seeing. For comparison, in version 4.0, Pangram split the exact same text into only 26 segments, ranging in length from 30 to 1,798 words. You can find that report here. This newest update allows for much shorter segments as long as they are separated by terminal punctuation, which is what gave rise to some one-word segments in my case.
Segment-level results currently consist of three main components: class probabilities (how likely the model thinks the segment was written by a human or AI), a provenance label (the entity to which the model ultimately attributes the segment), and a confidence label (how confident the model is about its provenance label).
Class probabilities
When we click into a segment, we might see something like this:
What we see visually represented here are the class probabilities. According to the most recent model card, these numbers estimate the “posterior probabilities for Human, AI-Assisted, and AI-Generated.” What this means is that after several layers of fitting and smoothing, these are the model’s final—hence the use of “posterior”—predictions about how likely this segment was authored solely by a human, fully generated by AI, or a combination of the two.3 Importantly, these numbers do not represent the probability that the segment’s final class label is correct.
The distinction between AI-generated and AI-assisted text is what’s meant to be the beauty of Pangram: It doesn’t just make a binary distinction between fully human-written or straight-up AI-generated text but considers authorship to be on a spectrum. An AI-assisted segment implies that the text is not assumed to have a single distinct author, which might result from a person having their own words edited or rewritten by AI.
The inclusion of class probabilities is a slight deviation from previous model versions, where the dashboard provided a visual representation of an AI assistance score:
From what I can tell, the two metrics are closely related. The AI assistance score simply condensed the AI-generated and AI-assisted class probabilities into a weighted fraction (the predicted probability of AI-generated text counting twice as much as that of AI-assisted text), ranging from 0 to 1. While the dashboard no longer provides AI assistance scores, they are still included in results obtained via API calls.4
Class label
The second main result at the segment level is the class label. Although this isn’t explicitly documented in the model card, the class label seems to be the discretized version of the class probabilities. What I mean by that is that whichever class—Human, AI-generated, AI-assisted—is attributed the highest probability gets the final class label.5
The implication of this is that even though the segment ultimately gets labeled as human, the model might still attribute a sizable probability to the segment—or elements in that segment—being generated by AI.
For this to make sense it’s important to understand that segments are built bottom-up: Pangram makes tokenwise predictions (think of a token as a word or part of a word), tokens are concatenated to sentences, sentences form segments. But because Pangram 4 uses something called sentence majority voting to determine the most likely label for a given sequence of tokens, all tokens in a sentence adopt the same class label (say, human) of the majority of tokens in that sentence, even if individual tokens receive a low probability that they’re human-written.
In the text I scanned, 203 of the 425 identified segments were flagged as AI-generated, none AI-assisted.
Confidence label
We’ve covered class probabilities and resulting class labels. Last but not least, every segment also provides a confidence label. This label expresses, surprise, surprise, how confident the model is in assigning the class label for that segment. This confidence can be low, medium, or high. If the model assigns low confidence to a segment, it now also includes some hedging by adding the word “Likely” to its output. So a segment with low confidence will be “Likely human”, “Likely AI”, or “Likely AI-assisted”.
Though tempting to assume otherwise, as per Pangram’s model card, the confidence label does not directly represent the probability that the segment’s class label is correct.6 It’s also distinct from the class probabilities I talked about above. This means that a segment can have a class probability of 100% that it was authored by a human while still being assigned a low confidence:
Recap and additional segment-level outputs
A quick recap: For each segment of text within a longer document, Pangram tells us how likely it considers each of three entities to have authored the text (class probabilities; adding up to 100% across the three classes human, AI, or AI-assisted), who it ultimately thinks authored the segment (class labels; human, AI, or AI-assisted, depending on whichever class got the highest probability), and how confident the model is about its judgment (confidence labels; high, medium, low; not a direct probability estimate of whether the class label is correct).
Although I would consider those the three most important outputs at the segment level, there are more elements the model provides. I’ll list them very briefly here:
AI assistance score: I alluded to this in the section on class probabilities, and this is just a weighted sum of the AI-generated and AI-assisted class probabilities. Pangram describes it as “a continuous AI-involvement score”, although I don’t think it actually quantifies how much of the segment was written with or by AI. As of v4.1, this is only outputted in scans made with API calls.
Humanization: A humanization attempt is when AI-generated output is altered to make it sound more human and thus evade AI detection. Pangram flags segments that it suspects have undergone a humanization attempt. In addition, the API output also provides a humanizer score quantifying the estimated probability that a humanizer was used.
Calibrated confidence: This is new to v4.1 and is currently not documented in the model card but is outputted in scans made via API. As it’s not explicitly defined, I can only guess that it’s the model’s estimated probability that the class label is correct (i.e., the thing that neither the class probabilities nor the confidence labels are). Nonetheless, at least in the scans I ran via API, a low confidence label seems to map onto a calibrated confidence estimate up to 90%, medium confidence covers a range from 90 to 98%, and everything at or above 99% got rated as high confidence.
Document-level results
That was a lot, and I promise the document-level results will be much quicker.
The main outputs at this level are the AI and human fractions as well as the final prediction of the whole text as human, AI, or mixed. For the text I scanned, the results came back as 46% AI, 54% human (fractions), and mixed (final prediction).
AI and human fractions
Unlike the segment-level class probabilities, the document-level AI and human fractions are actually a measure of AI involvement in the whole text. Or at least a rough measure. An AI fraction of 46% means that, according to Pangram, 46% of the whole text was generated by AI, the other 54% were human-written.7
But also unlike class probabilities, the document-level fractions are not computed from the ground up. They take their cues from the segments. So if a segment is labeled as AI-generated, it doesn’t matter if Pangram predicted a 0% chance or a 49% chance that it was human-written, 100% of the words in that segment count toward the document’s AI fraction.
An element that seems to be new in v4.1. is that for some texts, Pangram might now also provide a range of AI fractions if it’s less certain of its output:
If you want to approximate your document-level AI fraction, you can add up all the words in AI-labeled segments and divide them by the total word count. Though good luck doing that manually through 425 segments. I’ll see you in a few hours.
Fortunately, popping the results from the API call into a spreadsheet can save us some time. Here are the document-level fractions directly obtained from the API call.
And here are the document-level fractions I calculated from character counts in AI-generated and human segments. It’s not a perfect replication but a pretty close approximation.8
Final prediction
The other main document-level result is Pangram’s final prediction or classification. The text I scanned got “Mixed”, which happens when a document’s human fraction is lower than 90% and its AI fraction is lower than 80%. Conversely, if the human fraction is at least 90%, the text will get the human label; if the AI fraction is at least 80%, it will get the AI label.
As with the document-level AI and human fractions, the document-level final prediction is also not determined from the bottom up but relies on the final labeled segments.
The teensiest, tiniest nutshell that wouldn’t even fit Thumbelina
Another quick recap: Document-level results consist of human and AI fractions as well as Pangram’s final predictions of whether the text was human-written, AI-generated, or mixed. Both of these elements take their cues from the final labeled segments, meaning everything but the segment-level class labels are irrelevant to the document-level results.
What I said at the segment level—that a segment may be labeled as AI-generated but still have a substantial probability it was human-written or vice versa—also goes for the document as a whole. So even if you run your text through Pangram and you get back “100% Human”, it does not automatically mean that nothing in your text was flagged as AI-generated. It does not necessarily mean that your segment-level class probabilities always came back as 0% AI-generated, especially if your results include low-confidence segments. It just means that every segment was human enough, and because the segment labels determine the document-level fractions, the whole text can get classified as human.
Admittedly, I don’t think a scenario in which a text is solely comprised of segments with close-call 51% human/49% AI class probabilities is all that likely. And most texts written entirely by a human or generated entirely by AI will likely have a strong enough signal to come back 100% Human and AI, respectively. It’s probably the hybrid texts like the one I used throughout the article that will get results that seem extra noisy because they’re harder for the model to pin down.
Nonetheless, it’s important to know what the document-level AI fraction—the arguably most frequently cited number when it comes to Pangram scans—actually does and does not mean. And I would suggest always looking beyond the document-level results into the segments to look at class probabilities and confidence labels. Unfortunately, at the time of writing, people who use the integrated Pangram tool on Substack can only rely on the document-level results because no segment-level results are provided.
Ice cream flavors and recommended reading
As I noted up top, this was Part 1 of (probably) 3 in a series about Pangram. Although it was admittedly quite technical at times, I would still call this a pretty vanilla article. In Part 2, I’m hoping to dip into the bitter orange territory and talk about my qualms with this version of Pangram (ft. Highly Variable Results and Ultra-Short Segments).
If you’d like to be informed when the new flavor is available for tasting, you can subscribe to my newsletter and be the first to know.
If this article has whetted your appetite (I’m committed to this metaphor now) to learn more about Pangram, how it works, or how to feel about it, here is a small collection of articles that cover Pangram from different points of view:
If you’re interested in learning about some potentially fishy behind-the-scenes stuff on how this story came to be, I can recommend this article by The Drey Dossier.
Yea or nay on the nomenclature?
As per the technical report, the model estimates tokenwise marginal probabilities before segments are assembled, so my guess is that these segment-level probabilities might be averages across all tokens in the respective segment.
Based on the output from the API call, I was able to reconstruct the AI assistance score from the class probabilities using the formula documented in the model card of v4: P(AI-generated) + 0.5 * P(AI-assisted).
My hunch as to why Pangram replaced the AI assistance scores with class probabilities in the dashboard is that with the update to v4, the class labels could no longer be directly inferred from cut-off points in the the AI assistance scores but they can be from the class probabilities. Although at least in the scans I ran via API, AI assistance scores and class probabilities were very similar.
It indicates how peaked (i.e., how concentrated) the posterior distribution of the mean marginal probability of the segment’s final class label is.
As I’ve noted here, 46% AI is a huge drop from a 72% estimate using model v4.0. Considering that the whole text was outputted by ChatGPT in some form or other, the result from v4.0 more accurately reflects the text’s genesis
When I initially did this using v4.0, I was able to use the segment character counts to replicate the document-level fractions to the 7th decimal place. Not sure why I can’t get as close when I use v4.1. Also, if you’re wondering why the AI fraction from the API call is 51% when I was talking about 46% previously, that makes two of us. This version of Pangram seems particularly prone to producing different results upon repeated scans.