The three kinds of text analysis behind content tools

Nearly every tool that looks at your writing describes itself as doing text analysis. Content optimizers, readability checkers, plagiarism scanners, brand-voice graders, the thing your CMS bolted on last quarter — all of them analyse text, and the phrase tells you almost nothing about what is happening to your draft.

Underneath, there are roughly three techniques, and they are not variations on each other. They answer different questions, fail in different directions, and hand you results with very different standing. Knowing which one you are looking at is the difference between a verdict you can argue with and a number you have to take on faith.

The oldest technique is also the simplest to describe. Break the text into words, throw away the ones that carry no topic — the, and, your, how — and you are left with the concepts the page is built from. Do the same to the target query. Then compare the two sets.

This tool measures three things per section: how many of the query’s concepts appear at all, how often they appear, and whether any of them reached the heading. Weighting those three is where the design happens, and ours is public: coverage can move a section more than four times as far as raw frequency can, and the frequency term stops paying out entirely once query words pass about one word in fifteen.

That ceiling is the interesting part. Past it, repeating your target phrase earns literally nothing — not a diminishing return, a flat zero. Refusing to reward the behaviour that scored optimizers are famous for encouraging costs one line of arithmetic, which raises a fair question about why it is not simply standard.

The real appeal of layer one is not the arithmetic, though. It is that every step of it can be shown to you. A report can name exactly which of the query’s words a section used and which it never mentioned, which means you get to disagree with the verdict on the evidence. That is a meaningfully different relationship from disagreeing with a number.

The honest way to demonstrate a limitation is on your own tool, so: below is real output from the analyzer that runs this site. The query is how to store green coffee beans. The draft has four sections — an opening about shelf life, a section on keeping moisture away from the beans, a tangent about the early history of the coffee trade, and a section on freezing.

One of those four is genuinely off-topic. Here is how the analyzer ranked them.

The tangent about the coffee trade scores 44 and is marked Drift, which is correct. The section on freezing beans scores 48 and is also marked Drift, which is not. Freezing is one of the two things a person searching that query actually wants to know.

The mechanism is not mysterious, and it is worth spelling out because it generalises to every lexical tool you will ever use. The freezer section is about storage from beginning to end, but the specific token store never appears in it. The heading says Storing — and this analyzer’s stemming is shallow enough that it never connects storing to store. So the section matches one of the query’s four concepts, the tangent about Yemen also matches one, and the entire difference between them comes down to how often those words happened to recur.

The second-placed section is stranger still, and it goes wrong from the opposite direction. The most genuinely useful paragraph in the article is Keeping the moisture out, and it uses three of the query’s four concepts — more of them than any other section in the draft. It still came back Mixed, at 66, because not one of those concepts appears in its heading. A heading like Keeping the moisture out is written for a reader rather than for a query, and headings carry weight here, so the section is penalised for precisely the piece of editorial judgement you would want a writer to exercise.

This is not a bug report. It is the defining property of the technique: lexical analysis measures vocabulary overlap, and vocabulary overlap is a proxy for topical relevance that is right often enough to be useful and wrong often enough that a verdict has to come with its evidence attached. A tool that showed you 48 and nothing else would have quietly misled you. A tool that shows you 48 alongside this section never uses the words store, green or coffee has told you something true, and handed you enough to overrule it in four seconds.

The obvious fix for vocabulary blindness is to stop comparing words. Distributional methods — embeddings, vectors, whatever the marketing calls them — convert a passage into a long list of numbers positioned so that passages about similar things land near each other. Compare two positions and you get a similarity figure that does not care whether anyone said store.

Run the coffee draft through that and the freezer section is very likely rescued, because freezing, airtight containers and condensation sit close to storage in a way that no word-counter can see. That is a real gain and the reason the technique took over.

What you give up is the sentence that starts “because.” The output is a distance. There is no wording behind it, because wording is precisely what got dissolved to produce it. When a distributional tool says a section scores 0.71, the honest expansion of that is 0.71. An editor who disagrees has nothing to push against, and the failure mode is quieter than layer one’s: the number looks equally confident whether it is right or wrong.

The newest layer skips measurement. Hand the draft to a language model with instructions, and it will discuss your article — structure, argument, whether the introduction earns its length, whether a section belongs. The range of questions it can address is genuinely open-ended in a way neither of the other layers approaches.

Three things come with that, and they are all structural:

• —It may not agree with itself. Ask twice, and a section can be a problem the first time and fine the second. For a brainstorm that is harmless. For a check you run before and after an edit, it means the difference between the two runs is not cleanly attributable to your edit.
• —Fluency is not evidence. A model will produce a confident, specific, well-argued account of why a section drifts, and it will produce one just as readily when the section is fine. The prose quality of the explanation carries no information about whether it is correct.
• —Your unpublished draft goes somewhere, unless the tool runs the model itself. A client’s article, an unannounced launch, something under embargo — it has been sent to a third party to be read. That is often a perfectly acceptable trade. It should be a decision rather than a surprise.

The temptation is to read this as a ladder, with generative at the top and word-counting as the thing we have moved past. It is not a ladder. Each layer bought its strengths with a specific weakness, and the weakness never went away.

A better question than which is most advanced is what do I need this answer to survive:

• —You need to re-run it after an edit and trust the difference. Layer one. Determinism is the entire point, and nothing else offers it.
• —You need to catch a section written in the reader’s vocabulary rather than the query’s. Layer two, and accept that you will not get a reason.
• —You want a second opinion on structure and argument. Layer three, treated as a colleague with strong opinions rather than as a measurement.
• —The draft cannot leave your control. Layer one, or a tool that runs its own model. This one is a constraint, not a preference.

Most working editors want more than one of those on different days, which is a reasonable argument for owning more than one tool and a poor argument for a single subscription that claims to be all three.

Topic Drift Checker: layer one, all the way down. The analysis is lexical, it runs on our own server, no language model reads your draft, and the same draft and query return the identical report every time. The coffee example above is our own output, produced by the code that is running right now — not a competitor’s tool made to look bad.

The cost is exactly the one the example demonstrates: a section that covers your topic in vocabulary the query does not contain can be marked Drift, and you have to be the one who notices. Every verdict ships with the wording behind it specifically so that noticing takes seconds. That is the trade this tool makes, stated plainly enough that you can decide whether it is the wrong one for you.

The full method goes through how sections are split and how the bands are set, including what is still missing. The longer argument against a single score covers why the report has no overall number to chase.

The useful test of any of this is an article whose weak spot you already suspect. Run it, see whether the map agrees with you, and pay attention to the sections where it does not — those are the ones that tell you what kind of analysis you are actually holding. New accounts get 2 analyses and no card is required.

Analyze a draft