Type Default
Audit

AI-assisted font discovery is no longer hypothetical. Real type catalogs are already being put into conversational interfaces, and AI systems are beginning to mediate how designers find and evaluate type. But before deciding how those systems can be improved, it is worth establishing a baseline for what general-purpose AI does on its own.

Designers have been consistent in saying where they want AI to help. Monotype's 2024 Global Font Use and Forecasting Survey put a 30-question instrument to 4,777 graphic designers, type designers and brand leaders across more than thirteen countries. Ninety-one percent said AI and AI tools can be useful for their work in the type industry. Asked which design applications of AI would be most valuable, respondents named visualizing typography in real-world scenarios and suggesting font pairings at 45 percent each, and improving the discovery of new fonts at 43 percent. Finding the right font was separately the single largest pain point in the same survey, at 23 percent.

The question is not whether these systems can name plausible fonts. They clearly can. The more useful question is what the recommendation space looks like: how concentrated it is, how stable it is across repeated runs, how strongly it reflects existing popularity, how much of a defined type universe it reaches, and how recommendations change when the brief or the candidate universe changes.

I ran 1,890 controlled recommendation calls across 210 design briefs, three models, and three repetitions, producing 9,558 named typeface mentions after normalization. That run is the baseline condition; two further conditions reported later add 1,512 calls.

The result is a baseline for studying AI-assisted type discovery as a system rather than as a collection of individual recommendations. It does not tell us whether a typeface is right for a brief, whether a designer would choose it, or whether AI makes that decision better. Those are different questions. It does establish something more basic: a way to observe and measure the recommendation behavior that future font-discovery systems will need to change.

When the same broad pool of design briefs is put to language models, what properties of a brief are associated with the breadth, distribution and stability of the typeface recommendations produced?

A second question sits behind it, and this audit deliberately does not answer it: do those differences suggest the models are responding to semantic or perceptual characteristics of a brief, or are they mostly reproducing popularity, familiarity and other priors?

Across the full run, the three models named 1,029 distinct typefaces across catalog and non-catalog sources, from 9,558 named typeface mentions. That is the pool everything below ranks.

Of those, 228 are families in the Google Fonts catalog. This figure is observed catalog reach: the families that surfaced under one prompt shape, from 210 briefs, in three repetitions. A larger or differently written brief set would raise it. It is not a measure of what any model holds, and any reading of it as typographic knowledge is a reading the design cannot support.

Within the catalog, recommendations stayed near the most-used end of it: 89.5 percent of catalog recommendations landed in the 300 most-used families, and 60.8 percent in the top hundred.

Those two facts belong together and are easy to state separately and misleadingly. These models explore a large nominal recommendation space, 1,029 distinct names, while directing a large proportion of what they actually say toward a small part of it. Ten names take a quarter of every mention and forty take half. A wide vocabulary and a concentrated output are not in tension, they are the same distribution described at two different points, and only the second is what a designer experiences.

Which raises a question this study poses rather than answers. A large candidate vocabulary is not the same thing as useful discovery, and nothing here establishes that the tail of that distribution contains typefaces a designer would have wanted.

Concentration

source: release/concentration.csv · release/rank_frequency.csv

The ten most-recommended typefaces take 26.0 percent of everything said, led by Source Sans 3, IBM Plex Sans and Source Serif 4. Exactly forty names are required to reach half the total, out of 1,029. Shares are computed over the 9,558 named typeface mentions in the open condition, all three models.

Catalog and non-catalog

The obvious objection to measuring against a free library is that it says nothing about commercial work. That objection would hold if the models stayed inside Google Fonts. They do not.

That figure covers all three models across 1,908 Latin-only recommendation slots. For the two models in the restriction experiment it is 49.3 percent. Both count only names confirmed as non-catalog; unresolved names are excluded rather than assumed to be outside.

Reach is a fraction, and a fraction needs a denominator. Google publishes one: 1,955 families on a stated date. No equivalent list exists for licensed type, so nothing here says whether the non-catalog share represents a wide reach into the commercial world or a narrow one.

The denominator is doing more work than a denominator usually does. A language model asked for typefaces is operating over an effectively open universe with no edges, and almost nothing about that behavior can be measured, because there is nothing to measure it against. A finite, enumerable catalog converts the same question into one that has answers: what fraction of the available set was reached, which parts of it were ignored, whether a constraint moved the answer into a different region. Google Fonts is used here because it is public, enumerable and dated, not because it is the interesting catalog. The measurement is not specific to it, and the same logic runs against any defined library that someone is willing to name.

Every figure here rests on 210 briefs written for this study, so how they were built decides what the numbers can mean.

Each brief carries five tags, and every tag value appears an equal number of times because the grid was built that way rather than sampled. The generator sets the marginals first, so each value gets its exact share of the 210 before any brief is written.

source: release/briefs.json · release/grid.json · release/reweighting_weights.csv

The values are: industry across fintech, healthcare, editorial, gaming, luxury retail, public sector, nonprofit, developer tools, food and education; tone across authoritative, playful, warm, technical, elegant and brutal; medium across website, mobile UI, print editorial, packaging, presentation and long-form reading; script across Latin only, plus Cyrillic, plus Greek, plus Vietnamese and CJK adjacent; specificity across three word-count bands.

Equal marginals do not guarantee that combinations are covered. A set can hold 35 playful briefs and 42 Vietnamese briefs and contain no brief that is both. So the assignment was reordered to spread the pairs, scored by how many pairwise cells sit empty or overloaded. That cost fell from 1297 to 55. There are 347 possible pairwise combinations of tag values across the five dimensions, and all 347 ended up occupied. That is a statement about pairs; it says nothing about whether higher-order combinations are balanced. The seed is 20260831 and is recorded, so the grid rebuilds exactly. A residual cost of 55 means the pairs are occupied rather than equal, so cross-dimension confounding is reduced here, not eliminated.

Balance across the grid reduces structural confounding among the tagged dimensions by distributing their combinations across the brief set. It does not eliminate confounding and it does not establish statistical independence. In particular it says nothing about the semantic signals inside a brief, since one tagged elegant differs from one tagged playful in vocabulary, in register, and often in the project described. No effect below should be attributed to a single signal on the strength of the grid alone.

The balance is deliberate and it is not representative. Latin-only work dominates the commercial practice these briefs stand in for, but every script requirement carries 42 briefs so each slice is measured with equal precision. Any pooled figure needs reweighting before it describes typical practice, and the weights are stored alongside the briefs.

Brief length

The three bands are word counts, not a separate judgment laid on top of them. Across all 210 the range is 9 to 67 words, mean 34.5. Any finding about specificity here can be restated as a finding about length with nothing lost. The bands also overlap at their edges, 24 to 25 words and 42 to 44, so they are not a clean partition. Read this dimension as brief length rather than as a validated specificity construct: word count is what this study varies, and it is narrower than specificity as a design concept, which also covers named constraints, references and prior decisions. The labels vague, moderate and highly specific are retained because the published data files use them. They describe experimental brief-length categories and are not treated as validated measures of specificity.

What the briefs represent, and what they do not

These are not briefs from practice. Each was written by me to instantiate a tag I had already chosen, which means the intended meaning and the words expressing it were produced in the same act by the same person. Nothing in this study establishes what any brief calls for independently of my own labeling.

That limit reaches further than a vocabulary confound. A real brief sits at the end of a chain: a business situation, a person interpreting it, and language describing how the work should be perceived. This study holds only the last link, and that link was authored to match a label. So the question of whether a model responds to design intent or to vocabulary has no referent for intent that can be separated from the vocabulary. Establishing one requires independent designers reading the same briefs and agreeing on what they ask for, which is why a later study exists.

Two kinds of tag

The five dimensions are not five flavours of the same thing. Industry, medium and script describe business context: what the work is for and what it has to do. Tone describes a perceptual target: how the result should be read. Typography is expected to encode the second, mediated by the first.

Neither is a validated representation of design intent. Both are labels I assigned.

The chart below counts distinct catalog families named by the widest and narrowest slice of each dimension, ordered by the ratio between them.

source: release/breadth_by_slice.csv

Breadth is not direction

This exhibit counts families, and a count answers one question. A tone slice returning 140 catalog families against another returning 64 demonstrates different breadth. It does not demonstrate that the two slices reached different typefaces. In principle the 64 could sit inside the 140 and the ratio would look identical.

Breadth tells us how widely the model searches its recommendation space. It does not tell us where in that space it is searching.

Two things follow. Breadth is not direction, and variation is not understanding. A dimension that moves the count further is not thereby a dimension the model handles better.

Establishing direction requires analyzing the identities of the families, not their count, and that analysis has not been run. Two observations in this document do speak to direction, and neither came from this exhibit: elegant briefs sending 47.6 percent of their recommendations outside the catalog toward Minion, Arno and Bembo, and EB Garamond appearing under restriction and never without it. Both are incidental. Directional analysis is a next step, and the harness already stores what it needs.

The findings, stated at the level they support

The tone slices span the widest range of catalog families and the industry slices the narrowest. Tone ranges 64 to 140 across its six values, a ratio of 2.19. Industry ranges 73 to 99 across its ten, a ratio of 1.36. These are descriptive spans, not comparable effect sizes: slice sizes run from 21 to 70 briefs, category counts from three to ten, and distinct-family counts accumulate on a concave curve. Ranking one dimension above another would need every slice rarefied to a common recommendation count and each ratio compared against a null built by randomly partitioning the 210 briefs into groups of the same sizes. That has not been run, and no claim here depends on it.

This measures breadth, not direction. A broader output set does not establish that those faces are more closely aligned with the intended perceptual target, or that the model holds any particular typographic reading of the word "playful."

Script requirement moves the catalog share sharply. On briefs asking for Vietnamese support, 93.2 percent of recommendations resolved to a Google Fonts family, against 42.2 percent on Latin-only briefs. Both are catalog shares, so unresolved names count in neither direction. An earlier draft gave the Vietnamese figure as 94 percent, which was one minus the confirmed-outside share and therefore counted unresolved names as catalog faces. The constraint is visible in catalog reach as well, which is a different measure: 69 percent of the 118 Google Fonts families carrying Greek were named at least once, compared with 4 percent of the 1,944 Latin-carrying families.

Length does not move the answer in one direction. Distinct names rise with it: 446 vague, 491 moderate, 592 highly specific, as longer briefs reach further into licensed type. Catalog families do not. They fall from 163 vague to 139 moderate, then rise again to 144 highly specific, while the top-nine share climbs steadily, 21.5 to 26.5 to 28.0 percent. The catalog-family count is non-monotonic and should not be summarized as narrowing.

These figures describe within-dimension breadth. They should not be read as a ranking of which brief dimension has the greatest effect, because the dimensions differ in slice size and in number of levels. They also pool three models with different breadth and different stability, and no dimension effect is reported per model.

Before asking whether a brief reliably maps to a typographic region, we need to know how much the output moves when the brief does not change at all. Each brief was sampled three times per model. Pairwise overlap between two responses to the same brief runs from 0.41 to 0.56 by Jaccard.

source: release/model_breakdown.csv · release/null_overlap.json

Under this sampling protocol, repeated runs of the same brief do not return a fixed recommendation set. How much of that is the model and how much is the sampling configuration is not separated here, which is why the protocol is named in the claim.

Three kinds of stability are worth keeping apart, and this study measures one. Exact-name retention is what the table reports: how often the same five names come back. Distributional stability would ask whether the underlying probabilities over typefaces hold steady even when the sampled names differ. Semantic stability would ask whether the typographic character of the answer holds even when both the names and the probabilities move. Low exact-name overlap does not by itself mean there is no stable underlying preference: a model sampling from a steady distribution over forty plausible faces will return a different five every time. Nothing here separates that case from an unstable one, and nothing here measures semantic stability at all.

It also constrains study design. Any future test of whether two differently worded briefs produce different recommendations has to clear this noise floor first, and these numbers are what set it.

Defining retention

For three five-item responses A, B and C to the same brief from the same model, the primary stability metric is mean pairwise Jaccard similarity

J(A,B) = |A ∩ B| / |A ∪ B|

averaged across the three pairwise comparisons and then across briefs. The 0.41, 0.50 and 0.56 figures are that statistic. Intersection over five is a secondary sensitivity measure at 0.55, 0.64 and 0.66, and catalog-only Jaccard at 0.50, 0.62 and 0.65. A third statistic, the intersection of all three responses, averages 2.49 names of five; it is reported here once and used nowhere else, because pairwise Jaccard is the primary construct throughout and the two should not be mixed in one argument.

Model differences

The families-named and retention columns are not independent, and the dependence runs one way. A model that answers differently each time accumulates more distinct names over a run for that reason alone, so part of Gemini's wider range is its own instability rather than broader reach. The families-named column is uncorrected here, while the restriction analysis applies exactly the correction that would be needed. A reader is entitled to hold this column to the weaker standard.

Sampling configuration is a second confound on this column specifically, since temperature was set explicitly for two providers and assumed for the third, and retention is the measure most sensitive to it.

Twelve of the twenty-five most-recommended typefaces are common to all three models. That sounds like agreement until it is compared against something. Pooling every recommendation the three models made and dealing it back out into three sets of the same sizes, so all three have identical taste by construction and differ only in what each happened to draw, those sets shared about twenty of their top twenty-five, and never fewer than sixteen across two thousand attempts. The real models share twelve.

2,000 draws with replacement from the pooled empirical distribution, each model’s recommendation count preserved, seed 20260902 · source: release/null_overlap.json

Under the specified null, if the three models were drawing from the same pooled recommendation distribution and differed only through sampling, substantially higher overlap would be expected. Across 2,000 simulations the null mean was 20.1, the range 16 to 23, the 95 percent interval 18 to 22, and the probability of an overlap at or below 12 under 0.0005. The null draws with replacement from the pooled empirical distribution while preserving each model's recommendation count, seed 20260902. The result rejects the specific null of a single shared recommendation distribution with sampling variation. It does not identify the cause of the observed overlap.

One way to test whether recommendation frequency is strongly associated with prior familiarity is to compare it with how heavily each family is used in the catalog.

source: release/rank_frequency.csv · release/usage_correlation.csv

The case pairs are the firm part of this analysis and they do not depend on any correlation statistic. Raleway is the fifteenth most-used family in the library and was recommended three times. Source Serif 4 sits at usage rank 131 and was recommended 296 times. Literata at 196 was recommended 215 times. All three are free, so availability does not separate them.

Recommendation frequency is positively associated with prior usage frequency under this measure, but usage frequency alone does not account for the observed recommendation distribution. These cases show that catalog usage rank is not sufficient to determine recommendation frequency. They show nothing about what the ordering does follow. Catalog prevalence, recommendation frequency, model familiarity and semantic fit are four different things, and this analysis separates none of them.

The usage ranking itself needs care, because it records Google Fonts usage rather than a direct measure of designers' typeface preferences. Google Sans at rank 2 is Google's own brand face, Material Icons at 10 is not a text face, and Arimo at 11 is an Arial substitute picked for metric compatibility. None is something a designer would put on a client's brand, so their absence from the recommendations says nothing. All three are omitted from the chart above for that reason and included in the correlation below, which is why the two are not two views of one computation. Raleway and Source Serif 4 are ordinary text faces, which is why they make the cleaner comparison.

Disclosure

I previously ran public relations at Adobe for several typefaces appearing in these results. Source Sans 3 and Source Serif 4 descend from Source Sans and Source Serif. Noto Sans JP is Source Han Sans under its Google name. The share of recommendations falling to families from that work is recoverable from the released rank-frequency file, and is not restated here.

This is a disclosure about my relationship to some of the outputs. It is not evidence that the press cycle put those families where they are, and nothing in the analysis is adjusted for it. A reader weighing any interpretation about professional discourse should know it.

One sentence in the prompt moves the distribution. A grammatically identical sentence does not.

The same briefs went back to Claude and ChatGPT with one sentence added: recommend only typefaces available in Google Fonts. For the comparison below, only recommendations resolving to catalog families are counted, so removing licensed options cannot itself produce the difference. With the placebo control below it, this is the part of the study that directly manipulates a constraint and includes a control. It is also the section where the analysis moves from counts to the identities of the families that changed.

distinct catalog families on matched draws of 564 recommendations · Latin-only briefs, Claude and ChatGPT · source: release/restriction.csv

The right way to read it is as a constraint response. When the most available options are removed, how does the recommendation distribution change?

source: release/restriction.csv

The raw counts are not comparable. The open condition produced 1,260 recommendation slots and 1,278 named rows once compound fields are split, of which 564 were catalog faces. The restricted condition produced 1,260 rows and slots with no compound fields, of which 1,255 were catalog faces. More draws find more distinct families whatever the underlying behavior. Drawing the restricted run down to the same 564 recommendations gives a mean of 94.2 families against 73, an increase of 29.1 percent, with a 95 percent interval of 88 to 101. The top-five share is a proportion rather than a count and is unaffected by the correction; after rarefaction it holds at 25.1 percent.

Novel families are a separate count and a larger one. Across matched draws the restricted run named a mean of 34.5 families the open run did not, 95 percent interval 29 to 41, and 52 across the full restricted set. Eight families went the other way, appearing openly and not under restriction. The net difference of 21 is the difference between those two flows, not a count of new names.

The sharpest case is briefs asking for an elegant Latin text face. Open, the eight most common answers were Garamond Premier Pro, Minion Pro, Arno Pro, Bembo, Canela, Optima, Garamond and Baskerville: every one licensed, not one in the catalog. Restricted, the same briefs returned Cormorant Garamond, EB Garamond, Libre Baskerville, Crimson Pro, Spectral and Playfair Display. EB Garamond was named zero times across the open Latin recommendations and 35 times across the restricted ones.

Stated exactly: under this protocol, adding the catalog constraint changes the recommendation distribution. It produces more distinct catalog families in matched draws and surfaces families that do not appear in the open condition.

The useful consequence is methodological. Pointing a model at a defined type universe is not only an implementation choice, it is an experimental variable with a measurable effect, and one that a placebo shows is specific to what is constrained. So the question worth asking about a catalog-grounded system is not whether a model can search a catalog, which is uninteresting, but what access to a defined universe changes about the recommendations a designer actually receives. That question has a number attached to it, which is the only reason it is worth asking this way.

The placebo control

The obvious objection is that the sentence works because it is a constraint, not because of what it constrains. A control was preregistered before it ran, with the decision rule fixed in advance, and the placebo sentence was recommend only typefaces that include a bold weight. It is a genuine, checkable set-constraint of the same grammatical form as the catalog sentence, on a property unrelated to licensing or catalog membership, and it eliminates essentially nothing, since almost every family a model would name has a bold weight. The same 42 Latin-only briefs went to the same two models at the same settings, 252 calls.

preregistration: preregistration.md · result: placebo_result.md

The placebo did not widen the set. Its matched-draw mean was 72.7 families against 73 in the open condition, while the catalog restriction produced 94.2. The top-five share barely moved, 37.4 to 36.9 percent, against 37.4 to 24.7 under the catalog sentence. Its novel families, a mean of 11.8 on matched draws and 13 in the observed set, are ordinary substitutions such as PT Sans and Cardo, and every family it shared with the open condition also appears in the restricted run. It found nothing the catalog constraint did not.

Three manipulation checks ran before the breadth analysis. The model registered the constraint, mentioning weight in 49.9 percent of its reasons against 25.4 percent in the open condition. The confirmed non-catalog share fell 4.4 points rather than staying flat, so the sentence did remove a little licensed type, about a ninth of what the catalog sentence removed, which biases the placebo toward widening rather than away from it. And seven of the 75 recommended families carry no bold weight in the catalog record, a compliance rate of 90.7 percent that does not affect a count of families.

One check behaved worse than expected and it is worth recording. The catalog-restricted run mentions Google Fonts in only 2.6 percent of its reasons while having driven the non-catalog share from 49.3 percent to 0.1 percent. A model can obey a constraint completely without discussing it, so a low mention rate is not evidence that a sentence was ignored. The check is informative in one direction only.

Two things this does not show

A constraint alone did not widen the set. The worry was that adding any rule converts an open generation task into a constraint-satisfaction task, and that the conversion rather than the catalog does the work. The placebo above tests exactly that, and provides no evidence that the widening observed under the catalog constraint is caused simply by adding a rule. A registered, grammatically matched rule produced no widening at all.

It may not be specific to this catalog. The catalog sentence names an enumerable list. The placebo names a property. A model may respond differently to a list than to a predicate, and that difference is specificity of reference rather than constraint satisfaction. So the result supports an effect that is catalog-specific or named-list-specific, and the two are not separated here. Distinguishing them needs a further arm naming a different finite catalog.

Licensing removal is untested. The catalog sentence deletes Minion, Sabon and Graphik from the answer space and the model must substitute, which could produce the widening on its own. That would refine the mechanism rather than overturn the finding, since the models held those families either way, but the reason for the withholding would change. The next control is recommend only typefaces that are free for commercial use, which constrains on licensing without naming a catalog.

And it is not evidence that the wider set is better. Nothing here scores a recommendation, so a sentence that produces a third more of the catalog produces more, and only more.

Claims about this kind of behavior run from very weak to very strong, and most of the available vocabulary is stronger than the evidence. The ladder below marks the boundary.

The audit sits on rung two. It provides the baseline needed to investigate whether observed recommendation differences reflect meaningful sensitivity to briefs, candidate universes and constraints, and eventually whether those differences improve human selection.

It does not establish rungs four through eight.

Read downward the ladder is a claim boundary. Read upward it is a sequence of experiments, each of which has to succeed before the next is worth running. Output variation is trivially established. Association between brief and output is what this study measures. Whether those associations reflect meaningful typographic characteristics is untested, as is whether they hold under repeated sampling, whether they survive paraphrase and a change of vocabulary, whether the recommendations are useful to anyone, and whether they improve an actual selection. Each of those is a separate study and none of them has been run. The ladder is short, but the distance between rungs two and eight is where most of the unresolved work lies.

Recommendation. A model produces a set of plausible typeface candidates in response to a brief.

Selection. A person or system chooses among a defined candidate set according to an explicit criterion.

This audit measures recommendation. It has no candidate set, because the models supply their own, and no criterion, because nothing here scores an answer. A model producing a typeface that seems appropriate is not, by itself, evidence that it can reliably select type, and the two words are not interchangeable anywhere in this document.

Three consequences follow, and they constrain every number below. Recommendation frequency is not evidence of design quality, because nothing here asks whether a face was any good. A model returning a familiar typeface is not thereby making a poor recommendation, since the familiar face is often the right one. And what this study characterizes is the behavior of an automated recommendation system, not the quality of the decisions a designer makes with it.

Which locates the question this work does not reach. The interesting question is not which typefaces an automated system recommends. It is whether those recommendations help a designer arrive at a better decision, and answering that requires designers, tasks and outcomes that this design does not contain.

That gap is not a minor omission at the edge of the method. It is the boundary between a system that can generate plausible typeface names, which is now ordinary, and a system that is useful inside professional design work, which is not established for any of the three tested here. Nothing in this study should be read as evidence that better recommendation behavior produces better selections. That is the proposition to test, and testing it is a different experiment from this one.

The recommendation space changes with the brief. This audit does not say why.

A brief carries several signals at once. It describes a business, a medium, an audience, a desired emotional response, and it does all of that in words that carry their own aesthetic freight. Those signals usually point the same way in a real brief, which is why an observational study cannot separate them. A brief tagged playful also contains playful adjectives. When breadth changes, any of those signals could be responsible, and this design cannot say which.

Which points at what this line of work is actually about. The difficulty in AI-assisted font discovery may not be generating enough recommendations, since these systems already generate a large and structured space of them. The harder problem is turning that space into one a designer can trust, interrogate and act on. This audit finds a large, structured and unstable recommendation space, and finds brief characteristics associated with how that space is explored. Whether those associations amount to anything like semantic matching is exactly what it does not establish.

AI-assisted font discovery is no longer hypothetical. Purpose-built systems are already putting real type catalogs into conversational interfaces: Monotype released a MyFonts app for ChatGPT in May 2026, covering more than 250,000 fonts across 300 languages and returning licensable recommendations inside the conversation, as a United States beta. That makes the next research question less about whether such systems can recommend fonts and more about what changes when they do.

Several questions follow directly, and none of them is answered here. Each is a next-study question rather than a finding.

• Does catalog grounding improve relevance?
• Does grounding reduce generic or unavailable recommendations?
• Does it increase long-tail discovery?
• How sensitive are recommendations to intent and constraints?
• Do recommendations actually improve designer decisions?
• What is the tradeoff between speed and quality?

The second and third are testable with the harness as it stands, because availability and long-tail reach are both properties of a catalog. The first and fourth need a way to score relevance and intent that this study does not have. The last two need designers and outcomes rather than briefs and counts.

Underneath the list there are three separable problems, and conflating them is what makes assisted font discovery sound either solved or hopeless depending on which one is being discussed.

It does not measure selection, and the work that matters is connecting discovery to it.

The next question is whether the behavior observed here changes when the recommendation system has access to a defined type universe.

The first contribution of this work is not a list of good or bad typefaces. It is a way to observe and measure what an automated font-discovery system actually does. Concentration, catalog reach, popularity relationship, constraint response and run-to-run stability are all quantities that can be computed from stored responses, recomputed by someone else, and compared across systems that work in completely different ways.

Once that behavior is measurable, the same framework can compare systems rather than describe one. The ungrounded general-purpose model is the baseline measured here. The next comparison is a system grounded in a defined type catalog; after that, a retrieval-augmented recommender, a purpose-built font-search experience and, ultimately, a human designer can be evaluated on the same axes. The question is not whether one system produces "better fonts" in the abstract. It is what changes in the recommendation space when different kinds of knowledge and interaction are introduced.

None of these measurements was the point of the study. They exist because the study could not make a responsible claim without them. Reporting concentration required a denominator, which required an enumerable catalog. Reporting that a constraint widened the answer required a matched-draw correction, because more draws find more names whatever the system is doing. Reporting a difference between models required knowing how much an identical brief moves on repetition. Each measure was built to stop a specific wrong conclusion.

The consequence is that the measurements are more durable than the thing measured. Nothing in them depends on the answer coming from a language model. A retrieval system over a type catalog, a purpose-built recommender, a search product, a hybrid, or a human designer given the same briefs all produce a set of names per brief, and all of them can be scored on the same axes. The models tested here will change. The question of whether a discovery system reaches past the obvious will not.

Seven dimensions fall out of what this audit had to measure in order to say anything at all.

Relevance. Does the recommendation fit the stated design intent? Nothing in this audit measures it, because nothing here scores an answer against a brief.

Constraint fidelity. Does the system honour explicit requirements, whether script, medium, industry, tone, licensing or catalog availability? The restriction experiment is a first probe at one constraint, and the placebo control shows the response is specific to what is constrained rather than to the presence of a rule.

Discovery. Does the system surface appropriate typefaces beyond the obvious defaults? Catalog reach and long-tail behavior are the observable proxies, and this audit finds both narrow.

Diversity. Does it offer meaningful alternatives, or return a small set repeatedly? Concentration and distinct-family counts measure this directly.

Stability. Does it behave consistently enough to be trusted? Repeated sampling of an identical brief answers this, and the answer here is that agreement between two runs sits between 0.41 and 0.56 by Jaccard.

Intent sensitivity. Do meaningful changes in a brief produce meaningful changes in the recommendations? Breadth by brief dimension is a first approximation, and a weak one, because it counts how many families appear rather than which.

Human utility. Does the recommendation improve the designer's workflow or decision? This audit cannot address it, and no measurement of the other six substitutes for it.

What a useful system would have to optimize

These dimensions cannot be optimized independently, and most of them have a degenerate solution that scores perfectly and helps nobody.

• Perfect stability is returning the same five typefaces forever.
• Perfect diversity is returning arbitrary ones.
• Perfect constraint fidelity can be bought by collapsing discovery to whatever trivially satisfies the constraint.
• Maximum novelty reduces relevance, because most of the catalog is wrong for most briefs.
• Maximum catalog coverage increases availability while saying nothing about whether the covered families were worth surfacing.
• Relevance without diversity reinforces the defaults the designer already knows.
• Diversity without human utility is noise presented as choice.

So the problem is not building a single score. A composite number would hide exactly the conflicts that matter, and any system can be tuned to win on one axis by giving up another. The useful question is which region of this space a system occupies and which tradeoffs it has made, which means an evaluation has to report the dimensions together and name the conflicts rather than average them away.

Why the candidate universe matters

A general-purpose model recommends from an effectively open universe. A grounded system recommends from a defined and inspectable one, and that difference is what makes availability, licensing, script support, long-tail reach and substitution behavior measurable at all. The catalog is not merely a source of answers. It becomes part of the experimental apparatus.

This study uses Google Fonts as that apparatus because it is public, enumerable and dated. The same methodology runs against any other defined catalog, and the numbers are comparable only within whichever catalog was used.

What a serious evaluation would require

Measuring these dimensions at the level of a real product needs a defined catalog with accurate metadata, licensing and script coverage recorded per family, retrieval to ground against, the surfaces where recommendations are actually delivered, and working designers with real briefs and observed outcomes. Little of that is available to a general research setting, which is part of why this area is under-measured relative to how much is claimed about it.

One requirement sits in tension with the rest. An evaluation program should, where possible, preserve independence between the party building the discovery system and the party evaluating it; otherwise the distinction between product validation and marketing becomes difficult to maintain. A program that had the catalog, the designers and the surfaces would still need its measurements defined in advance, its negative results published, and its instrument open to inspection from outside. That is solvable, but it has to be solved deliberately rather than assumed.

The audit establishes the baseline. The next stage is to determine whether grounding, retrieval, interface design and human interaction can turn that baseline into a more useful form of type discovery, and the only way to know is to measure the same things again on systems built differently.

The evaluation program

Three proposed studies, none of them run. They are stated here to show what the baseline makes possible, not as results. The sequence runs from system behavior to recommendation utility to workflow outcome, and each stage is only worth attempting if the one before it produced a difference worth chasing.

Study 1. Grounded against general recommendation. Put the same brief set to four sources: a general-purpose model as measured here, a model grounded in a real and finite type catalog, a purpose-built font search or recommendation system, and human designers where that is feasible. Score every source on the same axes: relevance, catalog validity, diversity, concentration, popularity relationship, long-tail discovery, constraint fidelity, intent sensitivity, run-to-run stability and usefulness. The point is to measure what changes, not to confirm that grounding helps. A grounded system could plausibly return more valid names and less interesting ones, and the design has to be able to detect that outcome rather than assume it away.

Study 2. Recommendation against human selection. Move from what a system produces to what a designer does with it. Give working designers the same briefs and observe time to shortlist, number of candidates considered, confidence in the choice, quality of the final selection under independent review, satisfaction, perceived usefulness, and whether any unfamiliar but appropriate typeface was discovered. This is where recommendation metrics either earn their keep or fail to, since a system can look better on every axis in Study 1 and change nothing about the decision.

Study 3. Designer and system together. Test whether assistance changes the process rather than the output list. The question becomes whether assistance makes a designer more effective at finding and choosing type.

This audit and the immediate control are the measurement stage. Everything after them is proposed, and none of the outcomes is known. What a system does in a designer's hands will tell you something your measurements did not catch, and the framework has to absorb that rather than survive it. This is not intended as a benchmark to be run once and treated as a permanent score.

The research does not end when the baseline is established. The baseline is not an answer to the problem of assisted type discovery. It is what makes that problem measurable, and until behavior is measurable there is nothing to compare a change against.

Brief:
{brief}

Recommend exactly five typefaces for this brief.{condition}

Respond with only a JSON object in this exact shape and
nothing else:
{"recommendations": [{"font": "...", "reason": "..."}]}

Each reason should be one sentence.