Skewing a corpus, and watching what the model does with it

The corpus of this demonstration is written for it, in French, where the noun naming the occupation carries the gender itself, and its make-up is balanced to start with. Each slider sets, for one occupation, the share of sentences whose subject is feminine. At fifty, the corpus says one as often as the other. Move a slider, and you decide what the model will have read.

What the demonstration shows

The result is no surprise at all, and that is exactly what has to be understood. The share of the feminine that the model gives back is the share you set on the slider. Not a nearby value, not a trend: the same value. An n-gram model does nothing but count, so its result is a count.

Two lessons come out of it, and they are not the same lesson.

Bias is a consequence, not an intention

Nobody wrote into the model that an occupation belongs to a gender. The rule appeared in the counts, because it was in the data. That is why looking for intention in a model is a waste of time: the useful question is always what this corpus is made of, and that question is about people who collected, sorted and published data.

What is absent is impossible, not improbable

Take a city out of the corpus, and the model never offers it again. It does not offer it rarely: it cannot offer it, because it has no count at all. No temperature setting, no cut-off, no length of context will bring it back. And notice what this corpus never held: Paris, London, New York, Tokyo are not part of it. A model learnt on this would give the impression of a world where those cities do not exist, without ever flagging the absence.

That is the half of the problem people forget. A bias gets noticed when an answer is wrong. An absence never gets noticed, since there is nothing to look at.

The method

Here is the whole calculation. It can be redone without this page, with a pencil.

1. Build the corpus. For each occupation, write forty short sentences. If the slider is at ninety, ninety per cent of those sentences are « Elle est ingénieure. » and the rest « Il est ingénieur. ». The split is spread out by a whole-number count, never by a draw: the corpus is therefore the same for everyone at the same setting.
2. Cut it up. The text becomes a string of units: « Elle », « est », « ingénieure », “.”. The cutting used here is a cutting into words, more readable than the sub-word cutting of real models, which has a chapter of its own.
3. Count. For every context of two units, record all the units that followed it and how many times. Has the context « Elle est » been followed in eighty sentences? Then its table holds the occupations in the feminine, each with its count.
4. Divide. The probability of a unit after a context is its count divided by the total of the table. That is all the model knows.
5. Draw. To generate text, draw from that table according to those probabilities, with a generator seeded from the seed shown.

At step four, the match between the setting and the result stops being mysterious: a relative frequency is the ratio you set. That is what a model that counts does, and it is the base on which large models add their own mechanisms.

What a real model adds, and what it does not remove

This demonstration is honest on one condition: saying where the analogy stops.

What research has measured

Three published results place what the demonstration lets you feel.

Word embeddings learnt on large corpora of ordinary text, news articles among them, reproduce measurable gender stereotypes, by associating certain occupations more strongly with one gender. A standard statistical model learnt on an ordinary web corpus automatically reproduces human biases already known and measured by psychological tests, including on gender and origin. And the effect does not stay in the text: commercial systems classifying gender from images showed an error rate of up to 34.7% for women with darker skin against 0.8% for men with lighter skin, a gap measured on skin colour and gender combined.

That last figure says what it really costs. A performance gap of that size is not a technical imperfection: it is a service that works for some people and not for others.

The habit to keep

Faced with an answer whose regularity surprises you, do not ask the model why it thinks that: it has no idea, and its explanation is worked out like the rest. Ask what it was learnt on, and who documented that corpus. When the answer to that question does not exist, that is the answer.

Going further

Cutting text into units is covered in the chapter words in pieces. The distribution and the draw from it are covered in the chapter the next word. What becomes of content handed to a service is covered in the chapter what you give it. The situations where the tool becomes a trap are gathered in when not to use it. All the demonstrations on the site are gathered on the demonstrations page, and the contents of the course on the understand page.

Sources

• Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, Adam Tauman Kalai, 2016. Establishes that word embeddings trained on large corpora of ordinary text reproduce measurable gender stereotypes, for instance by associating certain occupations more strongly with one gender than the other.
• Semantics derived automatically from language corpora contain human-like biases Aylin Caliskan, Joanna J. Bryson, Arvind Narayanan, 2017. Establishes that a standard statistical model trained on an ordinary web corpus automatically reproduces human biases already known and measured by psychological tests of the IAT kind, including on gender and race.
• Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification Joy Buolamwini, Timnit Gebru, 2018. Establishes that commercial systems classifying gender from images showed an error rate of up to 34.7% for women with darker skin against 0.8% for men with lighter skin.
• Datasheets for Datasets Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford, 2018. Establishes that a standardised proposal exists for documenting the origin, the composition and the collection method of a dataset, so that an observed bias can be traced back to its origin rather than discovered only once the model is deployed.
• ODERSA synthetic corpus, written for this demonstration ODERSA, 2026, published under the CC BY 4.0 licence. Its default make-up is balanced, its sentences can be read in full on this page, and it describes no real population.