Corpus linguistics: evidence from texts

1. Ask a question your sample can answer

Before this lesson: None. Outcome: distinguish a collection of texts from evidence for a specified population.

A corpus is a collection of language material assembled for analysis. The plural is corpora. Its usefulness depends on what was collected, what was left out and how the records can be traced. Metadata means information about each text: origin, date, genre and the unit being sampled.

Begin with a narrow question: in our six invented service messages, where does please appear? Do not begin with a claim about all English speakers. Each line below is a separate document. Keep IDs even if you reorder the texts. The genre labels describe an invented task; they are not evidence of how shops or travelers actually speak.

For a real project, define the target population first, such as public English notices at ten named stations during one month. Decide how stations and notices will be selected before searching for your favorite word. Preserve original text separately from cleaned copies; document reuse permission and remove personal data you do not need. Keep duplicate notices identifiable. Ten copies of one sign are not ten independent wording decisions.

AI can help propose a search or inspect code. Generated sentences still need to be labeled synthetic: they cannot demonstrate what real speakers said. A fluent summary cannot replace retrievable examples, source records and sampling decisions.

ID | group | text
A1 | request | Please send the map.
A2 | request | Please check the map.
A3 | request | Send the ticket please.
B1 | notice  | The map is ready.
B2 | notice  | The ticket is ready.
B3 | notice  | Please wait here.

What to expect

You have six documents, two groups of three, and four documents containing please. All are synthetic. This supports a statement about this fixture, with no estimate of English usage outside it.

Watch for: A convenient internet collection is not automatically representative. Changing the sampling rule after seeing the result can favor your preferred answer.

• Lancaster: corpus survey and documented collection designs ↗

2. Make the counting rule explicit

Before this lesson: Lesson 1; save and run a Python file. Outcome: reproduce token, type and document counts.

A token is one occurrence under your chosen segmentation rule; a type is a distinct form. In 'the map the ticket', there are four tokens but three types. A lemma groups inflected forms under a dictionary headword; lowercasing does not perform lemmatization.

For this deliberately simple English fixture, a token is a run of ASCII letters after lowercasing. The regular expression [a-z]+ says one or more letters from a to z. It drops punctuation and splits contractions and hyphenated forms. That is a declared classroom rule, not a multilingual tokenizer. The same method would destroy many accented words and cannot segment Chinese appropriately.

The code keeps documents separate. Counter adds occurrences; set removes repeated forms when counting types. The last calculation asks how many documents contain please, a different measure from how often please occurs. A single document saying please ten times would raise occurrence frequency by ten but document frequency by one.

import re
from collections import Counter

texts = [
    'Please send the map.',
    'Please check the map.',
    'Send the ticket please.',
    'The map is ready.',
    'The ticket is ready.',
    'Please wait here.',
]
docs = [re.findall(r'[a-z]+', text.lower()) for text in texts]
counts = Counter(word for doc in docs for word in doc)
print('tokens', sum(counts.values()))
print('types', len(counts))
print('please', counts['please'])
print('documents', sum('please' in doc for doc in docs))
print('the', counts['the'])

Run it

# Save as counts.py in a new practice folder.
python3 counts.py

What to expect

tokens 23; types 10; please 4; documents 4; the 5. There is no network request and no file modification in the program.

Watch for: Do not silently change punctuation, capitalization or tokenization between datasets. Type counts also depend strongly on sample length; they are not a stand-alone vocabulary-skill score.

• Python: regular expression matching ↗
• Python: Counter ↗

3. Read context and compare rates

Before this lesson: Lessons 1–2. Outcome: connect a rate to its denominator and inspect every match.

A concordance displays occurrences with surrounding context. KWIC means key word in context. The code below prints at most two tokens on each side and retains a document number, so you can reopen the complete sentence. We do not let context cross a document boundary.

Normalization expresses an occurrence count relative to a corpus size. In group A, please occurs 3 times in 12 tokens: 250 per 1,000 tokens. In group B it occurs once in 11 tokens: about 90.91 per 1,000. Multiplication changes the reporting scale; it does not create another 1,000 observed words. Record the raw counts too.

Read each hit before explaining the difference. Our examples place please at the beginning in A1, A2 and B3, and at the end in A3. This is an observation about four occurrences selected by an author. It cannot establish a population preference, politeness ranking or statistical significance. Zero hits in another tiny sample would mean not observed, not impossible.

# Append to the unchanged counts.py from lesson 2.
for number, doc in enumerate(docs, 1):
    for i, word in enumerate(doc):
        if word == 'please':
            left = ' '.join(doc[max(0, i-2):i])
            right = ' '.join(doc[i+1:i+3])
            print(number, left, '[please]', right)
for label, group in [('A', docs[:3]), ('B', docs[3:])]:
    total = sum(len(doc) for doc in group)
    hits = sum(doc.count('please') for doc in group)
    rate = hits / total * 1000 if total else None
    print(label, hits, total, round(rate, 2) if rate is not None else 'undefined')

Run it

python3 counts.py

What to expect

After lesson 2's output, four concordance lines identify documents 1, 2, 3 and 6. The final lines read A 3 12 250.0 and B 1 11 90.91. Document 3 shows 'the ticket [please]' with no right context.

Watch for: A context window can hide negation or quotation. Inspect complete documents. A normalized difference alone supplies neither uncertainty estimates nor a causal explanation.

• AntConc manual: KWIC, file view and word lists ↗
• Corpus statistics: frequency and comparison ↗