Compare Corpus Statistics

Welcome to the webpage comparing selected Slovak corpora..

The webpage is here.

Work in progress.

This is a tool for exploring and comparing statistical properties of Slovak text corpora. It helps users examine corpus size, vocabulary richness, writing style, text structure, word frequencies, and linguistic features.

Getting Started

Open the web interface and select one or more corpora using the checkboxes at the top of the page. Available corpora include the Slovak National Corpus, Araneum web corpora, manually annotated texts, journalistic corpora, legal texts, and synthetic texts. The corresponding statistical results appear below the selection panel.

Each section presents measurements describing a different property of the selected corpus. Where available, click help to expand explanations. Many results are ratios or percentages, making them more useful for comparing corpora of different sizes.

Longest Words

Top 10 longest words shows the longest words in the corpus, measured by the number of characters.

Corpus Size and Structure

lines is the number of lines in the raw corpus data, including structural annotations. This measures the size of the corpus.

tokens is the total number of text units (tokens), like words, numbers and punctuation characters. It is the main measure of corpus size.

types counts distinct tokens rather than their repetitions. For example, if the word "dom" appears a thousand times, it contributes one distinct token type. This provides a basic indication of vocabulary variety.

docs is the number of separate documents in the corpus. Documents might correspond to articles, books, or other texts, depending on the corpus.

paragraphs counts paragraphs in the corpus. Together with the number of documents and sentences, it describes how the material is structurally divided.

sentences is the number of sentences identified in the corpus. Comparing sentences with tokens gives a rough idea of average sentence length.

Stylometry

Stylometry describes measurable features of writing style. These indicators help characterize differences in vocabulary, grammatical preferences, and the relative complexity or descriptiveness of texts.

TTR (Type-Token Ratio) measures vocabulary diversity by dividing the number of distinct tokens by the total number of tokens. Higher values suggest greater variety and less repetition. TTR is strongly affected by corpus size, comparisons are most meaningful between similarly sized corpora.

hapax legomena counts words that occur exactly once in the corpus (case insensitive).

HTR (Hapax-Token Ratio) divides the number of hapax legomena by the total number of tokens.

character entropy (Shannon) measures how unpredictable the next character is, based on the frequency distribution of characters. Higher entropy means that character frequencies are more evenly distributed, while lower entropy indicates that relatively few characters dominate.

token entropy measures the unpredictability of tokens based on their frequency distribution. Higher values indicate a more diverse or evenly distributed vocabulary, while lower values suggest that a smaller set of tokens accounts for much of the text.

activity measures the relative importance of verbs compared with adjectives and participles. It is calculated as verbs / (verbs + adjectives + participles). Higher values suggest a more action-oriented style, whereas lower values indicate a stronger emphasis on describing things.

descriptivity is the complement of activity, calculated as 1 - activity. Higher values indicate that adjectives and participles are relatively more prominent than verbs, suggesting a more descriptive style.

attributivity measures the tendency of a text to describe entities through their attributes. Higher values indicate richer or more detailed descriptions of things, rather than merely naming them or reporting events.

subjectivity measures a grammatical stylistic tendency related to expressing sentence subjects explicitly. Higher values are associated with more formal writing, where subjects tend to be stated rather than omitted. Lower values are more typical of informal or speech-like texts. Despite the name, this indicator does not measure personal opinions or emotional bias.

Length Statistics

This reports the distribution of lengths for documents, paragraphs, and sentences. Lengths are measured in tokens. The same statistical parameters appear in each table.

minimum is the length of the shortest unit. For sentence statistics, for example, it identifies the shortest sentence found in the corpus.

median is the middle length when all units are sorted from shortest to longest. Half the units are shorter and half are longer. Unlike the mean, the median is relatively unaffected by a few exceptionally long texts.

maximum is the length of the longest unit. It helps identify unusually long documents, paragraphs, or sentences.

mean is the arithmetic average length. For sentences, it provides an estimate of how many tokens a sentence contains on average. A small number of very long units can increase this value substantially.

stdev (Standard Deviation) measures how widely lengths vary around their average. A small value indicates fairly uniform lengths, while a large value means that the corpus contains a mixture of short and long units.

total is the number of units included in the calculation, such as the number of documents, paragraphs, or sentences. It is not the sum of their lengths.

Punctuation Statistics

These measurements describe the spacing of punctuation throughout the corpus. They provide an additional indication of how text is structured and segmented.

Distances between punctuation marks measures the number of tokens from one punctuation mark to the next. Two adjacent punctuation marks have distance 1, while punctuation marks separated by one word have distance 2. Shorter distances generally indicate more frequent punctuation.

Distances between dots or commas applies the same measurement specifically to full stops and commas. It provides a narrower view of punctuation spacing, which can help characterize clause and sentence structure.

Both tables report minimum, median, maximum, mean, standard deviation, and total number of measured distances, using the definitions given above.

Keywords

The keyword section identifies words that are unusually frequent in the selected corpus compared with a reference corpus. This uses the Simple Maths method, with the combined web corpus Araneum + HPLT + FineWeb2 serving as the reference.

lemma is the word in the basic form.

score indicates how strongly a lemma is associated with the selected corpus relative to the reference corpus. Higher scores indicate more distinctive vocabulary.

cnt_foc is the number of occurrences of a lemma in the selected (focus) corpus. It gives the actual frequency before adjustment for corpus size.

cnt_ref is the number of occurrences of the same lemma in the reference corpus. Comparing it with the focus count helps establish whether a word is unusually common in the selected material.

ppm_foc is the frequency of a lemma per million tokens in the focus corpus. Normalization makes frequencies easier to compare between corpora of different sizes.

ppm_ref is the corresponding frequency per million tokens in the reference corpus. A substantially higher focus frequency indicates that the lemma is relatively characteristic of the selected corpus.

smoothing N=1 gives relatively high prominence to rare and highly distinctive words. It is useful for identifying specialized vocabulary and unusual expressions, but may also highlight proper names or accidental occurrences.

smoothing N=10 gives somewhat greater weight to more established words. It provides a compromise between identifying rare distinctive words and emphasizing words with more substantial frequencies.

smoothing N=1000 strongly reduces the influence of rare words and brings common vocabulary. This is useful for finding broader differences in language use, including preferences for frequent verbs, pronouns, and other everyday words.

Click Show more... to expand a keyword table beyond its initial entries.

Part-of-Speech Distribution

POS (Part of Speech) describes the composition of the corpus in terms of parts of speech such as nouns, verbs, adjectives, pronouns, prepositions, etc. The tag column contains the annotation tag, count gives the number of occurrences, and [%] gives the percentage of all tokens. This helps compare grammatical preferences, such as the relative importance of nouns in technical writing or verbs in narrative text.

Rhythmic Rule and Word Properties

This section describes parameters connected to the Slovak rhythmic shortening rule. The rule generally discourages consecutive long syllables in Slovak words, although many exceptions exist.

The statistics are expressed as percentages relative to the number of Slovak-like words, defined as words containing only characters from the Slovak alphabet. Thus, the denominator is not the total number of corpus tokens.

Non-words measures the relative number of elements that are not ordinary words, such as punctuation marks and digits. Higher values indicate a greater amount of non-lexical material relative to Slovak-like words.

Foreign words measures the relative number of words containing characters outside the Slovak alphabet. It provides a rough indication of foreign-language elements.

CIA words measures the relative number of words ending in -cia with preceding long vowel, and their inflected forms.

Violations measures the relative number of words that violate the Slovak rhythmic rule. Higher values indicate that such patterns occur more frequently relative to Slovak-like words.

This work has been supported by the project APVV-22-0370 Language and Communicative Aspects of Disinformation.