From Boltzmann to Zipf through Shannon and Jaynes

The word-frequency distribution provides the fundamental building blocks that generate discourse in natural language. It is well known, from empirical evidence, that the word-frequency distribution of almost any text is described by Zipf’s law, at least approximately. Following Stephens and Bialek (2010), we interpret the frequency of any word as arising from the interaction potentials between its constituent letters. Indeed, Jaynes’ maximum-entropy principle, with the constrains given by every empirical two-letter marginal distribution, leads to a Boltzmann distribution for word probabilities, with an energy-like function given by the sum of the all-to-all pairwise (two-letter) potentials. The so-called improved iterative-scaling algorithm allows us finding the potentials from the empirical two-letter marginals. We considerably extend Stephens and Bialek’s results, applying this formalism to words with length of up to six letters from the English subset of the recently created Standardized Project Gutenberg Corpus. We find that the model is able to reproduce Zipf’s law, but with some limitations: the general Zipf’s power-law regime is obtained, but the probability of individual words shows considerable scattering. In this way, a pure statistical-physics framework is used to describe the probabilities of words. As a by-product, we find that both the empirical two-letter marginal distributions and the interaction-potential distributions follow well-defined statistical laws.

Statistical Analysis of Word Frequency Distribution in Lithuanian Texts of Different Genres

The paper examines Lithuanian texts of different authors and genres. The main points ofinterest – the number of words, the number of different words and word frequencies. Structural type distributionand Zipf’s law are applied for describing the frequency distribution of words in the text. It is obvious that thelexical diversity of any text can be defined by different words that are used in the text, also called vocabulary.It is shown that the information contained in a reduced vocabulary is enough for dividing the texts analyzedin this article into groups by genre and author using a hierarchical clustering method. In this case, distancesbetween clusters are measured using the Jaccard distance measure, and clusters are aggregated using the Wardmethod.

Quantitative analysis of Zamenhof’s Esenco kaj estonteco

The formation of a language community in a planned language is one of the most important steps in its development. Esperanto is the only fully functional language with relative success among more than one thousand planned language projects. Model texts have played a very significant role in the development of Esperanto. Esenco kaj estonteco de la ideo de lingvo internacia (Essence and future of the international language idea) by the founder of Esperanto, L. L. Zamenhof, is an important document on Esperanto. Taking the essay as the research object, this paper carries through lexical and syntactical analysis on Esperanto by adopting the research methods of quantitative linguistics and complex networks. The results show that the morpheme distribution of Esperanto follows a power law, word frequency distribution fits Zipf’s Law, word length distribution is an exponential curve, and word class distribution obeys linear law. It is also clear that Esperanto is a language with SVO word order preference, the mean dependency distance of Esperanto is 3.85, and the distribution of the dependency distance tends to a minimum, making it typologically a head-middle language. A 43.6% dependency relation appears in adjacent words. The complex syntactic networks of Esperanto display the characteristics of small-world, scale-free networks. All of these quantitative characteristics of Esperanto demonstrate that it is structurally a normal human language.