The Distribution of City Sizes in Turkey: A Failure of Zipf’s Law Due to Concavity

Types, Tokens, and Hapaxes: A New Heap’s Law

Glottotheory ◽

10.1515/glot-2018-0014 ◽

2019 ◽

Vol 9 (2) ◽

pp. 113-129

Author(s):

Victor Davis

Keyword(s):

First Principles ◽

Scaling Law ◽

Written Language ◽

Zipf’S Law ◽

Academic Press ◽

Zipf's Law ◽

New Words ◽

Size Dependent ◽

Link Type ◽

Combinatorial Model

Abstract Heap’s Law https://dl.acm.org/citation.cfm?id=539986 Heaps, H S 1978 Information Retrieval: Computational and Theoretical Aspects (Academic Press). states that in a large enough text corpus, the number of types as a function of tokens grows as N = K{M^\beta } for some free parameters K, \beta . Much has been written http://iopscience.iop.org/article/10.1088/1367-2630/15/9/093033 Font-Clos, Francesc 2013 A scaling law beyond Zipf’s law and its relation to Heaps’ law (New Journal of Physics 15 093033)., http://iopscience.iop.org/article/10.1088/1367-2630/11/12/123015 Bernhardsson S, da Rocha L E C and Minnhagen P 2009 The meta book and size-dependent properties of written language (New Journal of Physics 11 123015)., http://iopscience.iop.org/article/10.1088/1742-5468/2011/07/P07013 Bernhardsson S, Ki Baek and Minnhagen 2011 A paradoxical property of the monkey book (Journal of Statistical Mechanics: Theory and Experiment, Volume 2011)., http://milicka.cz/kestazeni/type-token_relation.pdf Milička, Jiří 2009 Type-token & Hapax-token Relation: A Combinatorial Model (Glottotheory. International Journal of Theoretical Linguistics 2 (1), 99–110)., https://www.nature.com/articles/srep00943 Petersen, Alexander 2012 Languages cool as they expand: Allometric scaling and the decreasing need for new words (Scientific Reports volume 2, Article number: 943). about how this result and various generalizations can be derived from Zipf’s Law. http://dx.doi.org/10.1037/h0052442 Zipf, George 1949 Human behavior and the principle of least effort (Reading: Addison-Wesley). Here we derive from first principles a completely novel expression of the type-token curve and prove its superior accuracy on real text. This expression naturally generalizes to equally accurate estimates for counting hapaxes and higher n-legomena.

Download Full-text

Dynamical approach to Zipf's law

Physical Review Research ◽

10.1103/physrevresearch.3.013084 ◽

2021 ◽

Vol 3 (1) ◽

Author(s):

Giordano De Marzo ◽

Andrea Gabrielli ◽

Andrea Zaccaria ◽

Luciano Pietronero

Keyword(s):

Zipf’S Law ◽

Zipf's Law ◽

Dynamical Approach

Download Full-text

Optimization of morpheme length: a cross-linguistic assessment of Zipf’s and Menzerath’s laws

Linguistics Vanguard ◽

10.1515/lingvan-2019-0076 ◽

2021 ◽

Vol 7 (s3) ◽

Author(s):

Matthew Stave ◽

Ludger Paschen ◽

François Pellegrino ◽

Frank Seifart

Keyword(s):

Structural Information ◽

Unit Length ◽

Zipf’S Law ◽

Zipf's Law ◽

Linguistic Structure ◽

Morphological Complexity ◽

Linguistic Assessment ◽

Linguistic Units

Abstract Zipf’s Law of Abbreviation and Menzerath’s Law both make predictions about the length of linguistic units, based on corpus frequency and the length of the carrier unit. Each contributes to the efficiency of languages: for Zipf, units are more likely to be reduced when they are highly predictable, due to their frequency; for Menzerath, units are more likely to be reduced when there are more sub-units to contribute to the structural information of the carrier unit. However, it remains unclear how the two laws work together in determining unit length at a given level of linguistic structure. We examine this question regarding the length of morphemes in spoken corpora of nine typologically diverse languages drawn from the DoReCo corpus, showing that Zipf’s Law is a stronger predictor, but that the two laws interact with one another. We also explore how this is affected by specific typological characteristics, such as morphological complexity.

Download Full-text