word frequency distribution
Recently Published Documents


TOTAL DOCUMENTS

15
(FIVE YEARS 0)

H-INDEX

4
(FIVE YEARS 0)

Entropy ◽  
2020 ◽  
Vol 22 (2) ◽  
pp. 179 ◽  
Author(s):  
Álvaro Corral ◽  
Montserrat García del Muro

The word-frequency distribution provides the fundamental building blocks that generate discourse in natural language. It is well known, from empirical evidence, that the word-frequency distribution of almost any text is described by Zipf’s law, at least approximately. Following Stephens and Bialek (2010), we interpret the frequency of any word as arising from the interaction potentials between its constituent letters. Indeed, Jaynes’ maximum-entropy principle, with the constrains given by every empirical two-letter marginal distribution, leads to a Boltzmann distribution for word probabilities, with an energy-like function given by the sum of the all-to-all pairwise (two-letter) potentials. The so-called improved iterative-scaling algorithm allows us finding the potentials from the empirical two-letter marginals. We considerably extend Stephens and Bialek’s results, applying this formalism to words with length of up to six letters from the English subset of the recently created Standardized Project Gutenberg Corpus. We find that the model is able to reproduce Zipf’s law, but with some limitations: the general Zipf’s power-law regime is obtained, but the probability of individual words shows considerable scattering. In this way, a pure statistical-physics framework is used to describe the probabilities of words. As a by-product, we find that both the empirical two-letter marginal distributions and the interaction-potential distributions follow well-defined statistical laws.


2018 ◽  
Vol 30 (05) ◽  
pp. 283-290
Author(s):  
VASTOSHPATI SHASTRI ◽  
◽  
RAKESH RANJAN ◽  
PRAVEEN KUMAR TRIPATHI ◽  
S.K UPADHYAY, ◽  
...  

2016 ◽  
Vol 55 (1) ◽  
pp. 61-69
Author(s):  
Neringa Bružaitė ◽  
Tomas Rekašius

The paper examines Lithuanian texts of different authors and genres. The main points ofinterest – the number of words, the number of different words and word frequencies. Structural type distributionand Zipf’s law are applied for describing the frequency distribution of words in the text. It is obvious that thelexical diversity of any text can be defined by different words that are used in the text, also called vocabulary.It is shown that the information contained in a reduced vocabulary is enough for dividing the texts analyzedin this article into groups by genre and author using a hierarchical clustering method. In this case, distancesbetween clusters are measured using the Jaccard distance measure, and clusters are aggregated using the Wardmethod.


2016 ◽  
pp. fqw005
Author(s):  
Daniel Gamermann ◽  
Carmen Moret-Tatay ◽  
Esperanza Navarro-Pardo ◽  
Pedro Fernandez de Córdoba Castellá

2011 ◽  
Vol 35 (1) ◽  
pp. 57-81 ◽  
Author(s):  
Haitao Liu

The formation of a language community in a planned language is one of the most important steps in its development. Esperanto is the only fully functional language with relative success among more than one thousand planned language projects. Model texts have played a very significant role in the development of Esperanto. Esenco kaj estonteco de la ideo de lingvo internacia (Essence and future of the international language idea) by the founder of Esperanto, L. L. Zamenhof, is an important document on Esperanto. Taking the essay as the research object, this paper carries through lexical and syntactical analysis on Esperanto by adopting the research methods of quantitative linguistics and complex networks. The results show that the morpheme distribution of Esperanto follows a power law, word frequency distribution fits Zipf’s Law, word length distribution is an exponential curve, and word class distribution obeys linear law. It is also clear that Esperanto is a language with SVO word order preference, the mean dependency distance of Esperanto is 3.85, and the distribution of the dependency distance tends to a minimum, making it typologically a head-middle language. A 43.6% dependency relation appears in adjacent words. The complex syntactic networks of Esperanto display the characteristics of small-world, scale-free networks. All of these quantitative characteristics of Esperanto demonstrate that it is structurally a normal human language.


1994 ◽  
Vol 1 (3) ◽  
pp. 212-223 ◽  
Author(s):  
Koichi Ejiri ◽  
Niklaus Staeheli ◽  
Shiori Ooaku

Sign in / Sign up

Export Citation Format

Share Document