Estimating High-Dimensional Zipfians & Language Distributions

<<< All talks

Estimating High-Dimensional Zipfians & Language Distributions

When	Friday, April 11, 2025, 1:30 PM – 2:30 PM (MT)
Where	WEB L112

Abstract

We study estimation of large discrete distributions under the structural assumption that they follow a Zipfian distribution, in which the ranking of alphabet items and/or level of decay need to be estimated from data. Empirical evidence for near Zipfian distributions has been found in diverse applications such as word and n-gram probability distributions in natural language text and chord probabilities in musical pieces. We introduce the Sort and Snap estimator for when the level of decay is known but the ranking function needs to be estimated, and show it is minimax in several high dimensional regimes. When both the ranking and decay level are unknown, we introduce an adaptive variant of Sort and Snap and show via Monte Carlo simulation that it outperforms state of the art discrete distribution estimators in these same high dimensional regimes. Our results motivate assessment of whether linguistically motivated marginal distributions for generating natural language that are claimed to be Zipfian in quantitative linguistics communities are truly Zipfian. Through Monte Carlo experiments on one such well-regarded distribution, Sort and Snap procedures lag behind even the simplest non-parametric estimator (empirical proportions), which brings into focus that this distribution thought to be Zipfian actually departs meaningfully from the Zipfian pattern.

Speaker

Tags: natural language processing statistics

---

Part of the Data Science & AI Lecture Series. Something wrong on this page? Edit _data/talks/2025-04-11-peter-jacobs.toml.