Google Ngram Data for Every Word in the CMU Pronunciation Dictionary

Google provides fantastic data on the frequency of words and ngrams in all the books it has scanned, in many cases going back centuries. The only drawback is that these files are enormous. The unigrams alone are 4GB when compressed, not including numbers and punctuation.

There are also many, many words that aren't actually words or of much interest. So I thought I'd parse the unigrams down to only those that show up in the CMU Pronunciation Dictionary. It's not a complete list of English words by any stretch, but it's remarkably complete.

The result is cmu_google.csv, a 4MB file of term frequency and document frequency for 86,972 words, plus the CMU pronunciation.

Google provides year-by-year data on each word. To reduce the file size, each word in this CSV is listed once with the total number of occurrences (the "term frequency (tf)") and total number of books in which it is found (the "document frequency (df)"). I also included the total number of years the word occurs in the original data, the first year it occurs, and the most recent year it occurs, through 2008 (the last year in the original data), like so:

Hopefully this can save you a few hours and a few gigabytes of hard drive space. Please let me know if you find it of any use!