Comments on: Measuring vocabulary richness with python

By: list(g)

list(g) — Mon, 10 Oct 2011 22:16:00 +0000

Thanks for this, but the M2 = sum(…) line is giving me a TypeError (‘list’ object is not callable) in Python 2.6.

list(g) seems to be causing the error, but I’m not sure why. Each g is an itertools._grouper object, so I’ll take a look at itertools, but if anyone has a suggestion please let me know.

By: A geek with a hat » I want to analyze your blog

A geek with a hat » I want to analyze your blog — Sat, 01 Oct 2011 00:00:48 +0000

[...] Yule’s I – this is a measure of vocabulary richness, where I’m assuming a broader active vocabulary is a positive thing [...]

By: Links 30/9/2011: FOSS Catchup, Many More Linux Tablets | Techrights

Links 30/9/2011: FOSS Catchup, Many More Linux Tablets | Techrights — Fri, 30 Sep 2011 08:33:39 +0000

[...] Measuring vocabulary richness with python [...]

By: Swizec

Swizec — Thu, 29 Sep 2011 23:12:00 +0000

Oh right, I always forget about .get

But I’ll take collections when I’m in 2.7. It’s just prettier

By: Anonymous

Anonymous — Thu, 29 Sep 2011 16:04:00 +0000

Most of the code is pretty basic Haskell. The only advanced things I used are the Arrow class, and the uncurry function. uncurry takes a multi-argument function, something of the form (a -> b -> c -> d) and converts it into a function that takes a tuple of arguments, so something of the form ((a, b, c) -> d). The Arrow class is a bit harder to explain. First in my opinion, class was a poor choice of name in Haskell. Scala did a much better job at this and calls them traits which helps avoid confusing all the OO programmers and also I think is slightly more descriptive. A trait is something part way between an interface and an abstract class. Unlike in OO, traits may be added to types at any time simply by providing an implementation of them. The Arrow class (trait) simply represent a transformation of some kind, that is, you’ve got an A type, and you want to convert it to a B type, then that conversion can be represented as an Arrow. The (>>>) operator combines two arrows, and the (&&&) operator combines and parallelizes (think of it a bit like a beam splitter). In this code I’m taking advantage of the fact that there’s an instance of Arrow for (->) which is the Haskell function type. In other words, any function in Haskell, is also automatically an instance of Arrow.

If you have a function from an b to a c called foo:

foo :: b -> c

It’s also an Arrow of the type:

Arrow a => a b c

That bit at the beginning is just a restriction for the benefit of the compiler telling it that a is some instance of Arrow. Technically the type of foo could also be written as:

foo :: (->) b c

which is just the non-infix version of the previous signature. Notice that since (->) is an instance of arrow it also matches the arrow constraint provided previously.

By: Foo

Foo — Thu, 29 Sep 2011 16:00:00 +0000

err so much for my line breaks

By: Foo

Foo — Thu, 29 Sep 2011 15:59:00 +0000

instead of the try … except surrounding the count increment, you can also do: d[w] = d.get(w, 0) + 1(not as pretty as Alice Atlas’s collection counter of course

By: Swizec

Swizec — Thu, 29 Sep 2011 10:30:00 +0000

That looks really cool. I should give this Haskell thing a try … at least enough of a try so I could read that code

By: Swizec

Swizec — Thu, 29 Sep 2011 10:27:00 +0000

Ooh nice, that is really cool. Did not know about collections

Thanks!

By: Anonymous

Anonymous — Wed, 28 Sep 2011 23:49:00 +0000

I wrote something equivalent in Haskell, mostly as an exercise for my own benefit. I made rather heavy use of arrows and map, and I replaced d with two different lists although one of them is derived from another. It’s probably not the best Haskell code around, and there’s probably all kinds of better ways to do this, but it looks like it’s working more or less to me.
https://ideone.com/lAnzc

I copied in a chunk of your post as some test data and it produces a number that at first glance seems fairly reasonable.

By: Alice Atlas

Alice Atlas — Wed, 28 Sep 2011 21:30:00 +0000

You can replace that dictionary and for loop with a collections.Counter object if you’re on Python 2.7 or later.

stemmer = PorterStemmer() d = collections.Counter(stemmer.stem(w).lower() for w in words(entry))

(also adding import collections to the top, of course.)