Show HN: Five Thousand Novels, Ranked by Vividness

This will come across as very mean-spirited but I find the idea of measuring a novel’s “vividness” based on the kind of vocabulary it uses (or the voice of the verb, etc.) to be completely bogus.

Here&#x27;s an article I wrote, describing the idea behind the project, defining the idea of &quot;vividness&quot;, and explaining how the linguistic analysis works:<p><a href="https:&#x2F;&#x2F;blog.shaxpir.com&#x2F;writing-vivid-prose-33283e861358" rel="nofollow">https:&#x2F;&#x2F;blog.shaxpir.com&#x2F;writing-vivid-prose-33283e861358</a><p>You can click around anywhere on the histogram chart, to see the different percentile buckets. And you can click on any of the books, to see detailed linguistics, including a snippet of the most vivid page in the book.

How&#x27;d you assemble the corpus? Only some of these books are public domain, did you have to buy&#x2F;license the rest?

Apologies if this is mentioned and I missed it, but does this account for changes in word meaning or context over time? Earlier literature, such as Austen, could be considered “not-vivid” unless you’re clued in for particular hints&#x2F;phrases. I’m thinking of, perhaps, the use of “Et cetera” for pudenda.

Here is the &quot;most vivid&quot; page of the &quot;most vivid&quot; book:<p>&quot;giants, and they were impaled by spear, lance, and crystal shard. A series of explosive reports echoed across the battlefield as the giants stumbled upon the Mistcloak’s tripwires, sending lethal blossoms of sharpened steel twisting through the air. Fell and his minions moved through the giants like an avalanche. The Under-King shifted his form to a flowing slab of stone and crashed down upon giant flesh, pulverizing it to blue powder and red ash. Even the animals, though weak and weary, tore into the giants with the primal fury of the wild. Claw and fang stood with horn and hoof, wounding with equal enmity. Beak and talon darted and gouged. The entire island of Mistgard stood united against the foul armies of frost and fire. Devastation was rampant on the mountain, but it was nothing compared to the wrath of the Storm Speaker. Even the stoic Under-King was surprised at the power of the Oldest of Cubs. At the back of the Pandyr’s armies, high atop the tallest of Fell’s battlements, stood the lone figure of the Storm Speaker. He called forth and charmed the very storms from the clouds beneath him and sent electric green-and-blue arcs of lightning into the giants’ lines, blasting hundreds of their bodies off of the battlefield and into the mist below. The world above burned. The radiant morning light was blackened by acrid smoke, making the golden skull radiate a brown and bloody glow. The Aesirmyr lay strewn with broken bodies: blue and red&quot;

This is an interesting analysis, but I have a serious problem with the strong implication that high vividness = good.<p>At the rock bottom of the vividness scale, we find Jane Austen, Isaac Asimov, Agatha Christie, C.J. Cherryh, and Danielle Steel -- all extremely popular authors. And at the very top, we find George R.R. Martin, Roald Dahl, Poul Anderson, Edgar Rice Burroughs, Ray Bradbury, and Kim Stanley Robinson -- also popular, but generally not quite of the same stature as those on the first list.<p>Possibly the reading public is slightly biased toward low vividness. Meanwhile, I have at least two favorite authors on both lists.

I was naturally curious about outliers so went to the most vivid book which is &quot;Pygmy&quot;. I haven&#x27;t read this book, but style of it is listed as incorrect grammar &quot;English&quot; written in a detached scientific tone. I wonder how much this threw off the algorithm to cause it to have such a high score (over 100% and nearly 25% higher than second most vivid).

This is great.<p>I wish we could do our own arbitrary style analysis on the data set sort of the way one can do a factor analysis on a portfolio.<p>I would look at words that are common between Marukami and McCarthy compared to the rest of the corpus for instance.

Interesting project. Not surprised to see Chuck Palahniuk at the top of the list, but was a little shocked how much he dominated it.

Incredible concept! On mobile (ios 6s) the website needs some work.

This is very cool. I want to search, can we search?