General and Surprising

An unfortunate corollary is that slightly different general ideas are very difficult to get people to listen to because they feel like they already know it and it is not much new or the difference is negligible. This might be true of any particular case someone might think of but when the entire landscape is taken into account the multiplier pg mentions applies and can change things dramatically. But people don&#x27;t seem to be well wired to think of the general.<p>And for any general multiplier to work, people generally need to be on board.<p>For example: there is a difference between economic inequality and desperate economic inequality -- the first is a mere difference in ownership and control. The second is &#x2F;also&#x2F; a difference in ownership and control but that leaves one party or group bargaining against their physical well-being. There is much said about mere economic inequality but, by leaving out the desperation factor, the public conversation is tied up in knots (e.g. &quot;Well why don&#x27;t we give everyone a million dollars?!&quot; nonsense) and the real problem, desperation and the conditions that lead to that state, persist despite having the tools to potentially tackle it. But any single example -- a single mother that has to take two minimum wage jobs to feed her kids -- can be reduced away. She could just do x, y, or z and her particular problem would be solved and she&#x27;d still be economically disadvantaged but not desperate to the point of worrying about her kids starving. And the conversation ends. We are back to the inconvenience of a weak bargaining position which is also described by the phrase &#x27;economic inequality&#x27; and it all feels like the same, well trod and religiously guarded ground.

&gt; It&#x27;s not true that there&#x27;s nothing new under the sun. There are some domains where there&#x27;s almost nothing new. But there&#x27;s a big difference between nothing and almost nothing, when it&#x27;s multiplied by the area under the sun.<p>I found that statement the most insightful of the essay. Helped me resolve some cognitive dissonance.<p>Whether there really are totally new things and not just existing things in new packaging depends on which level of abstraction you stop at with when analyzing. For example, at a very high level you could say that the Internet as a whole is really just a very advanced&#x2F;efficient printing press. (And automation like the printing press or assembly line, just efficiently organized production.)<p>That level of abstraction may be less than helpful, unless there are insights to be gleaned from where the internet is going by looking at printing press technology. Their impacts on empowering the masses are similar, but they each encouraged&#x2F;enabled network effects to consolidate much power (newspapers and goog&#x2F;fb) and thereby effectively countering much of the effect.

&quot;the more general the ideas you&#x27;re talking about, the less you should worry about repeating yourself&quot;<p>A rule-of-thumb I&#x27;ve generally followed when brainstorming is &quot;never write down the ideas&quot;. And if at all possible, try to forget them, good or bad. It inevitably forces me to re-think the ideas &quot;from scratch&quot;. I&#x27;ve never been worried that an idea would escape me or that I would forget it, as I think the exercise of re-thinking ideas from scratch has helped me build a relatively solid mental model of the world. I feel that, with practice, this method of brainstorming has started bringing me to the solid &#x2F; useful (general &#x2F; surprising) ideas faster.<p>It has likely, over the years, added weeks or possibly months worth of time that I&#x27;ve spent brainstorming the same (general) ideas over and over.

I&#x27;m glad Paul is still writing essays. It seems it has dropped off in frequency, this essay kind of hints at one such reason for that. I hope he continues to write even if people disagree with his ideas or if they are repetitive.

The essay itself is general and surprising, and the combination never loses its effectiveness.<p>Ideas keep evolving. There was nothing new about the idea of a search engine when Google came into existence in 1998.<p>In 1945, Dr. Vannevar Bush wrote an article titled As We May Think:<p><a href="https:&#x2F;&#x2F;www.theatlantic.com&#x2F;magazine&#x2F;archive&#x2F;1945&#x2F;07&#x2F;as-we-may-think&#x2F;303881&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.theatlantic.com&#x2F;magazine&#x2F;archive&#x2F;1945&#x2F;07&#x2F;as-we-m...</a><p>&gt; he urges that men of science should then turn to the massive task of making more accessible our bewildering store of knowledge.<p>In the 1960’s, Gerard Salton created and developed Salton’s Magic Automatic Retriever of Text (SMART). He also authored a book called A Theory of Indexing detailing his initial tests that search is largely based off of relevancy algorithms. Here’s a very interesting blog post titled Search Down Memory Lane by Tom Evslin who worked with Salton during this project:<p><a href="http:&#x2F;&#x2F;blog.tomevslin.com&#x2F;2006&#x2F;01&#x2F;search_down_mem.html" rel="nofollow">http:&#x2F;&#x2F;blog.tomevslin.com&#x2F;2006&#x2F;01&#x2F;search_down_mem.html</a><p>Fast forward to 1990, a man named Alan Emtage created the first search engine known as Archie which retrieved a database files by matching user queries using regexes. Following the growth of Archie’s popularity, two similar search engines Veronica and Jughead were created and they started indexing plain text files.<p>In 1993, the first bot called World Wide Web Wanderer was created and then it was upgraded to capture active URLs and store them in a database. Then came ALIWEB (Archie-Like Indexing of the Web), which crawled meta information of pages.<p>There’re others including WebCrawler (1994), Lycos(1994), AltaVista(1995), Excite(1995), Yahoo(1995), Dogpile(1996), Ask Jeeves(1996).<p>This is just my superficial understanding of how the idea of search engines was born and has evolved over time. They were very different from Google, but they’ve similarities to how data is processed and analyzed today.

Might a parallell be draw to Kant here?<p>General and bland would be as akin to &quot;analytic a priori&quot;. All bachelors are unmarried, that sorta thing.<p>Specific and surprising would be like &quot;synthetic posteriori&quot;. Such and such plant is blue and grows in the Himalayas.<p>The real gold would be in the &quot;synthetic a priori&quot; findings. Blending the generalness of logic with the newness of empiricism. Like figuring out geometrical laws from observing nature. New information AND far-reaching implications.<p>Am I on to something here?<p><a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Analytic%E2%80%93synthetic_distinction" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Analytic%E2%80%93synthetic_dis...</a>

<p><pre><code> [T]he more general the ideas you&#x27;re talking about, the less you should worry about repeating yourself. </code></pre> Organisms are used to train over and over to get better at things. Certain abilities are innate, like a calf knows to stand up as soon as out of the womb. But to stand firmly the calf has to keep trying.<p>When it comes to an idea, I believe that the broader I present the idea, the easier it is to start a conversation, because others are more likely to show interest, and eventually reach the point you find a focus point. However, I still repeat myself, because I wouldn&#x27;t invent a whole new dialogue next time I bring up the same broad idea with another person.<p>On the other hand, I feel developing a product (startup or not), the more general the idea is, the harder for me to explain my ideas concretely, much like films have leading roles. The ideas present by the film can be broad, but you don&#x27;t want camera rolling on 200 actors in 1 hour 30 minutes, would you? Your message can be interesting and novel, but I am going to be lost.<p>But yeah, on HN topics like &quot;I quite FB&quot;, &quot;security is hard&quot;, &quot;Google is evil&quot;, etc are common here, but I still comment on those topics when I feel like to. My responses are usually similar, but I might add new thoughts, or rephrase what I had written before. I called this introspection. I believe we are not comfortable repeating ourselves, because we want to be interesting and original, but we still have the urge to try to perfect our speech the next time someone asks.

When I read stuff like this and reflect on the fact that this is about the peak quality of the philosophical discussion that is going on in our little niche of the world, I get slightly depressed. I get that it&#x27;s hard to find people at the intersection of astounding writing and technical chops, but still, it makes me want to distance myself and just go back to reading the classics.

Only slightly related but there is a great book for your children: &quot;What to do you do with an idea&quot;<p><a href="https:&#x2F;&#x2F;www.amazon.com&#x2F;What-Do-You-Idea&#x2F;dp&#x2F;1938298071" rel="nofollow">https:&#x2F;&#x2F;www.amazon.com&#x2F;What-Do-You-Idea&#x2F;dp&#x2F;1938298071</a>

Paul Graham is running out of ideas these days. (Partly joking).<p>This goes on to my same discovery: Life itself is a constant process of incremental trial and error builds-up. Over a few billions years, you get something out of it (human being playing snapchat)<p>If you keep counting from 0 to an infinitely large number, at some numbers you&#x27;ll &quot;discover&quot;: Windows 7, Windows 7 Korean Language, Microsoft Office, OSX, and every other game that human designed and developed. Plus a bunch of other software from alternate realities that compile on our machines. Cool.<p>The process the human are doing is not much different (albeit one can argue that it is much more efficient than stupidly counting) &quot;Everything&quot; is already there. The new Audi A4? It was already &quot;possible&quot;.<p>If there is one thing that will change the future, it is our discovery of the link between &quot;abstract&quot; and &quot;real&quot; concepts. They share some important points that I&#x27;d not be surprised that &quot;real&quot; and &quot;abstract&quot; are the same thing&#x2F;continuum. Like space and time.

A very academic &#x2F; theoretical post. Not entirely sure the point.<p>It might have been easier to understand if bound to real world examples.

&gt;that territory tends to be picked clean, precisely because those insights are so valuable<p>People often think that a field has been &#x27;picked clean&#x27; and conclude that either no further progress is possible or that further discoveries are going to be rare and incremental.<p>e.g. &quot;the future truths of physical science are to be looked for in the sixth place of decimals&quot; -- Michelson, 1894<p>But there are always deep problems and this implies that unlimited progress is possible. The really big discoveries can happen at any time and are of an unpredictable nature, not least because they affect multiple fields. They leave behind plenty of smaller problems for everyone to pick up.

The essay describes a 2x2 grid with axes of generality and surprise. Things that are suprising and general are valuable. Things that are general but unsurprising are platitudes, and things that are specific but surprising are gossip. The essay itself is a platitude.

A couple of years ago I wrote a book about a new philosophy of science based on the following methodology:<p>- Obtain a large dataset of observations related to a phenomenon of interest (eg English text, bioimaging data, economic reports, car-mounted camera recordings, etc)<p>- Develop a theory of the phenomenon, or revise a previous theory<p>- Build the theory into a lossless data compression program<p>- Score the theory by invoking the compressor on the dataset, taking into account the size of the compressor itself; lower codelength is better.<p>- Adopt or discard the theory as appropriate and return to step #2.<p>I believe that this is a valid <i>variant</i> of the traditional scientific method, in which data compression takes the place of experimental prediction. In Graham&#x27;s terms, this idea is a small delta of novelty, but since it refers to a predecessor idea of tremendous significance and generality, it could be very important. The book is available here:<p><a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1104.5466" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1104.5466</a><p>I&#x27;ve spent a few years pursuing this methodology in the domain of English text. The result is a sentence parser, which works quite well, but was built completely <i>without</i> the use of hand-labelled training data (eg the Penn Treebank). You can check out the parser here:<p><a href="http:&#x2F;&#x2F;ozoraresearch.com&#x2F;crm&#x2F;public&#x2F;parseview&#x2F;UserParseView.jsp" rel="nofollow">http:&#x2F;&#x2F;ozoraresearch.com&#x2F;crm&#x2F;public&#x2F;parseview&#x2F;UserParseView....</a><p>I&#x27;m always happy to talk to people about these ideas, ping me at daniel dot burfoot at gmail if you&#x27;re interested.

I think pg is falling prey to the whole post-hoc founder story phenomenon here. He sees so many startups, he hears so many stories about them, and he works hard to distill it into some kind of actionable and scalable investment strategy.<p>The problem is that the facts which led to the success of the business are not the same thing as the best insights a founder can come up with about why it succeeded. The latter is subject to an incredible amount of confirmation bias and the simple desire to tell a good story—in fact you <i>have</i> to do that to get investment.<p>In reality, novelty of ideas is meaningless, because the massively overwhelming majority of ideas in the world never get executed on. Even within contexts where people and companies have the means, most ideas—even good ones—don&#x27;t get executed on because they conflict with other ideas and don&#x27;t win the battle for resources.<p>What <i>matters</i> for an idea is execution and context. If you bring the right resources: skills, connections, go-to-market strategy, smaller supporting ideas, and it gels with the current zeitgeist of whatever market you are in, then it can get some traction. From there you learn and gain more insights which can be leveraged to scale up. You do this over and over again, and if you are lucky you can make an 8, 9, 10-figure company.<p>By the time all this happens you&#x27;ll have some tremendous insights, but they will be specific to the context in which they were gleaned. The &quot;generality&quot; we observe of these insights is just a result of our pattern-matching and story-telling brains. The &quot;surprisingness&quot; comes after the fact to varying degrees based on existing preconceptions and perhaps how the world has changed over time. In other words, I don&#x27;t think there is such a thing as a hugely valuable insight per se, rather this: we have insights, we believe in them to varying degrees, and based on how well it maps to our actions and their results, we later declare the &quot;value&quot; of the insight.

On the &quot;surprising&quot; part of this - Murray Davis had some related research and writing about being interesting: <a href="https:&#x2F;&#x2F;proseminarcrossnationalstudies.files.wordpress.com&#x2F;2009&#x2F;11&#x2F;thatsinteresting_1971.pdf" rel="nofollow">https:&#x2F;&#x2F;proseminarcrossnationalstudies.files.wordpress.com&#x2F;2...</a><p>&quot;An audience finds a proposition ‘interesting’ not because it tells them some truth they thought they already knew, but instead because it tells them some truth they thought they already knew was wrong.&quot;

Is this a stealth &quot;I&#x27;m done with the small delta of novelness Arc already has and am stopping on it. If you can&#x27;t appreciate that then you need to look harder&quot; article? :-)

At the beginning:<p><i>The most valuable insights...</i><p>Toward the end:<p><i>So it&#x27;s doubly important not to let yourself be discouraged by people who say there&#x27;s not much new about something you&#x27;ve discovered. &quot;Not much new&quot; is a real achievement when you&#x27;re talking about the most general ideas. Maybe if you keep going, you&#x27;ll discover more.</i><p>I really feel like Paul is conflating valuable and novel here. When you create something that solves someone&#x27;s problem, who cares whether the ideas behind it are novel or not?

PG makes it all look so neat and tight, however he fails to mention the number of serendipitous instances that drive insights and new ideas.

Well, that was certainly general.

There&#x27;s also a class of valuable insights that are general and not surprising, but rather obvious. For these, the challenge is that no one previously had the insight to observe and <i>state</i> what is obvious. Maybe the surprise for these isn&#x27;t in the truth but rather that no one had stated them before.

&quot;surprising&quot; in patent law <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Inventive_step_and_non-obviousness" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Inventive_step_and_non-obvious...</a>

F = ma is interesting because there are universal forces like electricity, gravitation and electromagnetism that we use for our daily life. Moreover there is a marriage between maths and physics since acceleration is the second derivative of space with respect of time. If you are a new Newton you should create the new concepts and develop the new maths to create a surprising step forward. Looking for a word that goes from gossip to F=ma is like using a telescope to watch a worm. Concepts are about frameworks and using a right scale to focus concepts.

For me, general and surprising things are coming in the form of &quot;X doesn&#x27;t seem to work&quot;.<p>The reasoning goes like this: X is described like a good thing and a lot of resources poured in it. But there are documented failures of X - are they spurious or causes by X not actually doing what is advertised?<p>More often than not the working answer &quot;it probably doesn&#x27;t work&quot; which is often &quot;a thing you can not say&quot; due to general consensus, good feelings and above mentioned amount of resources poured in X.

This post is lacking in the latter.

&gt; It&#x27;s not true that there&#x27;s nothing new under the sun. There are some domains where there&#x27;s almost nothing new. But there&#x27;s a big difference between nothing and almost nothing, when it&#x27;s multiplied by the area under the sun.<p>PG, I see what you did there. The rest of the essay is just a setup for this single paragraph. This is your own small delta of novelty. Well done.

I&#x27;d say this post was &quot;general, not surprising&quot;.

Too abstract to be useful.

My personal image about this topic is the fact that the general relativity theory is more advanced than the special relativty theory.<p>It reminds me that creating something more general is harder than creating something specialized for a specific problem. For programmers this seems to be obvious, but for others this is counter intuitive.

&gt; It&#x27;s not true that there&#x27;s nothing new under the sun.<p>Philosophically this statement is false.<p>Mental health wise, it&#x27;s a lot nicer to keep in mind than the truth of reality that you reside within.

This guy pretends to be an intellectual but is the biggest wannabe I know. He tried to block the latest Steve Jobs film because it showed how much of an asshole Jobs was

There are also the platitudes that everybody knows but that nobody does anything about.... Oftentimes there is virtue in simply being unusually conscientious.

Diff eq-like behavior is everywhere: species population, popularity, technology, market saturation, economics, spring rates, etc.<p>(Malthus was partially ;) mistaken.)

What are some examples of past insights someone has made by discovering something slightly surprising about the general world?

I think Nassim Nicholas Taleb&#x27;s work epitomizes the &quot;general yet surprising&quot; territory.

Clearly Paul Graham is defending the Bodega vending machine here ;)

tl;dr if you have an idea that isn&#x27;t very &quot;new&quot; then keep working on it until it is &quot;more new&quot;<p>There should be a twitter account that just summarizes rambling think pieces.

F=ma is surprising? We took two values and combined them to define a new one. You can define all sorts of new things this way, how is that surprising?

It&#x27;s very resonant,ths for sharing.

Paul Graham doing what he does best. Pseudo-philosophical musings with no basis or any kind of rigorous pass through. Why is this gaining visibility? If anyone else had posted this it&#x27;d never see the light of day on here.

This is not true as often you find surprising things that are not general which ARE NOT GOSSIP. To stay with the physics theme, the Schwarzschild solution of general relativity is a very special one and I don&#x27;t think anybody thinks that black holes are gossip.<p>And it is certainly not surprising that amateurs in general forget that F = ma is only valid if the mass does not change. The more general expression is F = dp&#x2F;dt, where p is the momentum. But this, of course, is also only valid in inertial systems. It&#x27;s not really important to the article but it does kind of annoy me that he uses the most special case of an expression in an argument about it being general.<p>He could have actually made the point about generality by comparing this expression to the most general one for the force (in a frame of reference that accelerates). That would have also shown why generality can quickly become infeasible in practice. If he knew how many approximations people make in the real world, not because they want to, but because they HAVE TO, his worldview might be a different one.<p>I feel like he&#x27;s trying to make a point about a very specific scenario but doesn&#x27;t mention it explicitly. Instead he tries to be general and therefore fails to understand that his view doesn&#x27;t actually apply in general.

This genre of writing is also neatly categorised into one of three categories: Obvious, unactionable, or wrong.<p>PG sometimes meanders into actual meaning. But the likes of Tim Ferris or Seth Godin absolutely excel at this drivel.

It&#x27;s funny, a lot of my imposter syndrome comes from an anxiety about not having massive &quot;deltas of novelty&quot;, as PG would call it.<p>I guess all I can do now is hope that whoever is listening can see the value in incremental change :)

&gt;<i>Thanks to Sam Altman, Patrick Collison, and Jessica Livingston for reading drafts of this.</i><p>This (a very basic 400 word essay&#x2F;post) needed &quot;drafts&quot; and 3 people going through them?<p>I like some of PG&#x27;s essays, but this one just has a couple of trivial insights.