Norvig vs. Chomsky and the Fight for the Future of AI

It's a little bit frustrating to read a rehash of an argument that was cutting edge <i>maybe</i> back in the late 90s, especially one that is so poorly written, and framed as a battle between two intellectuals.<p>Chomsky's past his heyday. He has been seminal in his field, but he's no longer doing research which pushes at the boundaries of our understanding of language, how to model it, or what the fundamental nature of language understanding systems is. (as one might infer, I come from a non-chomskyian school of linguistics).<p>Given that we have actual data and research about large scale systems that do interesting things (including the massive artificial neural network that google built last month, see: <a href="http://www.wired.com/wiredscience/2012/06/google-x-neural-network/" rel="nofollow">http://www.wired.com/wiredscience/2012/06/google-x-neural-ne...</a> ) reporting as substance free and obfuscating as this is, is a real frustration, when we could be talking about more interesting things, such as what a solid operational definition of meaning is, or how exactly heuristic/rule based systems actually differ from statistical mechanism, and whether or not all heuristic systems can (or should) be modeled with statistical systems.<p>The framing of this article is particularly galling because there are so many non-chomskian linguists out in the world who operate fruitfully in the statistical domain. Propping Chomsky up as somehow representative of all linguists is pretty specious and a bit irritating.

I spent about ten years working on Markov based chat programs. I gave up on themwhen I realized that no matter how sophisticated your statistical model it will never be more than a statistical analysis of text, unless it includes some rich rule based model of mental processes and mental objects. It may be that such a model of mental processes must itself be fuzzy and probabilistic, but it must exist. Therefore I come down firmly on the side of Chomsky in this debate: we should pursue theories of intelligence, and stastical models without any theory do not advance our scientific understanding of AI, however practical their application may be at the present time. This is not to say statistical methods do not work, of course they work, what I am saying is it is not a path that leads to true understanding of intelligence any more than spectral analysis of the EMF emissions of a running computer would lead to a theory of computation.

This is one of those rare moments in intellectual life where being in the room and now seeing the debate develop, it becomes clear that the resulting hype isn't (wasn't) loud enough.<p>This distinction marks the real turning point in AI from abstract, grand claims with highly restrictive evidence toward engineering that simply works. Who cares about the ontology when we can recreate? It's like saying airplanes don't properly explain flight because they don't replicate how birds do it. Who cares? We can fly (and translate and soon reason) artificially.<p>It's clear that Chomsky and Universal Syntax has held back the entire field of AI (and at MIT). There isn't one algorithm in the human mind to decode all of our mental capabilities. That's mistaking subjectivity for objective lessons. Trying to recreate that Phantom has led to rule tables in AI, constraints on how the mind must operate. Instead, by allowing those fuzzy boundaries to accumulate with evidence, statistical approaches win in the long-term of our lives and in this debate.<p>Kuhn knew what happens to dinosaurs.

I've been in machine learning/AI for ten years now - from undergraduate research, to graduate school, to industry - and I find debate like this fascinating. My take on it is that our understanding of what we will be able to do in the future is very unclear, and what we will want to do is very open-ended. So the debate is worth having, but it won't really resolve anything.<p>Statistical models may (in my opinion probably will) end up being an "AI" dead-end, eventually falling into other fields such as algorithms, like game trees and logic-based agents did. That's not to say the current statistical approach is a bad idea; on the contrary, I think these techniques are useful and simple enough that they will become fairly ubiquitous in CS.<p>On the Chomsky side of the argument, AI researchers have consistently been frustrated in the past 50 years, to the point that studying AI today makes you sound like a joke. But their goal is a noble one. Anyone can understand how great it would be to have a human-level intelligence on a chip - this would fundamentally change the World. The fact that we haven't dented this problem doesn't mean the problem isn't worth solving, it just means our understanding of what it takes to build this kind of AI is in its infancy.<p>I almost feel like Norvig and Chomsky are arguing in parallel. They are both right, but their arguments are valid on different time scales. Today, the Norvig approach will easily win out; Chomsky has nothing and is largely irrelevant. But Chomsky is, IMO, correctly predicting what will need to happen to move beyond an eventual roadblock in a much grander AI.

They have two different definitions of "artificial intelligence," which is where the schism seems to be arising from.<p>Chomsky takes the academic approach - artificial intelligence is the simulation of humanlike (or even possibly mammalian) intelligence.<p>Norvig is taking the engineering approach - artificial intelligence needs only to pass the Turing test.<p>They're both right, both approaches have value, and they both are bound by our limited technology at the moment.<p>In the end, though, Norvig will lose out. Sure, he'll make the finish line first - an AI capable of 'passing' the Turing test, but in order to have real intelligence you need an analytical engine (or brain, if you will) that can prioritize data without fiddling with bits. In the Norvig solution, someone will always have to be fiddling with the bits.<p>Chomsky's approach, on the other hand, will result in a 'true' artificial intelligence, the way neurologists understand it. It's just going to take a lot longer to get there.

First thing, please read the actual article by Norvig, it is excellent,<p><a href="http://norvig.com/chomsky.html" rel="nofollow">http://norvig.com/chomsky.html</a><p>Second: I found it astounding that the article never mentions Skinner. Surely this article is trying to do to Chomsky what Chomsky did to Skinner in 1959 ("A Review of B. F. Skinner's Verbal Behavior", <a href="http://www.chomsky.info/articles/1967----.htm" rel="nofollow">http://www.chomsky.info/articles/1967----.htm</a> ).<p>Chomsky basically marked the beginning of modern era of cognitive psychology with that essay, displaing the previous paradigm of behaviorism. Norvig's article has similar form in some ways to that article, and similar goals (to argue for a new paradigm over an older one). As I was reading it, I was sure Norvig had that context in mind. So I was surprised to read<p>&#62; So how could Chomsky say that observations of language cannot be the subject-matter of linguistics? It seems to come from his viewpoint as a Platonist and a Rationalist and perhaps a bit of a Mystic<p>Well, no, Chomksy explained very well why he opposed observations being the subject matter of linguistics in his 1959 essay. Skinner's behaviorism looked only at observations and experience, and did away entirely with internal mental states. That might seem bizarre to us today, and the reason is in large part the shift heralded by Chomsky's article from behavioral psychology to cognitive psychology. In the latter, the goal is to understand the internal processes that are involved in psychology (or specifically language).<p>Statistical language models are not behaviorism. But they do share a lot with it, they are based primarily on raw empirical observations as opposed to deep models, so it is natural for Chomsky to oppose them on similar grounds (and not due to Platonism or Rationalism, although I suppose you can speculate that those motivated his 1959 essay too).<p>Side note, we can speculate that if Skinner had today's computers and statistical modelling methods, the shift from behaviorism to cognitivism might never have happened, seeing as the statistical approach is so successful.

I know a card counter. I showed him how to condition probabilities to determine how to best play. He went for the full Monte Carlo method and he lets his simulation run for a week before he starts using it "just to make sure". It's frustrating because he doesn't get that his results are statistically significant after about 30 seconds of runtime. He still makes money doing it. The results are tangible, but he's still just mucking about.<p>'Quantum mechanics is certainly imposing. But an inner voice tells me that it is not yet the real thing. The theory says a lot, but does not really bring us any closer to the secret of the "old one." I, at any rate, am convinced that He does not throw dice.' --Einstein<p>Statistical methods can work but they are unsatisfying to the scientifically curious. You're not really a scientist if you create something that works and you don't really know why. (Not to say that the method doesn't have value. Sometimes you have to play with your Lego before you grow up.)

I picture Chomsky as Kepler, trying to build orbits out of Platonic solids.<p>Until Kepler had access to Brahe's data, he was not going to be able to come up with his theories of planetary motions.<p>Worse than that, the laws of planetary motion present a simplistic view of the universe: what happens when a bunch of small objects orbit a very massive object. I think they wouldn't help you out at all, in trying to understand planets moving in a binary star system.<p>There is no analytic solution to the N-body problem. We can only simulate the motions of a group of massive bodies by iteratively applying the laws of gravitation that we have deduced. Knowing the mathematical properties of how objects behave in a gravitational field, and actually understanding HOW GRAVITY WORKS are two enormously different things. Newton was frustrated with the theory of Gravity, because it was, as Norvig's models, just a model - with no explanation of why. But the model allows you to make falsifiable predictions, and understand how the universe will behave. Looking for the Higgs Boson is awesome - but there is potentially no equivalent in the linguistic world.<p>Chomsky asks us to ignore F = G * m1 * m2 / r^2, because there's no WHY attached to it.<p>PS - this understanding of the history of science is brought to you by Carl Sagan's Cosmos TV series. I have no deeper insight than that.

Isn't this basically an argument over John Searle's Chinese Room thought experiment?<p><i>It supposes that there is a program that gives a computer the ability to carry on an intelligent conversation in written Chinese. If the program is given to someone who speaks only English to execute the instructions of the program by hand, then in theory, the English speaker would also be able to carry on a conversation in written Chinese. However, the English speaker would not be able to understand the conversation. Similarly, Searle concludes, a computer executing the program would not understand the conversation either.</i><p><a href="http://en.wikipedia.org/wiki/Chinese_room" rel="nofollow">http://en.wikipedia.org/wiki/Chinese_room</a>

Intellectually, there seems to be something as wrong with avoiding anthropomorphism when discussing human endeavors (such as language) as there is with anthropomorphic explanations of erosion or chemical reactions. Skinnarian approaches to language may leave people unsatisfied because there is no story, just clinical observation.<p>Norvig's approach (as characterized in the article) takes the the "Artificial" in "Artificial Intelligence" to include the mechanism by which an intelligence makes decisions. Chompsky's aesthetic of linguistics applied to AI would treat "Artificial" as a description of the platform in which an intelligence is embodied (i.e. non-biological) while requiring the platform to operate linguistically on the same principles as a "natural intelligence."<p>Norvig's approach (as characterized in the article) is essentially a better Eliza (or Ford's faster horse).<p>If one takes the Turing Test as scientifically meaningful rather than an engineering standard, then one falls in one camp or the other and the Norvig Chompsky debate is over a pseudo-problem. "Artificial Intelligence" is in that sense metaphysical jargon.

OK, let me start with two facts, one objective, one personal: (i) Noam Chomsky is a genius with many contributions to linguistics and computer science (ii) I think his overall influence had been damaging to linguistics.<p>Here's a summary of Chomsky's career in layman's terms: As everyone knows, Chomsky first came to prominence with his critique of Skinner (who, as everyone also knows, was a total psycho). He pretty much created linguistics as we know it (at least in the US, there were some numbskulls in Europe who still doubted the new order), starting from the main thesis of linguistic universals, which can be summarized as the fact that all humans possess <i>the same</i> language faculty, i.e. the wide range of linguistics differences between, say, English and Mandarin are just on the surface. This was a welcome relief against the Sapir-Whorf mumbo-jumbo which held that Eskimos had hundreds of words of snow and language constrained how we think. Chomsky has also been very active in politics (he's actually much better known to the general world by his political books), pointing out the evils especially of the American brand of capitalism (is there any other kind?) and its corrosive influence on the world, e.g. Iraq, Afghanistan, etc. He also points out errors in certain approaches in Economics, e.g. see <a href="http://en.wikiquote.org/wiki/Noam_Chomsky#Capitalism" rel="nofollow">http://en.wikiquote.org/wiki/Noam_Chomsky#Capitalism</a>, without holding a degree in the field, but everybody does that.<p>Chomsky's greatly damaging influence to linguistics is due to the fact that his speculative and simplistic (at least originally) views on how the brain processes and learns language has stifled research in promising fields by decades. The main problem I have with him is that the cause of the shortcomings of his theory seems to be not lack of knowledge (very little was known about cognition in the 60s), which, of course handicaps all pioneers of science, but politics (I detest politically motivated scientific theories). AFAIK, his universalist views were motivated from his political beliefs.<p>Luckily, starting in the 90s, Chomsky's chokehold on linguistics has slipped somehow. Researchers, such as Leda Cosmides, have ventured into research on linguistic relativity (<a href="http://en.wikipedia.org/wiki/Linguistic_relativity" rel="nofollow">http://en.wikipedia.org/wiki/Linguistic_relativity</a>). Skinner's theories are making a comeback in academic circles (<a href="http://www.theatlantic.com/magazine/archive/2012/06/the-perfected-self/8970/" rel="nofollow">http://www.theatlantic.com/magazine/archive/2012/06/the-perf...</a>).<p>So, what does all this mean for the current debate? I think it's time to retire and the "old guard"! Let us acknowledge their breakthroughs, their contributions, but also their limitations and move on.

Well, in the big picture, Chomsky created an activity which keeps liguists very busy. His approach, however, has contributed very little to language engineering.

Historically, AI has been divided into two related but different approaches. "Strong" AI is interested in understanding and creating Minds; figuring out what intelligence is, how it works, how we do it, and how it could be done <i>in general</i>. "Weak" AI is interested in doing things that couldn't be done before; things that we do not have good algorithms for, or don't have any algorithms at all.<p>Those two are not <i>opposed</i>. Any advance on either side helps the other. In this argument, Norvig is representing an extreme version of weak AI since he seems to be arguing that it's possible that statistical methods are <i>all there is</i>. (I suspect that he isn't actually making that argument, though, but that strong AI's models are currently too simplistic to capture what statistical approaches can do.) Chomsky, on the other hand, seems to be caricaturing strong AI by saying that anything that doesn't directly shed light on the Grand Theory is worthless.

It's a question about engineering vs science. Before Kepler, people actually could predict the motion of the stars and planets through the sky; perhaps not as elegantly or accurately as after Kepler, but to a certain degree, so what?<p>The AI case is clearly a point where the theories from linguistics are insufficient for engineering purposes. Watson could not have been built today based off of Chomskian linguistics. Maybe the statistical models will advance the theory of linguistics, maybe not. Either way they will give us useful tools <i>now</i> which is better than elegant tools later.

AI research, including speech recognition and machine vision, are currently ENGINEERING disciplines trying to make artifacts that do interesting things. Success is an artifact that works.<p>Several basic science disciplines are trying to understand how brains work. There is mostly tremendous amounts of experimental facts, difficult to put together, and some theory and modelling to go with it.<p>Norvig would be confused if he thinks that engineering AI systems automatically counts as models useful for understanding the brain. If there is application to understanding brains it is a welcome accident. It happens that there are signals in basal ganglia that look like the temporal difference error signal from reinforcement learning. So maybe RL research can help understand some brain circuitry in that case.<p>But in general the engineers are trying to get stuff to work, and they are deluded if they think they are simultaneously making progress in understanding how brains work.<p>EDIT:<p>For example: why does speech recognition use hidden markov models and N-gram language models? Because they're the best model of how brains understand speech? No! Not at all. HMMs and N-gram models are above all computationally tractable. Easy to implement, not too slow to run.<p>We have algorithms (such as baum-welch and N-gram smoothing techniques) to get them work work well in engineering applications. Nothing more. Might they help us understand brains? Maybe, but not at all necessarily so.

Just for the record, i consider this a simple model. And it's from norvig. <a href="http://norvig.com/spell-correct.html" rel="nofollow">http://norvig.com/spell-correct.html</a>

It is an interesting debate, though I think it's being shone in the wrong light.<p>According to the article, it almost sounds like Chomsky believes a statistical approach to AI is a disservice to the field. The point he's missing is that research in statistical based AI is just that - statistics research.<p>Chomsky and Norvig deal in two different fields, which happen to have similar applications. Norvig does research in statistical and machine based learning. Success in this field comes from a new model that can make more accurate predictions, or a proof that it is impossible to make valid predictions about X with only Y as input. Applications of this field include technologies which rival AI systems as envisioned by Chomsky, but the essential point is that this field focuses on statistics research, not AI research.<p>Chomsky is wrong in dismissing this as a disservice. I do agree with his main point, that AI research and knowledge is not necessarily furthered by statistics research, but that is simply because they are different beasts entirely.<p>Maybe one day, when the biology has caught up with us and we have a solid understanding of the brain, will we be able to create a highly intelligent computer. Until then, statistics research is most likely to yield fruitful results.

One myth I want to debunk:<p>Copernicus's theory did NOT do away with epicycles. Search on Google for "copernicus epicycle" and the first article demonstrates my point. The one who did away with epicycles was Kepler. Copernicus believed orbits had to be perfectly circular; Kepler recognized that the data fit better into an elliptical model.<p>It's not 100% clear whether the author believed the "myth," but hopefully I can set some people straight in this forum.

The main problem with Chomsky's approach is that it is quite likely that human intelligence mechanics are just incomprehensible for a human intelligence, and not because of some crazy construction tricks but simply plain old brute size and complexity it imposes. Judging from much simpler (thus deeper investigated) biological systems like some bacteria metabolisms we can see that there is no grand design there, only trivial primitive core and numerous layers of less or more subtle modifiers of modifiers. IMO there is no reason why the same can't work for the brain and thus the "transition to sentience" is way more continuous than we would like to expect.

<i>&#62; If the solar system’s structure were open for debate today, AI algorithms could successfully predict the planets’ motion without ever discovering Kepler’s laws, and Google could just store all the recorded positions of the stars and planets in a giant database</i><p>I'm sorry, but this bit is half wrong and simply numerically illiterate. We can store all of the recorded positions of the planets and other bodies in the solar system, but we need models to predict their future positions. This is an important distinction, since we might use such models to save the human race one day.

There must be some analogies made to the much smaller field of chess computer programs.<p>From 1950s to about 1980 or so it was thought that the best computer chess program would approximate the way a human would think about the game. Botvinnik in particular was adamant that such an approach would be the right one.<p>However, most of the progress was made through brute force. Modern chess programs select moves in a way that is far removed from the way a good chessplayer selects moves, yet they can now produce games that seem very "uncomputer" like and "human".

It's true that Engineering at times leads Science. But, from a scientific view, what's the point of a model if you can't understand it? After all, we already know how to create intelligence without understanding it.<p>While it's conceivable that intelligence is too complex for a human to ever understand (e.g. if not amenable to hierarchical decomposition), that would be very sad news for science.

Norvig is only trivially right. Sure, with enough stats you can infer a lot of the structure of all the information we humans have created, and thus replicate the structure, as Google is doing with its suggest service. However, this does not explain how humans created the structure in the first place. Such a form of AI will forever being playing catch up to humans.

When Einstein heard about Quantum Mechanics and the idea that everything is a probability, he said: "God doesn't roll dice". He meant that even though Quantum Mechanics does give us many answers about the world of the tiny, it doesn't truly explain it. I believe that a similar analogy can be made to this case.

I know that everyone has been careful not to mention Chomsky's political beliefs, but I am suspicious that this is actually partly about Chomsky's political beliefs, which I think are more in line with reality or at least more egalitarian than Norvig's must be, since Norvig has been running one of the hegemony's greatest tools recently. I see a parallel between the general derisive dismissal of Chomsky's academic views as being simplistic with the type of dismissal commonly given to a Chomskyish geopolitical viewpoint. I see this disagreement as a surrogate for the very different geopolitical worldviews.<p>I doubt that Chomsky is really so hard line about his old approaches to AI as we are led to believe, although he is probably farther behind the times than Norving.<p>I actually think that even Norvig is just applying recent contemporary AI to AI problems, but still is part of an old or establishment guard himself as far as AI goes. I think that the real cutting edge AI research is called AGI (artificial general intelligence) research.<p>The generation/category of AI research or machine learning that Norvig is tied into is much newer and steps beyond the earlier traditional AI that Chomsky might have been involved with, but the AGI researchers are a step beyond Norvig's clique. And the AGI researchers are, by the way, very optimistic about the Singularity or at least the likelihood of human-like and probably super-human artificial general intelligence in the short or medium term.<p>I mean the Norvigish machine-learning stuff isn't completely disconnected from the AGI stuff and completely behind and I assume it will result in extremely capable AIs relatively soon, but the AGI approaches will probably prove to be more powerful and more humanlike since they are closer to human models.<p>Take a look at what Brain Corporation is doing, or Numenta, or the OpenCog project. That stuff is beyond Norvig and friends' approaches.

Where is eyudkowski when we need him?

I am a entrepreneur/researcher working to create artificial intelligence. My approach follows Turing's suggestion that one should create a child mind and proceed to educate it. I employ Construction Grammar in my English dialog system - not a statistical parser/generator. Operating on a smartphone, I use available statistical speech recognition engines to transform speech to text, but from that point onwards the server-side processing in Construction Grammar is symbolic, thus engineered from first principles. Likewise, for English generation, my discourse planner emits structured RDF that the bi-directional Construction Grammar generator transforms into a text utterance. That symbolic text is then input to an available statistical text-to-speech engine available on the smartphone, to speak to the user.<p>As an example of the power of symbolic approaches, my parser has a complete symbolic analysis of English auxillary verb constructions, producing unique, meaning-rich, RDF-compatible semantics for:<p>I am learning about computers.<p>We are learning about computers.<p>We will be learning about computers.<p>I could be learning about computers.<p>I have been learning about computers.<p>I better learn about computers.<p>I had better learn about computers.<p>I dare learn about computers.<p>I did learn about computers.<p>I do learn about computers.<p>He does learn about computers.<p>I had learned about computers.<p>He has learned about computers.<p>I have learned about computers.<p>He is learning about computers.<p>I need learning about computers.<p>I ought to learn about computers.<p>I ought to be learning about computers.<p>I used to learn about computers.<p>I was learning about computers.<p>We were learning about computers.<p>Because of the so-far limited success of my work, I am inclined to agree with Chomsky's AI argument despite using a modern grammar opposed to his linguistic principles.<p>An artificial intelligence will use both statistical techniques and symbolic, e.g. procedural techniques, I think. With the most useful intelligent behavior being symbolic. E.g. an AI designing, writing and testing software.