Thread by Michael Wooldridge
- Tweet
- Jun 27, 2022
- #Neuroscience #Cognitivescience #ArtificialIntelligence
Thread
Perceptions of neural nets over the decades... A thread. [1/30]
I believe it is accepted that some in the neural nets community feel badly about the way that the field was treated at points over the past four decades - particularly, I guess, after the 1980s connectionism/PDP boom deflated in the 1990s. [2/30]
After a recent office move, I reorganised my book collection. And it turns out I have a ridiculous number of AI textbooks. So I thought I would revisit them, and see what they had to say about neural nets - and whether the hurt was justified. [3/30]
First up: Douglas Hofstadter's Godel, Escher, Bach (1979). A Pulitzer prize winner, incredibly influential in the 1980s, exploring connections between AI, logic, music, and art. [4/30]
GEB is quite the intellectual trip - a truly unique and head expanding work. I think many students will have been tempted into AI/logic/maths after reading this, so hats off to Doug Hofstadter. I haven't seen anything like it since. [5/30]
And GEB *does* talk about neural structures (pp339--) -- and doesn't shy away from the big questions (how does brain lead to mind?) Actually it tackles LOTS of big problems... [6/30]
However, no mention of perceptrons, Rosenblatt, or (perhaps fortunately) Minsky & Papert's ``Perceptrons'' book (though M+P's other work is cited). [7/30]
Next: The 3-volume ``Handbook of Artificial Intelligence''. Vols 1 (1981) and 2 (1982) edited by Barr & Feigenbaum, and Vol 3 (1982) edited by Cohen & Feigenbaum. I have a very soft spot for these books. [8/30]
They books are a huge nostalgia trip for me: I studied them intently as an undergrad (1985-89), and they give a fascinating insight into the world of AI in the 1970s, with lots to say that remains relevant... and lots that is now of strictly historical interest. [9/30]
Vol 3 covers vision and learning, and we find perceptrons mentioned in the context of pattern recognition... and gradient descent also makes an appearance (p376) - slightly to my surprise - described in one sentence as ``hill climbing''. [10/30]
However, the overwhelming bulk of the vision/learning section is on symbolic approaches, and buttonholing neural nets simply as pattern recognition sits oddly given what we now know. (This is 40 years ago, though.) [11/30]
Next up, Elaine Rich's 1983 ``Artificial Intelligence''. I think this was the first really widely adopted AI textbook. It was the main text for my undergrad AI courses. Its well written, makes lots of connections, and was *very* influential. [12/30]
Rich's book influenced AI courses right down to the present day. The structure is now canon: problem solving/search; game playing; knowledge representation; NLP; perception; learning; applications. You can find AI courses with this structure everywhere, even today . [13/30]
Rich's book was a classic, and remains a fascinating
read. Perceptrons only get one paragraph though, and are dismissed as having never met any degree of success (p.363). This was a couple of years before PDP/connectionism got everyone's attention. [14/30]
read. Perceptrons only get one paragraph though, and are dismissed as having never met any degree of success (p.363). This was a couple of years before PDP/connectionism got everyone's attention. [14/30]
Next is a special issue of AI journal published as a book under the title ``Foundations of Artificial Intelligence'' (MIT Press, ed D Kirsh, 1992, based on a 1987 workshop). [15/30]
Symbolic approaches dominate (Lenat, Feigenbaum, Newell...), but interesting to see very strong pushback against logicist approaches (``Rigor mortis'' is the name of an article critiquing logicist AI) and knowledge-based AI (see next tweet) [16/30]
Neural nets are scarcely mentioned. Rod Brooks seems to be the only person to do so, in his (wonderful) article ``Intelligence without representation''. This article was a clarion call for AI researchers of my generation - and doesn't pull its punches. [17/30]
Rod discusses neural nets, but mainly in terms of positioning his own work (reactive/behavioural AI) - Rod was a long way from logic/knowledge but his AI was at a higher level than neural AI. [18/30]
Matt Ginsberg's ``Essential of Artificial Intelligence'' was published in 1993. Matt's book clearly shows the field had matured since Rich's textbook - the approach is much more algorithmic/mathematical. [19/30]
It includes a very clear explanation of simple neural nets (pp310--), but no algorithms - no gradient descent/backprop. The approach is dismissed as because of the lack of a declarative basis, but perhaps more importantly because of concerns about scalability. [20/30]
``... the neural approach ... will be in serious difficulty if we need 10^12 thresholding elements!'' (p313) [21/30]
Well, the cumulative effect of Moore's law has caught out many of us. A salutary lesson. [22/30]
Finally, Russell & Norvig's classic ``AI: A modern approach''. I recall the ripple of excitement that passed through the AI community when this book was released in 1995. It was (is) a *breathtaking* work of scholarship. [23/30]
No other AI book has come close to this one in terms of its ability to tell a coherent story about the endless dizzying strands of thought that make up our crazy field. I've written multiple textbooks: I know just how hard it is. This was a major work - unique. [24/30]
And the 1st edition of AIMA gave solid treatment to neural nets (Ch 19): an extremely clear description of perceptrons, multilayer feed-forward nets, back prop, and so on. With proper mathematical definitions+actual algorithms. [25/30]
However, the book points out many problems with neural nets, and suggests that they ``don't form a suitable basis for AI in their present form''. [26/30]
The latest (4th) edition of AIMA was recently published - I should flag up I authored the chapter on multi-agent systems, and was very proud to be asked to do so. [27/30]
This edn includes a *fantastic* contribution on neural nets by Ian ``GAN'' Goodfellow. Its superbly written, and does full justice to developments up to 2020. [28/30]
Conclusions? Yes, neural nets clearly were regarded as of marginal interest and uncertain value by many in AI for a long period. I don't think that was necessarily unreasonable - unless those dissing the approach were proposing something hopelessly impractical instead... [29/30]
Personally? I saw neural nets as a *complementary* approach, which clearly demonstrated value in some area; I didn't anticipate what we've seen over past decade. One thing's for sure: AI will continue to surprise us, and we'll come to laugh at many opinions we now hold. [30/30]
Mentions
See All
Venkatesh Rao @vgr
·
Jun 28, 2022
Good thread
