Absolutely it is -- your talking to a neuroscientist who did just that, sort of....
But it does help to take the classes. You can learn a lot of music theory in books, but until you <i>play</i> music in practice, you never get close. You can learn neurosurgery theory, but it doesn't change the first day when you actually perform it. They're quite different!
The classes are an accelerator. You can learn it all on your own, but the classes can, in many cases, offer a short-cut to learn things you might not find in the books. For example, you will find many sorting algorithms in books, but will you discover what leftist-trees are? Or needlesort? These are "tools" in the "toolbox".
Don't get me wrong, you can do it, but you'll be buying a lot of textbooks on your own. Again, notthing wrong with that -- we both know most college classes <i>you</i> do the work and the lectures are there to assit - you can skip the lectures if you desire. So, you can teach yourself -- it just takes longer. A history buff turned system admin once put it to me this way....
"I learned CompSci myself, but added the degree later to be accepted. You can it yourself and become expert in your field in about five to ten years - or take the classes and learn a lot in two. That being said, even an old history major can do it."