Binary ANSI Art in the Terminal

I built this and it looks proper in the terminal... but when I ssh to localhost its just the ansi characters... its garbled and there is no color. Same in iTerm and Terminal. I can cat the file and its fine. Somehow the ssh login breaks it?

libcaca: <a href="http:&#x2F;&#x2F;caca.zoy.org&#x2F;wiki&#x2F;libcaca" rel="nofollow">http:&#x2F;&#x2F;caca.zoy.org&#x2F;wiki&#x2F;libcaca</a>

I love ANSI art and I think this is a great way to introduce people to it!