Yes, it's a noxious poison because they are not holding it. If they did, I reckon they would be its biggest supporters in Wall Street.
Also, I'm starting to doubt in how Warren Buffet is presented by the media. I do not reckon he reached his status purely because he is a good investor. I speculate that it's also possible that he can be Wall Street's biggest market manipulator.
Warren Buffet made his wealth by basically getting in at the bottom of the stock market itself... then he just held long time. He has been wrong in the past about many other investments, but he's seen as a god by most people because they only see the ratings on Forbes.
Bitcoin is subversive in it's nature and some people can't see the value on that. His whole empire depends on the well being of the USD and he knows Bitcoin can be a threat to it at some point, so he isn't going to support something unless he got in at the bottom.
I'd say that his empire is built on the US economy rather than the US dollar as such. Basically, for him it doesn't matter what the USD exchange rate is as long as the American economy is kicking. I don't he holds any important amount of his wealth in dollars or other fiat currencies. If the US economy prospers, so will Buffett and his investments.