For me it's the size of the blockchain.  Even with the exponential improvement of storage, computing power, and bandwidth.  Something that big simply won't scale and is not sustainable for the 100's of years that is expected of Bitcoin.