I named my security "mBTC" because bitcoin denominations have so many decimal places. I just multiply all my trade quantities by 1000 and divide the price by the same. This works well because Quicken is setup to handle lots of decimal places in a security's price, but not so many in its shares.
For example, buying 5 bitcoin for $250.45 would go in as:
5,000 mBTC @ $0.25045
The other difference in what I do is that I mark network transfer fees using ShrsOut transactions in the investment account. This workaround isn't ideal because they show up as cost basis related transactions when you generate a capital gains report, but it's the best I could come up with. I'm curious to hear if anyone else has a better solution here.
An update to my suggestion here... I have found a very workable solution for the last point on withdrawal fees, network transaction fees, etc. Instead of using ShrsOut, I now enter these as ordinary SellX transactions. I enter the number of bitcoins taken in fees as the amount for "shares sold" and use an approximate price based on a historical quote service or another transaction the same day. Then, the key point is that I enter the full value of the transaction into the commission/fee field so that the transaction amount ends up as $0.00. This way Quicken correctly recognizes that I've lost some bitcoins, asks me to identify the lot(s) for cost basis purposes, and then shows that the whole profit went to fees. It works out perfectly for accounting at tax time.