The way Bitcoin mining hardware works is to put a bunch of SHA hash units on a single chip so the miner can do parallel work. SHA hashes are designed to be computed quickly and with minimal resources, so they're naturally quite easy to run in parallel. So in other words the design of a Bitcoin ASIC miner isn't necessarily as expensive as other kinds of chips that you could make. The main problem I see is that the hardware is low-volume and thus expensive to make. Hardware is very much a volume business and Bitcoin hardware is extremely specific. Nobody else is going to want a chip that brute forces SHA-2.

Even if hardware costs for ASIC machines dropped, the hardware is still very much special-purpose. With CPU mining everybody with a computer could have a chance of mining some bitcoins. When GPU miners were released this got specialized down to computer with a decent graphics card. This later specialized down into having to have a computer with a very specific AMD card, then a computer with multiples of those, then having multiple computers altogether. Now, the average Bitcoin mining hardware resembles no computer one would actually build or want to use. So already mining has specialized to the point where it's a profession rather than a task everybody on the network carries out.

The problem isn't just the cost of obtaining ASIC hardware, it's the fact that it has to be obtained at all. Mining is a very computationally expensive operation with no other benefits than either going to a mining pool or solo-mining. Unless you have one of these mythical ASICs, or a very specific purpose-built GPU mining rig, you aren't going to make anything resembling a substantial amount of money. But you will be spending lots of electricity to do practically nothing.
So in all practicality, the more specialized the mining gets, the less people can actually start mining, which is contrary to Bitcoin's design.
Even if the ASICs get really cheap, Bitcoin will adjust difficulty to compensate, in a sense, and you'll have to buy a lot of them again.

-kmeisthax