You can use getrawmempool to get a list of all the transactions in the memory pool which should be everything the node knows about that is not in a block in the current chain.  getrawtransaction will then give you the unparsed transaction data.

If you want to process all new transactions immediately you will probably need something that connects as a peer into the network (a node you trust) to get broadcasted transactions and new blocks.  Clients like Armory do this now.  There might already be some code around to do this depending on the language you are using.

I think bitcoind needs an interface to request transactions and notifications of specified addresses or with specific outputs.  Do you actually need every transaction or just all transactions associated with some set of addresses?