That is not quite correct. The API invokes your callback URL when it receives the payment (i.e. 0 confirmations) and then keeps invoking your callback URL for every new confirmation (1, 2, 3, ...). It will stop doing this when you acknowledge the callback (I think you need to return the string "*ok*" or something like that). So this is perfect for zero-confirmations payments.
Actually I do send the *ok* confirmation when blockchain.info sends a callback with 0 confirmations, but still take 1-10minutes (I see this in the log). Same if I go though the shared wallet (which is protected against doublespending at 0 confirmations). I have the browser's page be redirected to a page ready2download.php that refreshes every 10 seconds until this 0th confirmation is received, at which point that page turns into an file download.