Currently two getblock calls are made for verbose param false and true, then getrawtransaction for each transaction, and then getrawtransaction for each transaction input txid. If getrawtransaction fails (for example block 0), then decoderawtransacion call. Most waiting goes if getrawtransaction is called recursively till generation transactions remains.
That is horribly inefficient. You should not be relying on Bitcoin Core for all of that. Rather, you should be calling getblock once and then getrawtransaction for the each transaction in the block. Then you should be storing data in a separate database which you can access much more quickly. Relying on Bitcoin Core's RPC to get you a lot of data quickly is a terrible idea. You should also batch the getrawtransaction calls so that you aren't sending it thousands of times but just once.
Also, you can try apply the patch in the PR I linked and use the extraVerbose option it provides to get you all of the transaction data in a getblock call.