Got it.
You said earlier I would have to aggregate it by following the blocks but would it not be easier to just modify the daemon to stop syncing at a specific block height and then use existing RPC api's to query the balances from that node?
Having already done this once, yes, it is way more easy to do this way.