SPA is about power analysis, not time. Reading memory from different areas will result in different power signatures for that code. Of course balancing this is highly tricky and architecture dependent.

Performing SPA on the TREZOR crypto scalar multiplication code is harder when USE_PRECOMPUTED_CP is defined (otherwise you're doing a textbook SPA), but I believe it's still doable (you read 2 bits pattern together)

It's definitely not the main concern if you have physical access to it though