If you have single IPv6 address per interface then your IPv6 deployment is somewhat nonstandard. The canonical way of deploying IPv6 is with at least two IPv6 aliases per interface: one "permanent" and at least one "temporary" that is changed on a regular schedule. The "permanent" is mostly for incoming connections, the "temporary" is mostly for outgoing connections. If you have man long-lasting outgoing connections (like e.g. Bitcoin, unlike e.g. HTTP) you should have multiple simultaneous "temporary" IPv6 addresses active on the interface.
On both my VPS' of Bitcoin nodes, I have a single eth0 interface with 1 IPv4 and 4 IPv6 addresses.
And my setup on IPv6 was initially "standard". But I thought that causes the problem on IPv6 Bitcoin node connections as the the outgoing packets use different IPv6 address than the IPv6 address used by the incoming packets which Bitcoin node listens to. That is why I made sure that Bitcoin only uses a single IPv6 address for both incoming and outgoing packets. But this turned out to be in fact an issue according to you.
I am not sure what you meant by "permanent" and "temporary" addresses. All IPv4 and IPv6 addresses on my eth0 are all permanent addresses. Perhaps the "temporary" address that you meant is the "link-local addresses".
This becomes more confusing than I expected. Unfortunately, there is no documentation related to this that I can find.