But it is impossible to prove this from the generated signature.
That's why we would love to setup a deterministic build environment using Vagrant or Docker. So independent "builders" can easily run it and confirm that the signed distributed binary comes really from the published source which can be audited. Currently other tasks have higher priority, though.