I don't think that's a bad thing. Local board users won't get any votes from the majority of users who doesn't read that board, so it's good that some votes reach the local boards if the user is the best match for the category (from your perspective).