Now, responding to your remark: "compare them along any measure and you will find they are different if you look close enough".There are always differences, that's granted. What matters is if it is statistically significant. To be sure there are different experimental designs to filter out, changing criterion, reversal and other that are very well known in medical research, double-blind testing, control groups. We useAnalysis of Variance to tackle that problem, which is the same statistical tool used in any other "hard" science.
I would suggest to stop using that. ANOVAs rely on assumptions (eg normality) that are either known to be false when describing human behaviour or impossible to prove. Do permutation testing instead.
Well that will depend on the type and nature of the experiments that are performed, but thanks for the suggestions
Are you a statistician?
No, just someone who deals with similar situations and realized ANOVAs, etc are not the correct tool for that job. People just use them because its the only thing they were ever taught.
Lets stop this OT convo tho.