They're complaining because you're late and others are early (relative to you).  There's nothing more or less to it than that.