I think watching a hardware-accelerated video triggers the UVD profile in the BIOS of the video card. This profile is almost always at 400MHz core. After you close the video, the clock goes back to whatever it should be; idle or full-load.