MINOR SPOILERS ABOUND - (character progression, not plot or twists)
As far as the film goes, it would be silly of me to say that I marked out during every scene(except the very beginning which was extremely boring) of the movie, and there were fights and shots in the film that I will likely never forget. I didn't realize that Whedon has only the Serenity directorial credit to his name, which kind of makes this an amazing feat, but given his television and script record, he was a perfect fit.
I really enjoyed Black Widow a lot more than in IM2, and while she had much more fluid dialogue in the film, both her character and Hawkeye's were pretty wasted. Her interrogation scene with Loki was genius, however, and that made an impression on me that she was well involved in the film. Hawkeye being turned baddie at the very beginning left no room for him to expand as a character and everything after that involving him had no impact on me and the relationship I was supposed to latch onto with him.
The most dear scenes to me were of Stark and Banner talking about Hulking out, every Cap and Stark conversation, the "flying monkeys" joke that single-handedly dropped a ton of empathy for Captain America's "man out of time" feelings onto the audience, the numerous jabs at the Avengers cast that Stark hurled at them(Point Break, Legolas, GALAGA jab at S.H.I.E.L.D. operative, prodding Banner to see if he changes, Shakespeare in the Park). I would say that Downey, Jr. stole the show, but I was so involved in what Ruffalo was doing with both Banner and Hulk that I'd have to say he stood out more.
The scene with Loki and Thor where he questions what chaos he's caused is pretty incredible. The pathos of Hiddleston's character is spot on to the character of the comics, and though we don't get to see anguish from Loki for very long, it puts into perspective the relationship he and Thor have and drives home the point of their relationship that began in the Thor movie.
A lot of the fight scenes were incredible, noting the ones that stood out most for me as a fan were Hulk vs. Thor(Thor is the only superhero in the Marvel Universe with enough power to match the Hulk's), Iron Man vs. Thor(and the resulting interference from Cap), Hulk vs. Loki(and the line Hulk drops afterward is awesome), and the numerous beat downs in the streets as the drama really unfolds.
I'll be seeing this one again in theaters a few more times, only without the 3D. The one thing I didn't like about the 3D was that it turned anything not super brightly colored into a muddy mess. The night time scenes were hard to make out and the characters clothes/suits were not as bright as they should be, but the 2D version of the film doesn't suffer from that.