Under Magneto's leadership, the team often encountered another group of mutants, the X-Men, in battle. Wanda reluctantly remained with the group out of feelings of loyalty, indebtedness, and fear. But when Magneto was abducted by the powerful alien known as the Stranger, she and Pietro left the others.
The Scarlet Witch and Quicksilver petitioned for membership with the Avengers. They, along with Hawkeye (another individual with a criminal background), were inducted while the charter members took a leave of absence. They were led by Captain America and became affectionately known as Cap's Kooky Quartet.