When Lucy DeVito, Brandon Jackson and Cat Deeley left after season two, the show took a much different turn and lost a lot of its charm. Replacing them with Kal Penn made the show feel like it was just one long Harold & Kumar movie.
Honestly, if you only watch seasons 1 & 2 no one would blame you. I'd actually recommend that.