In my view, it's slavery. In a direct election system, the South would have lost every time because a huge percentage of its population was slaves, and slaves couldn't vote. But an Electoral College allows states to count slaves, albeit at a discount (the three-fifths clause), and that's what gave the South the inside track in presidential elections. And thus it's no surprise that eight of the first nine presidential races were won by a Virginian. (Virginia was the most populous state at the time, and had a massive slave population that boosted its electoral vote count.)
This pro-slavery compromise was not clear to everyone when the Constitution was adopted, but it was clearly evident to everyone when the Electoral College was amended after the Jefferson-Adams contest of 1796 and 1800. These elections were decided, in large part, by the extra electoral votes created by slavery. Without the 13 extra electoral votes created by Southern slavery, John Adams would've won even in 1800, and every federalist knows that after the election.
And yet when the Constitution is amended, the slavery bias is preserved.