I’m not entirely sure if downvoting should be removed entirely, but I think “what it does” should be changed fundamentally. The value of downvoting is so that people don’t have to wade through enormous piles of submissions in order to find the good ones. But it does this whether it removes the offending entries from competition entirely, or just shoves them to the bottom of the list.

So what I would like to see is:

However many submissions fit on a page (I think it’s 15), even before voting, the first 15 you see when you go to vote are the 15 of the unvoted which currently have the highest score, in random order. The next 15 you see are the 16-30 of the unvoted, and so on. No submissions ever go away, no matter how bad their scores are. So that way, if someone really wants to dive in whole hog, more power to them. But if someone just wants to play their part in figuring out what the very best of the best ideas are, they only have to go through one page or maybe two of submissions, after a few people have bothered to vote.