If we’re so concerned about very special amounts of downvotes, the solution could be as simple as tying in what votes count to your voting status.

If you’re a brown nose, your positive votes don’t actually count. You’ll see the number and think they make a difference, but they don’t. Likewise, for whatever the negative version of a brown nose is, your negative votes don’t count.

then favorable and unfavorable would have half of your positive/negative votes count, or will only count them when you’ve voted negative/positive at least once.

Fair voters get normal vote counts.