Here is a suggestion - have it displayed at the bottom of every story, more than just the rating and number of votes, but the usernames and times of voting (if the latter is stored in the sql database) of each one. Or possibly a button that when pressed makes that information visible, in case something has a vast quantity of votes and this would be a long list. It would also be damning if suddenly, minutes apart, 2 low votes happened because that would be telling that it was one individual with multiple accounts voting it down multiple times. Because then you'd be able to see not just the rating, but if it was only low because the same little bloodsucking insect came along for every one of your stories, so that you could neglect that one mosquito's opinion across all stories whenever you see them on OTHERS' stories too, whenever you see it, and if the time of voting was displayed as well, everyone would also know if two accounts were actually the same person because there would be multiple incidents of those two accounts voting together.
The Galactic Emperor 10 pm Eastern 7-08-17: 10 (1 vote)
1 am Eastern 7-09-17: 10 (2 votes)
9 am Eastern 7-09-17: 6.6 (5 votes)
It's obvious what's going on here (none of these are me voting on my own story btw - I gave it a 10 after this just to bring up the unfairly low average, but you can read it for yourself and see if it deserves a 1 or if I'm being pursued by a vindictive idiot - and I'd like to see who, I'd like to see this idiot's stories if they're supposedly so great by comparison).