Don't feel the need to vote high or low on a build just because everyone else does. Peer pressure is no reason to change your vote. Even though your vote is different from other votes, it cannot be removed without a legitimate reason.
Other Votes Edit
If you believe that another vote is factually wrong, post the reason why on the voter's talk page. Usually, this is accessible in the voter's signature. Make sure to add a link to the build. By looking on most users' talk pages, you can see examples of this. If the voter does not respond, post the problem on the Administrator's Noticeboard.
Your comment should reflect the rating you have given the build, you should not vote a build "great" and say "this build is trash". The comment you give should explain why you've given it what you have, why do you think it deservers X in effectiveness or Y in universality?. For builds you don't give a perfect score (and sometimes even those that you do), give feedback on how to improve the build "It lacks skill X, which would make it a lot better because of Y". Note that votes without sufficient reason can be removed at any time by Administrators, so ensure you have explained your reasoning thoroughly.