As the comments have already mentioned, downvoting spam really doesn't cost you any rep at all. Downvoting questions never costs rep, and downvoting answers temporarily costs 1 point, but if the answer gets deleted (as it will with spam), you get your point back.
However, it is still recommended that you not downvote spam and simply flag it instead, per this Meta.SE question:
Why shouldn't I downvote spam that I've already flagged?
The reason given is that downvotes hide the spam from the front page, making it less likely that it will get the required number of spam flags to delete it completely without moderator attention. Downvotes don't really hurt the spammer's rep, since they likely have a rep of 1 and the account will be deleted soon anyway.
Having said that, I still question the importance of this recommendation.
The whole point of deleting spam is so that people won't see it. If it gets downvoted enough to hide it from the front page, then fewer people will see it, and that is a good thing.
As soon as it gets one spam flag, more people will be able to see it and flag it in the review queues. There are several review queues it could show up in: First Posts, Close Votes, Low Quality. It'll get its opportunity for more flags. At the very least, a moderator will see it and delete it, and a moderator is required to destroy the spamming account, anyway.
My recommendation is definitely flag any spam you see; that is important. If you happen to downvote it as well, I don't think it matters much.