I think part of the issue is that everyone has different ideas about what is on- or off-topic. Yes, we all can read the on-topic ruleson-topic rules, but applying these rules to individual questions is very subjective. Some of us tend to be more open toward questions that are on the edge, and others of us like to close anything that approaches the edge.
Looking at my own answers: Of my most recent 60 answers, 2 were on questions that were eventually closed as off-topic (oneone, twotwo). A few others did collect a few close votes, but not enough to actually close the question. For each of these questions, I can understand why someone would vote to close, and yet, in my opinion, they should remain open. (If I didn't see these as on-topic, I certainly wouldn't have answered.) I often issue a Vote-to-Reopen on questions after they are closed, but it is rare that a question gets five reopen votes after being closed. It is just the way Stack Exchange works; the system, in my opinion, is biased toward closing questions and keeping them closed.
As to your proposal to start mass deletion of questions that are closed as off-topic, I have to disagree. Questions with positive score answers should not be deleted, in my opinion, unless there is a specific reason why a particular question needs to be deleted. If you look at the help page for the deletion privilegedeletion privilege, you'll see warnings against deleting questions that have good answers. In fact, the more answers a question has, the harder the system makes it to delete. This is all by design. Closed questions are not supposed to be automatically deleted, unless there is absolutely no value (perhaps the question makes no sense, or it only has negative score answers).