I think part of the issue is that everyone has different ideas about what is on- or off-topic. Yes, we all can read the on-topic rules, but applying these rules to individual questions is very subjective. Some of us tend to be more open toward questions that are on the edge, and others of us like to close anything that approaches the edge.
I definitely fall into the camp of users who like to leave questions open. It's not a secret; I've posted before that, when evaluating questions, I look for a reason to keep them open, rather than looking for a reason to close. It is only when I can find no justification for them being open that I vote to close.
Looking at my own answers: Of my most recent 60 answers, 2 were on questions that were eventually closed as off-topic (one, two). A few others did collect a few close votes, but not enough to actually close the question. For each of these questions, I can understand why someone would vote to close, and yet, in my opinion, they should remain open. (If I didn't see these as on-topic, I certainly wouldn't have answered.) I often issue a Vote-to-Reopen on questions after they are closed, but it is rare that a question gets five reopen votes after being closed. It is just the way Stack Exchange works; the system, in my opinion, is biased toward closing questions and keeping them closed.
As for the motivations of people writing answers, I can only speak for myself. My initial motivation was to try to give something back for the answers that I had received from experts on other Stack Exchange sites. The reasons I'm still here are to learn, to improve my writing, and to help people. Competing for fake internet points, although it might be fun to increase a score, is not a great reason for being here.
I like seeing new questions that I can answer. I think it is a shame when someone has a question, takes the time to post, and then the question gets a knee-jerk reaction of rude comments, downvotes, and closure votes. I just don't understand this. Our site is not overrun with questions, and it doesn't hurt anyone to leave interesting questions up. I understand that question quality is important to the site, but look at the example you posted about median retirement savings: It's about how much people have saved for retirement. Retirement savings are something that we cover every day here. It's a clear, well-written question that has generated a bunch of interesting answers. It has made the hot network questions list, and is bringing people to our site (also a good thing, in my opinion). Why would we want to stop that?
As far as migration goes, I'm not personally a fan of widespread migration. When a question gets closed as off-topic that should have remained open, it at least has a chance of getting reopened. Once it is migrated, it is gone. In addition, when a question gets migrated, the OP often loses control of the question, which means that no further info from the OP is possible. Also, we are not necessarily great at knowing what will be well-received on another site. Stack Exchange seems to agree that migration is a controversial subject. Instead, when I see a question that would be much better on another site, I like to comment and suggest to the OP that he moves his own question, allowing him or her to retain ownership.
In conclusion: When you see a high rep user answer a question that you think is off-topic, ask yourself: Could this question be considered on-topic? Does it hurt the site to keep this question open? The users that answered have already made the decision that yes, it should remain open. You are allowed to disagree, but your opinion, like mine, only represents one person. Vote, state your opinion, and move on. Sometimes your opinion will ultimately win out, and sometimes not.
To address your question: How do we discourage high-rep users from answering questions that are clearly off-topic? I did my best to address it by asking the meta question that you referenced. But, I'll be honest, when I see someone who answers and votes to close, usually my problem is less with their answer and more with their close vote.
As to your proposal to start mass deletion of questions that are closed as off-topic, I have to disagree. Questions with positive score answers should not be deleted, in my opinion, unless there is a specific reason why a particular question needs to be deleted. If you look at the help page for the deletion privilege, you'll see warnings against deleting questions that have good answers. In fact, the more answers a question has, the harder the system makes it to delete. This is all by design. Closed questions are not supposed to be automatically deleted, unless there is absolutely no value (perhaps the question makes no sense, or it only has negative score answers).