No matter what one is doing in life, there are people who know what one ought to be doing instead. Once people give up on talking you out of having so many kids, they will change tactics and try to talk you out of having kids when you're old (unless, of course, you have no kids and are old; then it is still OK for you to have a kid and actually not really OK for you not to).
Since we are very knowledgeable nowadays, the reason old people with kids should not have kids is that it's dangerous for old people to have kids. All your kid making components are rotting (and so are his, even), so you're going to have rotten kids, and having a rotten kid would be really bad and actually even selfish and pretty much almost immoral.
This is basic eugenics, and we are all naturally eugenicists in our own ways. It's just one of those things we have to recognize and learn our way out of. Here are the basic lessons for dealing with this particular line of eugenic thought:
--God puts an end date on each woman's fertility. When she's too old to have kids, she can't. If she can still have kids, she's not too old. (I mention this just to help us keep from mixing biology with ethics or morality.)
--Every human life is worth living, no matter how miserable it looks to the strong, the healthy, the gifted, the intelligent, the rich. To consider another life and say that it is not worth its own cost is nothing but prideful condescension. This judgment does not happen in a vacuum. To say, "You shouldn't risk having a Down's baby" is to look at a living person with Down's and say, "There should not be another person like you." There is only one person of whom God ever said, "It had been good for that man if he had not been born," and it wasn't because that man had a genetic defect.
--There is no person so un-rightly made that he does not show forth the image of God. There is no human so poorly endowed that she cannot receive the gift of forgiveness and salvation. There is no one so ugly, so weak, so sick, or so empty that he is beyond our risen Lord's power to make all things new.
--You don't know what baby you're going to have until you have it.
To avoid the increased risk of genetic problems or whatever other harms to which the children of older parents are more subject is to take the wrong gamble. So here is the general answer I have on hand for the caring people who want to make sure I know there's a bigger chance now that a baby I have could have something wrong with it:
"Thank you for telling me about that risk. I read an article about it too. If our Lord sees fit to give old people like us another baby, the greatest likelihood is that that baby will be healthy and well."
Practice it in your head so you can say it instead of what you're really thinking.