Jessica, I agree with you that some of the hostility that women who give birth in late middle age face derives from the way that giving birth at that age basically requires you to have immense amounts of privilege that others don't. Of course, a lot of women in their early 30s also go through IVF hell, and they don't face that prejudice--maybe because they "pass" as someone who conceived the free way?--but I am surprised at how little the original New York magazine story by Lisa Miller on the experiences of much-older parents avoided discussion of sexism. In fact, Miller went out of her way to make it seem like the disgust people have for older parents is gender-neutral, using the word "parent" frequently and even invoking a middle-aged gay couple who have two daughters by a surrogate.
But one group of people has been having kids in late middle age since roughly forever, and I think any discussion of the phenomenon without mentioning them is falling short: middle-aged men married to younger women. These men tend not to face much social blowback for the choice to spend their retirement years parenting. Part of that is no doubt for the reasons you cite: Since they conceive naturally and for free, people excuse it. But I have to imagine that male privilege plays a role in it; since fathers aren't expected to do as much of the hands-on work of parenting, they're given a pass for criticisms that their age will make them less energetic when their kids are young and less present when their kids are older.
I have some personal experience with this. My paternal grandfather was a month shy of 48 when my father was born. My grandmother wasn't scandalously younger than my grandfather--she had her kids later than most women of her generation, for various reasons, though she strongly implied that a fun-loving spirit was amongst them to us grandkids--but obviously, she was younger. I have to say that for all the tight-lipped judgment of vast age gaps in the family tree that Miller describes, for myself it was pretty cool having really old grandparents. My grandmother, for instance, would joke about how she dragged my grandfather to a date to a Technicolor film, and he was reluctant because he'd given up on movies when they introduced sound. The downside, of course, is you lose them too soon. Even though he had a very long life, I was still only 17 when my grandpa passed away. But I can't say that I ever felt anyone judging my family for this. At worst, it was treated as a novelty that he was so old, and certainly he was no less beloved for it. I would also recommend having kids spend a lot of time around truly elderly people. I grew up with a conception of the elderly as being the kind of people who could get up and do interesting things, which gives me a lot of hope for my own future, thanks to my dad's parents.
So, I agree with you that a lot of the hostility to middle-aged mothers comes from the privilege it takes to achieve that. But I can't help but think that it's also just a combination of people blanching because it's weird and because we have more firmly entrenched ideas of who has what it takes to be a mother.