For most of the now almost-forgotten vice presidential debate, Indiana Gov. Mike Pence kept his cool, ignoring, deflecting, or outright denying any effort by Virginia Sen. Tim Kaine to tie him to his running mate, Donald Trump. But it’s hard to keep your composure for the length of a debate. It takes work. And toward the end of the 90-minute showdown, Pence began to falter, and then with a single infelicitous phrase he evoked the only wall Trump will ever build: the one between the Republican Party and Latino voters.

It happened after Kaine returned to Trump’s rhetoric, pressing Pence to answer for his running mate’s insults and bigotry. “When Donald Trump says women or Mexicans are rapists and criminals … or John McCain is not a hero, he is showing you who he is,” said the Virginia senator, to which Pence had a reply. “Senator,” he said, “you’ve whipped out that Mexican thing again.” Adding, “There are criminal aliens in this country, Tim, who have come into this country illegally who are perpetrating violence and taking American lives.”

That Mexican thing. That Mexican thing, to be precise, is Trump’s anti-Hispanic demagoguery, which stretches back to the beginning of the campaign. “When Mexico sends its people, they’re not sending their best,” said Trump when he opened his campaign last summer. “They’re sending people that have lots of problems, and they’re bringing those problems with us. They’re bringing drugs. They’re bringing crime. They’re rapists. And some, I assume, are good people.”

The Trump-Pence ticket is well behind its Democratic opponent, trailing by more than 7 points in the Pollster average. And the reason, in part, is his incredibly poor performance with Hispanic voters, who see Trump as an unacceptable risk. This is no small thing. Political parties can make inroads with groups that disagree with them—that’s just persuasion. But it’s difficult, if not impossible, to make gains with groups that see you as a danger to their futures. By placing Donald Trump at the top of the ticket and indulging his nativism and xenophobia, the Republican Party has said with its actions that it doesn’t want Latinos in its tent. Republicans may thus end up estranged from another group of nonwhite voters—not because there aren’t conservative or right-leaning people in the Latino community, but because the GOP has shown itself hostile to the idea of a pluralistic, multiracial America with room and opportunity for Americans of all origins.

You’ve seen this throughout the election, from Trump’s attacks on federal judge Gonzalo Curiel to his performance at the Republican National Convention—where he cast Hispanic immigrants as an existential threat to American life—to his marquee speech on immigration in Phoenix, Arizona, where he called for a new national focus on rooting out unauthorized immigrants. “Under my administration,” said Trump, “anyone who illegally crosses the border will be detained until they are removed out of our country and back to the country from which they came.”

For Trump, Latino immigrants join Muslims and Syrian refugees as potential threats, fundamentally incompatible with American life. If they’re here, they have to be removed, and if they’re not here, they need to be kept out. In turn, for Latino Americans and their families, this makes Trump an existential threat to their lives and livelihoods. Only 21 percent of Latinos say the GOP cares about their community, and 70 percent say that Trump has made the Republican Party more hostile to them. In another survey, polling and research firm Latino Decisions asked Latino registered voters to gauge two statements: “Donald Trump’s campaign talk and policy views make me fear for the future of my family and our country” and “Donald Trump truly has the best interest of my family and our country in mind.” Eighty-two percent of respondents agreed with the first statement, that Trump makes them fear for their families and their country. Eighteen percent agreed with the latter.

This isn’t the first time the Republican Party has lost an entire demographic group through hostility and disdain. At the dawn of the 1960s, as Arizona Sen. Barry Goldwater rose to become the voice for ideological conservatism, black Republicans warned that his ideology would create an irreparable breach between the GOP and its black supporters. “Any Negro who helps the cause of Goldwater, should be declared anything but a Negro, because they will be a traitor to the Negro people,” wrote a party member named Jackson R. Champion in an August 1964 editorial letter to the New York Amsterdam News, one of the oldest black newspapers in the United States.

As historian Leah Wright Rigueur notes in The Loneliness of the Black Republican: Pragmatic Politics and the Pursuit of Power, this was far from atypical. Baseball pioneer Jackie Robinson, who’d endorsed Richard Nixon in 1960 and attended the 1964 Republican National Convention as a “special delegate” for New York Gov. Nelson Rockefeller, “asserted that any black leader who demonstrated support for [Goldwater] would lose power and influence since ‘the Negro is not going to tolerate any Uncle Toms in 1964.’ ” She recounts the story of one black voter, a Cleveland man who backed Richard Nixon in the 1960 election, stating that Goldwater “isn’t for us … and I’d be a damn fool if I voted for a man who’s against me.”

There was no mystery to black hostility for Goldwater. The Arizona senator rejected the moderate Republicanism of Dwight Eisenhower (and many black party members) in favor of a doctrinaire conservatism that closed all doors to government action in civil and economic life. Rather than a “dime store New Deal,” he promised small government, individual liberty, and states’ rights.

If Goldwater’s libertarian-style ideology drew worry from black Republicans—who believed the government should work to uplift black Americans through economic programs—then his “states’ rights” advocacy triggered outright alarm. Then as now, “states’ rights” was a clear sop to Southern reactionaries who opposed a robust federal role in civil rights. Worse, Goldwater disparaged the Republican effort to win over black citizens, a key swing vote in 1960 who, with a little more outreach, might have delivered the White House to Richard Nixon.

“As he expounded during a March 1961 meeting,” recounts Rigueur, “the party should ‘quit trying to win Negro votes’ since the African American electorate had no sense of loyalty or gratitude. ‘Party leaders should face up to the fact that although Republicans have done more for the Negro,’ Goldwater complained, ‘Democrats are getting more and more of their votes.’ ” A later speech was even more direct. “We’re not going to get the Negro vote as a bloc in 1964 … so we ought to go hunting where the ducks are.”

The result of Goldwater’s rhetoric and ideology was a historic rout with the black electorate. Just 6 percent of black voters cast ballots for the Arizona senator, an 81 percent drop from the last presidential race, when Nixon won 32 percent of black ballots. (And before Nixon, Eisenhower had won 39 percent of black voters in the 1956 election.) The repudiation of Goldwater would be the last step in a realignment of black Americans that had begun in the days of Franklin Roosevelt. From then on, the national Republican Party would struggle to crack double digits with black voters, despite the strength of traditional beliefs and practices among black communities, from religious practice to traditions of self-help and self-reliance. The reason was straightforward: Goldwater wasn’t just offensive; he articulated a vision of national life that would inevitably leave black Americans on the margins as second-class citizens, subject to the whims of segregationists and their allies. And if Goldwater didn’t see it, his explicitly anti-black allies did. As Rigueur writes, one black delegate to the 1964 Republican National Convention, George Fleming, “ran sobbing from the convention floor, crying that he was sick of being abused by Goldwater supporters. ‘They call you ‘nigger,’ push you and step on your feet,’ he muttered to reporters, wiping tears from his eyes. ‘I had to leave to keep my self-respect.’ ”

The ideology that Goldwater articulated and his party embraced would put the modern GOP fundamentally at odds with America’s people of color. No matter the temperamental affinities that might exist between some nonwhites and the Republican Party, attempts to bring them into the fold inevitably run up against a key reality: that movement conservatism—the starve-the-beast, libertarian mode that dominates contemporary Republican politics—is a white ideology. Unresponsive to the particular concerns of nonwhites—around issues of discrimination and racial inequality—it is a drive to preserve a status quo built around the political and economic dominance of white Americans. This is more explicit in the age of Donald Trump, but it’s always been true, from Goldwater to the present.

Like their black counterparts in the first half of the 20^th century, Latino voters have been a fluid constituency.  The GOP’s worst performance was in 1996, when Bob Dole won just 21 percent of Latino voters. The culprit was California’s Proposition 187, passed in 1994 under Gov. Pete Wilson, which explicitly targeted Hispanic immigrants. It denied public services like health care and education to anyone who had illegally entered the country, and was part of a wave of anti-immigrant sentiment in the state’s politics. Most of its provisions were eventually struck down, but its political impact would transform California politics.

George W. Bush’s 2000 and 2004 campaigns were explicit efforts to undo the damage Proposition 187 had done to the national Republican Party, and it worked. In the 2004 election, for example, exit polls had George W. Bush with 44 percent among Latinos, an all-time high for a Republican presidential nominee. Subsequent analysis revised that number down to 40 percent, still an improvement over 2000, when Bush won 35 percent of Latino voters. His performance put him above Ronald Reagan, who won 37 percent of Latino voters in his 1984 landslide.

But the gains were only temporary. In the wake of Bush’s failed attempt to pass comprehensive immigration reform, the Republican Party made a decisive return to immigration restrictionism. In 2008, under pressure from the Republican base, John McCain abandoned his support for his own immigration bill, hitting a hard-line note to secure his right flank. “One thing we would all agree on, the status quo is not acceptable. We have to secure our borders,” said McCain during one presidential debate.

The Great Recession supercharged anti-immigration—and anti-immigrant—sentiment among GOP voters, and Republican lawmakers followed along, passing restrictive new laws after the party swept statehouses in the 2010 midterm elections. Indeed, we saw an inkling of rising anti-Latino sentiment in 2009, when Obama’s nomination of Sonia Sotomayor to the Supreme Court was met with a wave of racist criticism. “She is an affirmative action pick,” declared commentator Pat Buchanan. “Clearly the president was down to four choices, all of them women, and he picked the Hispanic.”

In 2010, Arizona embraced a measure that all but criminalized brown skin. Not only was it now illegal for unauthorized immigrants to apply for jobs or solicit work, but police were empowered to consider “race, color, or national origin” when questioning someone’s citizenship. Deemed the “papers please” law by critics, it inflamed sentiment against the Republican Party, and was eventually invalidated by the Supreme Court.

Despite the backlash, Republicans would move further down the path of restrictionism. In 2012, Massachusetts Gov. Mitt Romney endorsed “self-deportation,” a strategy—devised by anti-immigration activists like Kris Kobach and Mark Krikorian—to force unauthorized immigrants to leave the country of their own volition, by making life intolerable for them and their families. As Romney explained during a Republican presidential debate, “We’d have a card that indicates who’s here illegally. … And if people are not able to have a card, and have through an E-Verify system determine that they are here illegally, then they’re going to find they can’t get work here. And if people don’t get work here, they’re going to self-deport to a place where they can get work.” Romney took the right-most position on immigration, and used it to sink his strongest rivals, like then–Texas Gov. Rick Perry.

Unfortunately for Romney, however, “self-deportation” would cling to him through the general election, and weigh him down with Latino voters. In the aftermath of his defeat, Republican leaders—including the Republican National Committee—would finger his immigration stance and call for reform, prompting a new effort, led by Florida Sen. Marco Rubio, to pass comprehensive immigration reform.

Like Bush before him, Rubio would crash against conservative opposition, organized by the most hard-right and anti-immigrant voices in the party, from Alabama Sen. Jeff Sessions and Iowa Rep. Steve King to conservative activists and radio hosts. By 2014, immigration reform was all but dead, killed by hard-line Republicans in the House of Representatives.

The underlying fact, through all of this, was the intense anti-immigration and anti-immigrant sentiment of Republican voters. It fueled explicitly anti-immigration politicians like Arkansas Sen. Tom Cotton and played a part in House Majority Leader Eric Cantor’s surprise primary defeat to Dave Brat, an obscure conservative activist. And in the summer of 2015, it would shoot Donald Trump to the top of the polls, where he would stay, feeding sentiments with wild and prejudiced claims against a constellation of foreign targets, and Hispanic immigrants in particular.

What’s popular with the Republican base has been poisonous with Latino voters. The latest tracking poll from Latino Decisions shows Republican nominee Donald Trump with 16 percent support, versus 74 percent for Hillary Clinton. Looking ahead to November, the group expects that electorate to cast the vast majority of its votes for Clinton, 82 percent to 15 percent for Trump, which would be the most lopsided total in history. If this happens, Latino Americans will join blacks as one of the most Democratic groups in the country, and at 57 million strong (including 27.3 million eligible voters), one of the largest, too. According to a new survey from the Pew Research Center, 69 percent of Latino registered voters say they are “absolutely certain” to vote, and 44 percent say they are voting to oppose Trump.

Both voting and party affiliation are habits. Once set, they tend not to change. And looking forward from the present, there’s a real chance that Latino voting takes the same path as black voting, with routine and overwhelming support going to Democratic presidential candidates. Again, California gives us a glimpse of what this looks like. Following Republican Gov. Pete Wilson’s anti-immigrant crusades in the 1990s, Latino Californians moved decisively into the Democratic column. By the end of the last decade, the state was a Democratic stronghold, with Democratic officeholders at almost every level of government. The once-dominant California Republican Party, by contrast, was a rump force, with little power or influence outside of its historic strongholds in the state’s interior.

In the wake of Obama’s election, the national Republican Party was already on this path. But Trump has been an accelerant, driving Latino Americans away from the GOP with xenophobia and unyielding hostility. And in fact, this has had an unintended side effect: Asian Americans are leaving the Republican Party, too, in record numbers, and for similar reasons. A GOP that nominates Trump—and embraces nativism—is one that lacks room for all immigrant and nonwhite groups.

Black Republicans resisted Barry Goldwater and his brand of doctrinaire, ideological conservatism. They backed moderate candidates for president, built alliances, and formed organizations meant to roll back the rising tide of movement conservatism. In its manifesto, one of those organizations—the Negro Republican Organization—promised to “oppose any attempt to give the public any impression that Negro Republicans can unite behind Goldwater” and “make every effort to defeat the minority segment which appears determined to establish a lily-white Republican Party.”

The NRO lost its fight. That “minority segment” is now a majority, and in the past 10 years, it has finally established a lily-white Republican Party. It’s tempting to say that this GOP won’t survive—that it’s too homogeneous to succeed in a modern, diverse America. But if Donald Trump loses this election, he will likely lose by just a few points. With a nicer suit, steadier hands, and a better attitude, Trump—or at least Trumpism—could succeed nationally.

And for those Republicans who don’t want Trump or Trumpism? It may be too late. The thing about a lily-white Republican Party is that it doesn’t have the diversity it needs to resist white resentment and white rage. Republicans crossed a point of no return. Raw ethnonationalism is their future, even if they don’t want it.

Read more Slate coverage of the 2016 campaign.