Maryland Gov. Larry Hogan (R) on Sunday said conservatives are slowly pushing back against former President Trump but that it’s “taking longer than it should.”

Hogan told moderator Chuck Todd on NBC’s “Meet the Press” that Trump’s “influence is diminishing” and that the Republican Party will slowly distance itself from the former president. The Maryland governor, who has long been critical of Trump, also said he no longer feels he is a lone dissenter.

“I’ve been talking about this for years now, and I felt like I was on a lifeboat all by myself. But now we need a bigger boat because more and more people are speaking out every day,” Hogan said. “I said Trump’s influence on the party was going to diminish over time. It hasn’t happened rapidly, but it has diminished dramatically.”

“It’s just taking longer than it should,” he added.

Trump is facing backlash from some in the GOP for his role in the Jan. 6, 2021, attack on the U.S. Capitol.

The House panel investigating the riot has held a series of hearings highlighting the events leading up to Jan. 6, including one that featuring a former White House aide who said Trump encouraged armed rioters to march toward the Capitol.

Hogan on Sunday said the diminishing influence of the former president could be seen in several Republican candidates Trump has worked against who have prevailed in their primary challenges, including Georgia Gov. Brian Kemp and Rep. Nancy Mace (S.C.).

The Maryland governor, whose term ends this year and who is flirting with a potential White House bid, said America wants to move on from Trump.

“There are an awful lot of people — I would call them the exhausted majority of Americans who are frustrated with the far left and the far right — they want to see us go in a different direction,” the moderate Republican told NBC. “I’ve seen nothing to dissuade me from thinking that there’s a growing demand for exactly what we have done in Maryland over these last eight years.”

Updated at 12:24 p.m.