Facing the threat of losing his job as House speaker, Paul Ryan spoke confidently Monday about his future.

“I feel very good where I am,” Ryan told a handful of reporters at a campaign stop in Racine, Wisconsin, according to Politico’s Jake Sherman and Rachel Bade.

“I’ve gotten such a great outpouring of support from members,” he added. “They know I took the job as a sense of duty, that duty is not done, and I plan on continuing doing that duty.”

In recent days, Ryan has faced a growing chorus of colleagues who refuse to publicly endorse him. Some of those Republicans are disappointed in Ryan’s treatment of presidential candidate Donald Trump while others remain unimpressed with his lack of conservative accomplishments or question his future aspirations.

“At this stage of the game, I don’t know who all is running,” Rep. Jim Renacci, R-Ohio, told Politico. “I’m very supportive of Paul, but when it comes to elections I want to see who’s in before committing [to] who I’m supporting.”

Rep. Mike Pompeo, R-Kan., also declined to endorse Ryan’s bid for speaker. Pompeo considered running for the job last year.

“I’m trying to get to an outcome; all of my conversations with my colleagues have focused on the outcome, not the ‘who,’” Pompeo told The Hill. “If we can get the Republican conference united, we can be successful.”

Rep. Thomas Massie, R-Ky., suggested Ryan’s future aspirations could cost him support.

“It would be very difficult for him to get my vote based on what I assume his motives are, which are to run for president in 2020,” Massie told The Huffington Post last week.

And then there’s the House Freedom Caucus, the conservative group of around 40 members who could ultimately determine Ryan’s fate in January. Last week, members of the group gathered in Washington, D.C., to plot their strategy. And so far, they’re unwilling to endorse Ryan.

“I’m not commenting on leadership elections until after Nov. 8,” Rep. Mark Meadows, R-N.C., told CNN.

Rep. Jim Jordan, R-Ohio, wouldn’t commit either.

“There’s time to deal with who’s in leadership, who may not be in leadership, how leadership is done, what the rules are, what we’re going to focus on in the lame duck,” Jordan, the chairman of the House Freedom Caucus, said on CNBC.

Rep. Dave Brat, R-Va., said the next House speaker needs to make a firm commitment on several conservative priorities, including opposition to a trade deal negotiated by the Obama administration and a firm stance against amnesty for illegal immigrants.

“If he commits to regular order—in writing—and no Trans-Pacific Partnership until we reduce regulations and no amnesty bills and a commitment to reduce federal spending, then I will vote for any candidate who backs these conservative positions that 80 percent of Republicans share,” Brat told CNN.

Next week, Ryan will face his first test when Republicans gather for House leadership elections. The vote is set to take place Nov. 15, just one week after Election Day. It is a precursor to the official vote for House speaker in January.

But there’s already a move afoot to delay that vote until after Republicans have more time to reflect on the election results and discuss the path forward. The effort is being led by Renacci, a centrist Republican from Ohio who claims to have at least 15 other supporters.

“There are fractures in the conference which truly need to be discussed, vetted, and healed,” states the Renacci letter, which was obtained by Politico. “Asking members to vote for a leadership team within 24 hours of their return to Washington without time to reflect on ways of coming together as a conference is truly ill advised. That in itself ignores the reality that the conference is divided. … There is no reason to hastily hold elections.”

Renacci isn’t the first to suggest a delay. In September, the House Freedom Caucus considered making that push as well.

Ryan, meanwhile, is staying positive and continues to tout the policy proposals that are part of his “A Better Way” agenda. He called it “a godsend” on Monday.

“I feel pretty darn good about where we are with our majority,” he said.