The US is rebuffing a demand to halt NATO expansion, with the Kremlin saying it will quickly see if it's worthwhile to even keep negotiating.

At Monday's talks in Geneva, Russia insisted on guarantees precluding NATO's expansion to Ukraine and other ex-Soviet nations and demanded to roll back the military alliance's deployments in Eastern Europe. The US firmly rejected the demands as a nonstarter.

The US envoy to NATO set a tough tone for the next talks with Moscow, ruling out any concessions on the alliance's eastward expansion.

"We will not allow anyone to slam NATO's open-door policy shut," said US Ambassador Julianne Smith.

Kremlin spokesman Dmitry Peskov described the Geneva talks as "open, comprehensive and direct," but emphasised that Moscow wants quick results.

"We see no significant reason for optimism," he told reporters.

Peskov said Russia-NATO talks in Brussels on Wednesday and a meeting of the Organisation for Security and Cooperation in Europe in Vienna on Thursday would show whether further negotiations are worthwhile.

"It will become clear in what direction and how to proceed and if it makes sense," he said.

"We absolutely wouldn't accept dragging this process out endlessly."

Smith said "not a single ally inside the NATO alliance is willing to budge or negotiate anything as it relates to NATO's open-door policy".

"We stand firm in pushing back on security proposals that are simply nonstarters," she told reporters.

The US estimates Russia has massed about 100,000 troops near Ukraine, a buildup that has stoked fears of an invasion.

Moscow says it has no plans to attack and rejects Washington's demand to pull back its forces, saying it has the right to deploy them wherever necessary.

President Vladimir Putin has warned Moscow would take unspecified "military-technical measures" if the US and its allies don't meet its demands.

He spoke with members of his Security Council, saying he wanted to discuss unspecified issues related to security and infrastructure in border areas.

White House press secretary Jen Psaki said it was "too early to tell whether the Russians are serious about the path to diplomacy or not," or whether they will use the talks as a "pretext to claim that diplomacy couldn't possibly work" and move forward with an invasion.

Australian Associated Press