Speaking in Brussels earlier this week, Duff said, "The real problem, post 23 June, is the division in the EU between the 'extend and pretenders' and federalists."
European Council President Donald Tusk, Eurogroup president Jeroen Dijsselbloem, European Commission First Vice-President Frans Timmermans and German finance minister Wolfgang Schauble are classed as "extend and pretenders," according to Duff.
This group, warns Duff, could be lined up against "the federalists" who he says include Commission President Jean-Claude Juncker, European Central Bank president Mario Draghi and the European Parliament.
Duff, who supports the UK remaining in the EU and is a noted Europhile, was speaking at the European Policy Centre (EPC) where he is a visiting fellow.
The ALDE group deputy said, "Britain has been in and out of Europe for hundreds of years.
"The choice is between hard Brexit (leave) and soft Brexit (remain). The former is much worse for the UK - but the latter is not unproblematic for the EU-27."
Participating in an EPC debate on Brexit on Tuesday, he said, "The European Council gave in too easily to David Cameron on an opt-out from an 'ever closer union' in February.
"When it comes to real treaty change, they will have to insist that ratification of all further treaty amendments takes place at a lower threshold than unanimity.
"Only this will neutralise the Damocles Sword of the British government's European Union Act 2011 which imposes endless UK referenda on all constitutional developments in the EU."
The meeting heard that while the Brexit referendum is of the utmost importance, the debate so far there has been an "alarming lack of strategic and positive narrative about Britain's historical and future role within Europe."
Another speaker was Fraser Cameron of the EU-Asia centre, based in Brussels, who took part in the last EU referendum campaign in the UK in 1975.
Cameron said that whatever the outcome next week "the EU poison in the British political bloodstream will not go away."
Duff and Cameron are joint authors of a new EPC publication which draws on the past to lay out the case for a "positive future of engagement" within the EU.
It includes contributions from 14 prominent Britons who have extensive first-hand experience of the EU institutions, including Stephen Wall, David Hannay, Heather Grabbe, Malcolm Harbour and Caroline Lucas, both former MEPs.
The debate was part of a wider series of activities on the EU-UK relationship funded by the King Baudouin Foundation and the Robert Bosch Stiftung.
It comes as UK Labour leader Jeremy Corbyn issued "an appeal to the whole labour movement and Labour supporters across Britain to support a Remain vote in next week's referendum".
The call is backed by leaders of eleven trade unions.
Elsewhere, an online YouGov poll for The Times puts Leave on 46 per cent, up three points since the end of last week. Remain is on 39 per cent, down three - giving Brexit its largest lead since the start of the campaign.
An ICM online poll gave the Brexit campaign a six-point lead, with Leave on 53 per cent and Remain on 47 per cent.