In a statement on Tuesday, she said she would not be "formalising my recent nomination".
The 56-year-old MEP for South East England said she did not have "sufficient authority" to see through changes she had planned.
Reaction was swift, with Roger Helmer, a former Tory MEP, telling this website, “I am sorry to lose her, but glad she was prepared to bite the bullet and not to let the situation drag on.”
ECR group MEP Ashley Fox, leader of Conservative deputies in the European Parliament, said, "Diane James' resignation after just 18 days in charge reveals what has been clear for some time, that Ukip is consumed by infighting and in total disarray.
"Her admission that she does not have the support of either her MEP colleagues in Brussels or party officials in the UK to implement the changes that formed the basis of her successful leadership campaign is extraordinary.
"With Labour also a rudderless shambles, it is clearer than ever that the Conservatives are the only party fit and able to lead Britain."
More comment came from the UK's former Europe Minister Denis MacShane, who said, “Ukip has always been a one-man, one-policy show. Once the entertainer-in-chief, Nigel Farage, retired having won his only policy objective - quitting Europe, the show was over. I debated with Diane James in the Brexit campaign and Nigel Farage she isn’t.
“It was one of the worst performances from any elected politician I ever heard. Ukip have never one a single seat in the UK Parliament unless via Tory attention-seeking defectors.
“They win a few council seats, often inherited from the BNP after the BNP faded away to be replaced by Ukip as the main vector of xenophobic and anti-Muslim politics in Britain.
“But Ukip has never been more than a single issue party and now that issue has gone away and all their MEPs will be on the dole after May 2019 there is little point in the party existing, let alone fretting about who leads it.”
James, who says she will continue as an MEP, succeeded Nigel Farage on 16 September after he quit in the wake of the UK's vote to leave the EU.
James said in a statement that she had decided "for personal and professional reasons" not to take up the helm of the party, but would stay on as an MEP.
"It is with great regret that I announce that I will not be formalising my recent nomination to become the new leader of the party with the Electoral Commission," she said.
"Having won the enthusiastic support of party members, I was nominated by them as the new leader at the recent Ukip Bournemouth conference.
"Since that time, I have been in discussion with party officers about the role. It has become clear that I do not have sufficient authority, nor the full support of all my MEP colleagues and party officers to implement changes I believe necessary and upon which I based my campaign.
"For personal and professional reasons, therefore, I will not take the electoral process further.
"I will continue to concentrate fully on my activities and responsibilities as an elected Ukip MEP for South East UK Region.
"This is my final media statement on the issue.”
Ukip now faces the prospect of a second leadership contest within the space of weeks, after a divisive race over the summer that exposed deep rifts between rival wings of the party.
Front-runner Steven Woolfe, who is also an MEP, was barred from standing after submitting his nomination papers late, while Suzanne Evans was unable to take part as she was suspended.
The pair were immediately named by bookmakers as favourites in what looks set to be a second leadership election within a matter of weeks, with Ladbrokes putting Farage as a 10-1 shot to return.
Speaking on Wednesday, Farage poured cold water on speculation he might return to lead the party for a third time, after stints from 2006-09 and 2010-16, telling the UK’s Press Association: "Not for 10 million dollars."
And asked if he would take 20 million dollars, he said: "No, I'm not coming back, I'm retired."
However, he has since changed his mind, stepping in as interim party leader.
Ukip Chairman Paul Oakden said he “regretted” James’ decision, adding, “I will now look to convene an emergency meeting of our National Executive Committee to confirm the process for electing Diane's replacement.”