The warning comes from Rye county councillor Keith Glazier who says continued criticism could jeopardise the long-awaited project.
But objector's claim current plans would put pedestrians at risk and insist safety must come first.
Plans for the new library, at Cinque Ports Street, have been attacked from all quarters. The highways authority has objected on safety grounds, The Friends of Rye Library say it would not be big enough and the design has come under fire from both Rye Conservation Society and English Heritage.
Rother's planning committee has put the application on hold until solutions can be found.
The highways authority said: "The proposal fails to provide any designated area for loading and unloading or a drop off zone for visitors.
This would result in traffic and pedestrian conflict.
"It would introduce hazards by the slowing, stopping, turning and reversing traffic which would be created."
Cllr Glazier told Rye councillor's on Monday: "I am being pragmatic - unless a library is approved on that site you will not get one.
"I believe the safety concerns are being addressed.
"As to library design, I have had assurance from the library service that everything is in accordance with what they want.
"These are the professionals and they have said it fits their criteria exactly.
"This is the last opportunity Rye will have to get a new library in a central location."
Cllr Peter Dyce said: "There are a number of grounds for objection. There are concerns over safety, apartments above the new library would overlook other properties and English Heritage have severely criticised the design."
Cllr Keith Taylor said: "The highways authority at Lewes rejected the scheme because it is an extreme danger.
"Many library users are old and have mobility problems.
"As much as we would like to see a new library, we don't want something that would be lethal to library users."
Cllr Sam Souster said: "The friends of Rye Library will have to decide soon if they are going to continue to oppose it or accept a compromise because I think there is a real risk that the owner of the site will lose interest soon."