English Heritage says that it is "appalled" by the "incomprehensible" decision of communities secretary Hazel Blears to approve the 43-storey Doon Street Tower on the South Bank.
It emerged on Wednesday morning that Ms Blears has overturned the recommendation to refuse permission made by planning inspector Philip Wilson who heard three weeks of evidence at a public inquiry earlier this year.
Coin Street Community Builders boss Iain Tuckett says that the leisure centre, part-funded by the sale of 329 apartments in the high-rise building, will be the best in London.
"We are delighted that the Planning Inspector supported our advice that this scheme would cause serious damage to London's most perfect 18th century space, the magnificent courtyard of Somerset House, as well as the iconic view of Whitehall seen from the bridge over the lake in St James' Park," said English Heritage in a statement issued on Wednesday.
"We are therefore appalled to learn that the Secretary of State has not only overturned the advice of English Heritage as her expert advisors, but she has also chosen to ignore the learned opinion of an independent Inspector.
"English Heritage finds it incomprehensible that her reason for doing so was because she considered that community benefits outweigh harm to the historic environment as though one must be at the sacrifice of the other.
"There are alternative options that would have provided the same community benefits but would have been more sensitive, without causing serious damage to historic buildings spaces and views.
"Obviously we do not consider this matter closed and we are considering our next steps and the options open to us."
The decision by Hazel Blears can be challenged in certain circumstances by making an application to the High Court within the next six weeks.
For the latest local news and events direct to your inbox every Monday, you need our weekly email newsletter SE1 Direct.
7,000+ locals read it every week. Can you afford to miss out?
Read the latest issue before signing up