Self-employed scaffolder Russell Samuel was working on the Meridian Quay development in 2008.
Carillion Construction was lead contractor on the project and Samuel worked for sub-contractor Febrey.
Both firms admitted health and safety breaches at Swansea Crown Court. Carillion was fined £130,000 and Febrey £85.
The Health and Safety Executive (HSE) told the court father-of-two Samuel was working on a nine-storey block of flats, part of the Meridian Quay development that includes Wales' tallest building, the 29-storey 'Tower'.
Samuel, 40, suffered multiple injuries, including a fractured skull. He was taken to Swansea's Morriston Hospital, but died two days later on 24 January 2008.
The HSE said Wolverhampton-based Carillion admitted breaching sections two and three of the Health and Safety at Work Act 1974.
The company was fined a total of £130,000 and ordered to pay £52,500 in costs.
Bristol-based Febrey, which has gone into liquidation, also admitted the same breaches and was fined £85. The judge said he would have fined it £250,000 before it became insolvent.
HSE inspector Anne-Marie Orrells said: "There were recurrent indicators that should have alerted Carillion to Febrey's persistent and systematic failures throughout the whole project.
"Yet Carillion failed to adequately address Febrey's significant failings. As the principal contractor on site, Carillion had a clear duty to plan, manage and monitor the construction work."
She said falls from height were still the biggest killer in the construction industry.
