Men are eligible to take up to 26 weeks leave when a new baby is born. However, if they do so they receive only the statutory rate of £136-a-week, which is rarely topped up by employers.
The TUC said the main reason for the low take-up was that most fathers could not afford to take a significant drop in pay.
The decision to keep statutory pay rates below inflation and employers' refusal to top-up pay beyond two weeks is preventing new dads from spending time with their babies, said the TUC.
Although nine in ten (91%) fathers take some time off following the birth of their child, fewer than one in three (29%) spend longer than two weeks at home.
TUC general secretary, Frances O'Grady, said: "A good gift for fathers [after father's day], would be for ministers to increase statutory paternity pay rates and for employers to top it up for longer, so new dads can spend more time with their children.
"Poor levels of financial support are preventing new dads from taking extra time off and are particularly affecting low-paid fathers who simply cannot afford to take leave."
She added: "Extending paternity pay from two to six weeks and paying a better statutory rate would make a massive difference, as has been shown in other countries."
A spokesperson for the department for business, innovation and skills, said: "The current system for parental leave is old-fashioned and too rigid. This is why we are introducing a system of shared parental leave from April 2015 so that fathers can take more leave if they want to in the early days of a child's life.
"We want to challenge the myth that it is the mother's role to stay at home and care for children."