A survey commissioned by Nacro found 61% of people said it would be the worst thing to have to reveal. Having a mental health problem came second with 47% and bankruptcy came third with 44%.
Jackie Lowthian, director of policy at Nacro, said: "It’s not surprising that having a criminal record is a source of such shame. People who have a record are aware that their past will count against them in the job market. Yet the truth is that many people who commit an offence move on.
"Work is the most effective way of preventing offending. If we don’t provide the right help to these people, and the right advice to employers, we are throwing good people with valuable skills on the scrap heap."
Minor offences such as fines and cautions for shoplifting, theft, driving offences remain on your record as well as more serious crimes.
In response Nacro has launched its Change the Record Campaign to reform the Rehabilitation of Offender Act and will be presenting a report to MPs and peers today.
Nacro argues the Act is an outdated law that makes it incredibly hard for people with criminal convictions to get and job and move on from their past.
The report outlines how the current Act is out of tune with the Government’s rehabilitation revolution, its initiatives to get people off benefits and back to work and its Big Society plans to create a more equal society.
Paul McDowell, chief executive of Nacro, said: "Around a quarter of the adult population has a criminal record. Outdated legislation and unlawful practices are preventing people from moving on with their lives and finding work."
