According to their research, a team at Boston University Medical College found that women who take the pill regularly have much lower levels of the hormone that drives sexual desire, and even more worrying, that hormone remains suppressed even when they come off the pill.
They are calling on GPs to warn women of the potential danger before prescribing the pill and they criticise the medical profession for handing out oral contraceptives "like candy".
But the Royal College of General Practitioners has stepped in to reassure women that the pill is safe and to discourage them from shunning the pill in light of the findings.
In the study Claudia Panzer and Irwin Goldstein tested 124 women who were being treated for sexual dysfunction, and in that group half used the pill regularly, 39 had just come off the pill and 23 had never used oral contraceptives.
Blood samples from all the women were analysed for traces of a substance called sex hormone binding globulin (SHBG). The pill encourages the over-production by body of SHBG, which mops up testosterone, the hormone that drives sexual desire.
The blood tests results showed that women who regularly used the pill had very low levels of testosterone, but four times as much SHBG than women who had never been on the pill.
Further blood samples from the women who had come off the pill revealed that four months later, levels of SHBG had dropped but were still nearly double that found in women who had never taken oral contraceptives.
Dr Panzer says it concerns them that the levels of SHBG show no sign of dropping any further in those who came off the pill, the expectation was that a return to normal levels would be seen after about six weeks. That these women will always have more is a cause for concern because it means they will have very low testosterone, which has implications for their sexual function.
The researchers fear that levels of SHBG, which is produced by the liver, might be permanently raised in women who go on the pill, regardless of whether they later stop.
Dr Panzer says because of the easy availability of birth control pills the situation is 'quite frightening', as no one is told what the pill might do for a woman's sexual function.
The pill is the most common contraceptive used in Britain, with 26% of women between the ages of 16 and 49 using it.
Dr Mayur Lakhani, chairman of the Royal College of General Practitioners, says loss of libido is a recognised side effect of the contraceptive pill but in the experience of GPs and practice nurses this is uncommon among most women.
He says women should be reassured about the safety and efficacy of the contraceptive pill and he stresses that there is no need for them to stop taking the pill as a result of this study.
Dr Lakhani added that anyone worried should discuss their concerns with their practice nurse or GP.
The study, which is reported in New Scientist, was presented at the American Association of Clinical Endocrinologists in Washington DC last week.