Up to eight British soldiers are believed to have been captured as they escorted the diplomatic mission through rebel-held territory in the east of the country.
According to the Sunday Times, the elite SAS troops were taken by rebels to Benghazi, the largest city held by the opposition, where they were detained.
The mission is thought to have been an initial attempt to contact Gadafy’s opponents ahead of a visit by a senior colleague to establish diplomatic relations.
But the SAS intervention apparently angered the rebels.
The situation came to light as the battle for control of the country continued to rage and fears grew over the impact of instability in the region on oil prices.
Fierce fighting was under way in the city of Zawiya, just 30 miles from the capital Tripoli, as rebels struggled to repel an assault by Gadafy forces.
Meanwhile, insurgents were reported to have taken control of the oil port of Ras Lanuf after a series of bloody engagements, a key strategic site.
British defence secretary Liam Fox today confirmed that a British diplomatic team is in Libya talking to rebel forces.
But he declined to comment on reports that the SAS unit guarding the team had been detained by forces opposed to Gadafy.
Interviewed on the BBC's Andrew Marr Show, Dr Fox said there was a "small diplomatic team" in the eastern city of Benghazi.
“We are in touch with them but it would be inappropriate for me to comment further on that for reasons I am sure you will understand,” he added.
Asked if the UK team was in danger, Dr Fox replied: “We are in touch with them but I’m not going to be giving further comment on that.”
He added: “It is a very difficult situation to be able to understand in detail. There are a number of different opposition groups to Colonel Gadafy in Libya who do seem relatively disparate.
“We want to clearly understand what the dynamic is here because we want to be able to work with them to ensure the demise of the Gadafyregime, to see a transition to greater stability in Libya and ultimately to more representative government.
“So getting a picture of that is relatively difficult as is widely reported. Communications are being interrupted, there are difficulties with mobile phones, with the internet potentially being interfered with.
“So we are trying to build a picture, it’s essential that the [British] government does that and it’s essential that all Western governments do that so we are able to get a clearer idea of what we are able to do in terms of helping the people of Libya.”
Dr Fox indicated that a no-fly zone was still on the cards, saying the prospect would be discussed at a meeting of Nato defence ministers later this week.
“The prime minister has been making it very clear that it would not be acceptable to see the air force used to slaughter the people of Libya. That is why the prime minister was very keen that we did the contingency planning for a no-fly zone,” Dr Fox said.
“We’ve been discussing with officials at Nato exactly the sort of scoping we would like to get done.”
He said there the Nato meeting at the end of this week would “have to make some very difficult decisions”.
“For example, if we were to decide on a no fly zone, there would be a number of options. Would it be the whole of Libya, would it just be the population centres that we focused on?” he added.