New York-based escort Joey Anderson reportedly took a polygraph test before telling the National Enquirer the raunchy details of his supposed three-hour encounter with Joe.
"We'd take breathers and then start up all over again," he said.
"He seemed to have an insatiable appetite for sex and never really appeared tired."
The escort, who says he has text message proof of their affair, told the tabloid he didn't think he was Joe's first gay experience.
"Whatever the case, I'm probably not the only one," he said.
"After being a family man, and a religious one to boot, for all these years, I'm sure he's out sowing his wild oats with a bunch of boys.
"Joe has lived a lifetime of deceit regarding his sexuality. Before the dust settles, I wouldn't be surprised to hear about a lot more guys coming forward."
Joe separated from Tina Simpson, his wife of 34 years, last week, but in a joint statement the pair insisted there was no third party involved in their separation.
Speculation that Joe had "come out" to his family kicked off when 21-year-old Chandler Bryce Hall alleged he was Joe's former lover.