The Intimate Life Of Simon Cowell, written by former BBC journalist Tom Bower, reveals details of a brief relationship between the two X-Factor judges back in 2008.
Simon, 52, is quoted as saying he felt "genuine love" for Dannii, 40.
"I had a crush on her," reads a passage in the book. "It was Dannii's hair, the sexy clothes and the t--s. I was like a schoolboy. She was foxy. She was a real man's girl. Very feminine."
Rather than deny he made the quotes, Simon has responded on Twitter, "Just seen the TV ad for the unauthorised book on me. Yikes!"
Rumours of a relationship began in 2008 after the pair were snapped holding hands in the back of a car (she claimed it was just a trick of the camera) and Dannii jokingly accused Simon of staring at her butt ("I probably was," he admitted at the time).
The book describes how Simon considered Dannii his new "toy" when she joined the show in 2007. She sat next to him at the judge's table until Cheryl Cole joined the panel the following year and Dannii got shafted to the other end signalling that the relationship had ended.
According to the biography, Simon also had a brief "kiss and cuddle" on board his yacht with another Aussie beauty Natalie Imbruglia.
Wide World of Sports
Latest celebrity news
Fat Tony & Co
The Big Bang Theory
2 Broke Girls
Two and a Half Men
House & Garden