Any lawyers in the house? I was under the impression that (in the US) refusing to take a polygraph was not considered an admission of guilt though could be grounds for dismissal from federal employment.
If the opposing side brought up refusing to take a polygraph you would object and ask the judge to explain to the jury why that had to be disregarded, right?
I doubt that any judge would allow that to even be brought up in a trial. If it were, it would probably be an appealable issue that would get the case thrown out.