If the opposing side brought up refusing to take a polygraph you would object and ask the judge to explain to the jury why that had to be disregarded, right?
I doubt that any judge would allow that to even be brought up in a trial. If it were, it would probably be an appealable issue that would get the case thrown out.