Partly its that. Partly its about tricking juries, "He failed/refused the polygraph" is a very nice way to discredit someone. At least 5 of the 12 people there will find that concerning and not know its BS.
Any lawyers in the house? I was under the impression that (in the US) refusing to take a polygraph was not considered an admission of guilt though could be grounds for dismissal from federal employment.
If the opposing side brought up refusing to take a polygraph you would object and ask the judge to explain to the jury why that had to be disregarded, right?
I doubt that any judge would allow that to even be brought up in a trial. If it were, it would probably be an appealable issue that would get the case thrown out.
It's used in criminal courts 19 states and at the federal level by the disgression of the judge. Plus it's used in a bunch of "non-court but legal process" like for people on probation.