For books, clean, dry, bare hands is recommended by the Library of Congress, the British Library, and Library of Canada. They explicitly recommend against using gloves.
Interesting summary, though the argument is limited to old worn books that are still physically sturdy and they explicitly disclaim the recommendation for art and photographs.