The current zoom tool is a scroll wheel zoom, not a box selection zoom. Guessing that might be the issue since others have run into the ambiguous description as well. It will be labeled more precisely in master in a few days and in the next release as well.
If that is not the issue, please file a ticket on GitHub!
Do you have the tool selected in the toolbar above? If so, please file a ticket on GH about this. Most of the Bokeh devs are on OSX so it would surprise me if it does not work on OSX, but if there is a bug we want to fix it!
If that is not the issue, please file a ticket on GitHub!