.line-number { user-select: none; -moz-user-select: none; -webkit-user-select: none; }
...should be sufficient to prevent that bit from being selected without JS.