[teddyh]

There seems to be a problem parsing backslash escapes:

echo 'foo="\\"' | PYTHONPATH=/tmp/yapf/yapf python /tmp/yapf

Raises a lib2to3.pgen2.parse.ParseError.

Also crashes on \n, but not, strangely, on \t.

I’m guessing that the backslashes get interpreted twice, somehow.

[eliben]

Are you sure this is not a shell artifact? Does the same problem happen when there's a \\ in a file? In any case, feel free to open a Github issue for yapf, we'll take a look

[teddyh]

Yes, it was when parsing a file I encountered it. The one-liner shell command is just a minimal repro of the bug.

Apparently I need to “Sign in” to report a Github issue, so I won’t.

[eliben]

Sorry, I can't reproduce this. I'm trying on this file: https://gist.github.com/eliben/9727758f4847d2e7d86e

And yapf runs just fine on it. Does this file work for you?

[teddyh]

Nope.

  Traceback (most recent call last):
    File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
      "__main__", fname, loader, pkg_name)
    File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
      exec code in run_globals
    File "/tmp/yapf/yapf/__main__.py", line 18, in <module>
      sys.exit(yapf.main(sys.argv))
    File "/tmp/yapf/yapf/__init__.py", line 104, in main
      verify=args.verify))
    File "/tmp/yapf/yapf/yapflib/yapf_api.py", line 99, in FormatCode
      tree = pytree_utils.ParseCodeToTree(unformatted_source.rstrip() + '\n')
    File "/tmp/yapf/yapf/yapflib/pytree_utils.py", line 100, in ParseCodeToTree
      tree = parser_driver.parse_string(code, debug=False)
    File "/usr/lib/python2.7/lib2to3/pgen2/driver.py", line 106, in parse_string
      return self.parse_tokens(tokens, debug)
    File "/usr/lib/python2.7/lib2to3/pgen2/driver.py", line 71, in parse_tokens
      if p.addtoken(type, value, (prefix, start)):
    File "/usr/lib/python2.7/lib2to3/pgen2/parse.py", line 116, in addtoken
      ilabel = self.classify(type, value, context)
    File "/usr/lib/python2.7/lib2to3/pgen2/parse.py", line 172, in classify
      raise ParseError("bad token", type, value, context)
  lib2to3.pgen2.parse.ParseError: bad token: type=55, value=u' ', context=('', (1, 5))

[eliben]

Interesting. Can you cite the exact version of Python 2.7 you're using?

Mine is 2.7.6

I'm asking because bugs get fixed in lib2to3 occasionally, and we see differences between minor Python versions in this regard.

[teddyh]

Python 2.7.3, the normal one from the current Debian stable.