[micksmix]

One of the main benefits of Semgrep is its unified DSL that works across all supported languages. In contrast, using the Go module "smacker/go-tree-sitter" can expose you to differences in s-expression outputs due to variations and changes in independent grammars.

I've seen grammars that are part of "smacker/go-tree-sitter" change their syntax between versions, which can lead to broken S-expressions. Semgrep solves that with their DSL, because it's also an abstraction away from those kind of grammar changes.

I'm a bit concerned that tree-sitter s-expressions can become "write-only" and rely on the reader to also understand the grammar for which they've been generated.

For example, here's a semgrep rule for detecting a Jinja2 environment with autoescaping disabled:

  rules:
  - id: incorrect-autoescape-disabled
    patterns:
      - pattern: jinja2.Environment(... , autoescape=$VAL, ...)
      - pattern-not: jinja2.Environment(... , autoescape=True, ...)
      - pattern-not: jinja2.Environment(... , autoescape=jinja2.select_autoescape(...), ...)
      - focus-metavariable: $VAL

  

Now, compare it to the corresponding tree-sitter S-expression (generated by o3-mini-high):

  (
    call
      function: (attribute
                  object: (identifier) @module (#eq? @module "jinja2")
                  attribute: (identifier) @func (#eq? @func "Environment"))
      arguments: (argument_list
                    (_)*
                    (keyword_argument
                      name: (identifier) @key (#eq? @key "autoescape")
                      value: (_) @val
                        (#not-match @val "^True$")
                        (#not-match @val "^jinja2\\.select_autoescape\\("))
                    (_)*)
  ) @incorrect_autoescape

People can disagree, but I'm not sure that tree-sitter S-expressions as an upgrade over a DSL. I'm hoping I'm proven wrong ;-)

[codelion]

That's a really interesting breakdown of the DSL vs. S-expression approach. I can see your point about the potential fragility of relying directly on tree-sitter outputs, especially with grammar drift. It took me a while to wrap my head around the S-expression syntax when I first started using tree-sitter, so I appreciate the comparison to a more human-readable DSL like Semgrep's.

The other benefit of a DSL like Semgrep's is that LLMs have become very good at generating it. See https://github.com/lambdasec/autogrep on how to automatically generate Semgrep rules from existing CVEs.

[sanketsaurav]

> One of the main benefits of Semgrep is its unified DSL that works across all supported languages.

> People can disagree, but I'm not sure that tree-sitter S-expressions as an upgrade over a DSL.

100% agree — a DSL is a better user experience for sure. But this is a deliberate choice we made of not inventing a new DSL and using tree-sitter natively. We've directly addressed this and agree that the S-expressions are gnarly; but we're optimizing for a scenario that you wouldn't need to write this by hand anyway.

It's a trade-off. We don't want to spend time inventing a DSL and port every language's idiosyncrasies to that DSL — we'd rather improve our runtime and add support for things that other tools don't support, or support only on a paid tier (like cross-file analysis — which you can do on Globstar today).

[micksmix]

That makes a lot of sense. I wish you the best of luck and will be happy to try it out as you continue to develop it!