[closeparen]

Be deliberate and skeptical about how and why your rule definition language is a better fit for the problem space than a general purpose programming language. When you find yourself recreating a general purpose programming language, stop. Just drop down to one. Or start with one. A very successful rules engine at my employer is Python minus features, as opposed to the typical "config plus features until it becomes a shitty Python."

Realize that what you are doing is a programming language, and create as much of the infrastructure for programming as you can for rule authors (version control, code review, unit testing, incremental deploy, etc).

[zaphirplane]

What about security and safety. A user provided rule in a DSL has controller access to the rest of the environment. While a user provided script in a language will have a lot of security and safety issues, even if you trust the user there is security in depth and safety of limiting the accidental damage

[gdy]

Not sure about Python, but it is very easy to embed Lua in an app in way that executed scripts have access only to what is deliberately exposed to them.

[adrianN]

Sandboxing the code is a solved problem, is it not? There are a number of websites that run code for you somehow.

[FabHK]

It's a surprisingly tricky problem, btw, at least for some languages. Here's a nice 2014 talk by Jessica McKellar: Building and breaking a Python sandbox that gives insight into some pitfalls. Might be "solved" by now though, don't know.

https://www.youtube.com/watch?v=sL_syMmRkoU

[zaphirplane]

Running stand alone and throw away code in a container, is very different from running a user provided script within your long lived application securely. Think credentials, Db access, file system access, network

But you want to access the DB and write to files and the network just not anywhere, so you have different process and communicate via rpc

[jlj]

I prefer to use SQL because it handles the set definition very cleanly. It is basically the "where" clause of a query. And it's a good middle ground where programmers and business analysts can speak the same language to describe a dataset.

Once the set is defined, the programming language of choice can be used for the action. It could also be SQL, or since we use Spark for a lot of our compute it could be Scala, Python, or Java.

I've been in recent discussions about building a new DSL for this, but I haven't been convinced why we need a new DSL when SQL is already a widely supported DSL.