That's right. Make connections to databases (and most other things) the first time your Lambda handler runs, and stash them in a static/global variable for re-use on future runs. That allows you to amortize the cost of forming the connection over many executions of your function, which improves latency, reduces cost, and reduces load on the backend.
TLDR; you can define the connection to DB outside of the scope of a given function, so it’s scoped to the container and can be reused so long as the container is not recycled. Seems promising!