[weberc2]

We're struggling using containers for our Python and Javascript applications. I think we're using venvs and node_modules and we oughtn't (or we shouldn't keep them in our project directory where they sometimes--but not always--get overwritten by source code volume mounts).

[TheDong]

It can be a little tricky to do things right here.

There are a few things of note:

1. The .dockerignore file can be used to prevent use of your `node_modules` folder during `docker build`, even if you have something like `COPY . .` in your Dockerfile.. This can let you create a new `node_modules` folder from your lockfile as part of the docker build process to create an image for testing/deployment.

2. You can maintain separate Dockerfiles and such for development and for release, so e.g. for development you might use volumes and not copy things in, but for release you wouldn't use volumes for source code.

I can't tell from what you said, but it seems like one of those two tips might be relevant.

It's okay to have a development setup which doesn't use containers and then use containers for deployment (so long as you have an integ or staging environment that uses containers) as well. It's quite reasonable to have a venv during development, but inside the docker image to not use a venv at all since things will already be reasonably isolated inside the container's fs.

[weberc2]

Thanks for the suggestions! I had to check, but we do both of those things. To make sure that node_modules aren't overwritten, we mount an empty named volume to the node_modules directory. I guess mounting the named volume directory somehow causes the node_modules directory from the base image to appear inside of the bind-mounted directory. We do the same thing for venvs as well.

We use venvs inside of the Docker container because we use pipenv and apparently pipenv's support for installing to the system is buggy and/or idiosyncratic.

[TheDong]

Named volumes are hacky to no end and you shouldn't be using them to make sure it's empty; you should be using `.dockerignore` or being careful about what you copy in.

The pipenv bit is probably more reasonable, and I'm afraid I haven't used it enough to be sure what rough edges are likely there.

[devxpy]

I've been using shiv recently for my django deploys.

You might wanna check it out.

Here is a small readme I put together...

https://github.com/devxpy/shiv/blob/8d8298d21380dcf0b1970856...

[dharmab]

For Python, this works well for my team: https://www.dharmab.com/programming/python/docker/2017/07/24...

[weberc2]

Sorry if I'm a little dense; that doesn't seem like it solves any problems. It just says "use virtualenv and/or docker". I guess I was hoping for "how to manage dependencies in a localdev-friendly way for a Python/Docker app" or something.

[VonGallifrey]

Not sure if that helps, but here is what I do:

We just use a requirements.txt [1] for each service and run pip with the -r flag in the dockerfile. No Virtualenv in the container.

Most of the time we run these containers with docker-compose locally, but sometimes we want to run the service outside the container. For that we create a virtualenv outside the folder where we keep the source for the service. The reason for that is that we don't want to accidental copy the virtualenv in to the container. It wouldn't do much there but it would increase the container size.

You can do this easily with just "python3 -m venv", but there are some tools that help with that as well. I personally use pyenv-virtualenv [2] which just keeps all virtualenvs in ~/.pyenv/versions/<env>, but there is also conda [3] and virtualenvwrapper [4] which also store the virtualenvs in a central directory.

I am not sure if there really is any more to it.

[1]: https://pip.pypa.io/en/stable/user_guide/#requirements-files

[2]: https://github.com/pyenv/pyenv-virtualenv

[3]: https://conda.io/docs/user-guide/getting-started.html#managi...

[4]: https://virtualenvwrapper.readthedocs.io/en/latest/

[dsnuh]

I'm not sure if I understand correctly. Are you saying that you run venv commands as part of the build step for your containers (i.e. a RUN line in you Dockerfile) and then pip installing modules in another build step?

[weberc2]

No, we use pipenv instead of pip, and pipenv manages a venv. There are `--system` flags, but I guess support for installing things to the system is a little buggy or idiosyncratic or something so we use the venv behavior. I'm not sure why we tell pipenv to install the venv in the project directory though. We do the pipenv install as a RUN line in our Dockerfile. Does this answer your question?

[dsnuh]

Yeah, I think I have an understanding now. We use tagged Docker base images with the app dependencies baked in so you can just use a FROM your Dockerfile and know you have the dependencies in the image and you just add and run your app code.

[weberc2]

Yeah, we do the same thing. This seems to work pretty well, except right now we build these images out of band of our normal build process (if you change these base images, you're expected to kick off another CI job that builds/pushes new versions and updates the versions in the FROM statements in our production Dockerfiles (and our docker-compose.ymls). I'm not sure why we're doing this, since the build cache is actually pretty good at avoiding unnecessary base image builds.

[dharmab]

Plop a requirements.txt file down in each repo. Use direnv or a similar tool to automatically manage virtualenvs on the local dev machines. In your container, don't use virtualenv; just add requirements.txt to the container and run pip install -r requirements.txt

See the linked code at the bottom of the page

[dev_dull]

There are so many places where this will never work. For example, if you're using openssl and you're system is missing the header files.

[dharmab]

Which is why I run and debug locally with an interactive container; installing the packages locally is just for my editor autocompletion.

[dec0dedab0de]

Have you tried pipenv? It creates a lock file that hashes all your dependencies so it makes sure you get the right one when you deploy.. And it automatically creates the virtualenv if its nog there when you install them.

[weberc2]

We do use pipenv. I like the idea of it, but I think we ran into issues with where the pipenv lives. We used to use `--system`, but that seemed buggy and/or idiosyncratic, so we started using a venv and then we started using PIPENV_VENV_IN_PROJECT=true (for some reason) and now it's causing other issues. It seems unclear how it's supposed to be used with Docker containers, but maybe we're just overcomplicating things?