Ask HN: Coming back to Python after a 5 year hiatus–-what do I need to know?

[alexkehayias]

The last serious Python I wrote was 2.7 (I narrowly avoided the python3 migration fun) about 5 or 6 years ago. I’m picking it back up, but I’m having trouble sifting through the current state of the art.

- Which typing library do folks use?

- Is Django still the best?

- What do you use for servers? Gunicorn?

- Any improvements with the interpreter to make it more reloadable?

- Is Celery still a thing or do you just use Lambda?

- What’s ideal for deploying Python? Ec2 and some Ansible? App engine? Something else?

- How about writing APIs? GraphQL?

- How about parallelism? Multi processing and futures for concurrency?

- What’s the async story? What should I be looking at for async web frameworks?

- Numpy still great? Sci py?

Thank you in advance!

[Townley]

Disclaimer that basically everything below is a personal opinion:

- MyPy is a great (fine, decent) library for typing, but even without it, the type hints in newer python versions and typing standard library are super powerful. Not “rust” levels of powerful but at least on par with typescript

- Django is still excellent and now supports async views via asgi. Outside of async, django+DRF+djoser are an excellent choice for a backend. If you really like typing and cool, smaller frameworks, Starlette and the derivative FastAPI frameworks are a lot of fun. Any of those options are great for a web app backend these days

- Gunicorn is still easier to configure. UWSGI is also still around. Both are well supported - Django run server has gotten better about reloads, but it’s still not as good as node’s offerings

- asyncio has taken a bite out of celery (heh) but celery is still common and useful for deferred tasks

- I think dockerizing the app is the way to go these days. There’s just too many odd system dependencies and python compilation differences to deploy directly to an EC2 instance. Digital Ocean’s app platform or AWS Amplify (or EBS) make deploying a container image decently easy. And if youre willing to take the plunge into Kubernetes, I find it has many python-specific advantages (eg horizontal scaling to avoid pYtHOn DOesNt sCaLE” nonsense) at the cost of a tremendous learning curve. If you do end up going for a non-containerized deployment, infrastructure as code is big now, so both terraform and ansible might be your friends

- graphQL support is bad. I think Graphene (Django) is the best you’ll find. DRF being decent is one of the reasons I generally stick to REST if I can help it

- Asyncio has come a long way but still has issues with non-async codes (eg it’s hard to make a sync request or access a DB in an async block)

- Tornado was the initial async bad boy and is still around, but again FastAPI is doing some really cool stuff in async

- For data science, pandas is pretty universal but it plays very nicely with numpy. If you haven’t kept up with Tensorflow or PyTorch, you’ll be astounded by how far ML has come

[habitue]

> but at least on par with typescript

I wish this were true, but mypy is far behind typescript. For example, mypy has `Literal["foo"]`, but typescript has the ability to template literals like `"foo-${bar}"` etc, and actually understand the results statically.

That's not to say mypy isn't great, it has some really good stuff, but I mostly think you're underselling Typescript here.

[Townley]

Huh, good to know. Does this issue apply to f-strings or are you talking about format methods? I did a brief look through issues and MyPy seems to support f-strings fine, but I’d 100% believe that there are still kinks to work out

[habitue]

Ah, so mypy supports fstrings, but not in types. So you can do

    Direction = Literal["left", "right"]

But this won't work:

    eft = "eft"
    ight = "ight"
    Direction = Literal[f"l{eft}", f"r{ight}"]

(Yes, this is super contrived, I bad at coming up with examples, but https://mariusschulz.com/blog/string-literal-types-in-typesc... shows some realistic examples where it's useful in typescript)

[alexkehayias]

Thanks for the heads up about graphQL—I think I'll avoid for now. It will be for a new project so not a big deal.

I just had a quick look at PyTorch... you're totally right this is miles ahead of what I remember.

I forgot to ask about package management. Do you use pip, conda, poetry?

[Townley]

Personally I love poetry, if for no reason other than making publishing a breeze. If I can’t use it for some reason, I like managing virtual envs with pyenv (not to be confused with pipenv. You’re better off for having missed the controversies around that)

Oh btw virtualenv is gone now; you make envs using python -m venv myEnvName

[alexkehayias]

Nice, I’ll check out poetry.

That’s great that some of the virtualenv functionality made its way into the standard library!

[habitue]

Opinions:

1) mypy is the best type checker, you could look into pylance or pyre though

2) Django's ORM is kinda bad vs. sqlalchemy, but this isn't new. I'd consider any framework where you can use sqlalchemy (most everything)

3) No improvements to interpreter really. In python3 you now have to import reload though :(

4) Celery still exists, it's inside of Airflow. In general "scheduling asynchronous jobs" is a thing that is ubiquitous and celery is a bit limiting with its queue model. Many people use things like kubernetes to let a more advanced scheduler find space & time to run your async jobs, vs forcing them to be evaluated by a fixed set of workers. That being said, celery is fine for simple stuff.

5) Deploying python: everything is docker + kubernetes these days. There are the serverless true believers out there, so AWS lambda might be up your alley if you like that sort of thing. Definitely wouldn't invest time in building on app engine, and I'd just use ec2 as nodes for your docker deploys. Immutable deployment artifacts are great

6) APIs: use REST/json, it's fine. Everything speaks it. If you have a big team where front end and back end can't work in lockstep, or you have multiple clients with diverse needs, consider GraphQL, but if you're doing a web frontend... might not bother? GraphiQL is really nice, but if you generate OpenAPI definitions the tooling for REST is acceptable.

7) Parallelism: still use multiprocessing. GIL is alive and kicking, no change there. I'd say more parallelism is going into horizontally scaling containers vs. trying to run a lot of processesin a single server. Have nginx (or your favorite kubernetes ingress) multiplex your containers

8) I've ignored async mostly, so take this with a grain of salt: you probably don't need async. Tornado is a good bet if you want async though for things like websockets etc. You can do sync or async with tornado, which is nice.

9) Numpy and scipy still alive and kicking. Not sure what your use case is, but the projects are very much active. There are obviously a lot of deep learning libraries etc in python now, so depending on what your needs are there is a lot of statistics / ML you can do in python.

[alexkehayias]

Thank you! re: sqlalchemy, are there more options for python micro web frameworks like Flask? I enjoyed flask + sqlalchemy in the past.

[sharmi]

You will love fastapi. It also works with sqlalchemy.

[tucif]

You might benefit from looking through recent conference talks, they are nicely organized in https://pyvideo.org/

Search topics you want to find out and see what’s being presented in the years you’ve been out of the loop.

[alexkehayias]

Thanks, didn’t know about py video. I will take a look!

[antman]

Don’t worry python is not js, you’re good. See docker, fastapi but depends on what you want to do.

[alexkehayias]

:-) Thanks for pointing out FastAPI!

You're right, far less churn than when I came back to js after leaving off at backbone/coffeescript/grunt...

[rustacean69_420]

It's immoral to not use Rust in 2021. Python trying to add types is proof.