[cuteboy19]

For python at least, most of the dependencies are very justifiable. The python stdlib is very huge and satisfies most regular programs such as glue code. But for web and ML it is not possible to include these libraries in stdlib nor is it feasible to write it from scratch

[shanebellone]

It's not difficult to write most of it from scratch. It just takes some time and attention.

[cuteboy19]

It's not possible. Even very basic numpy would be too slow to use, if you end up writing pure python equivalents.

If you import numpy might as well import the entire scipy ecosystem

[shanebellone]

It's absolutely possible. My only dependency is Flask and I'll be eliminating that in time too.

Why do you need numpy for web?

Edit: I will concede that there is no point in retooling ML. Web is an entirely different circumstance though.

[anakaine]

Let's say you are writing an API that works with some particular scientific file types on the back end, and you want to load that data into memory for fast querying and returns. Now, that data is a multidimensional time series for each file. You could spend the next months writing libraries and bashing your head against the wall, or you could leverage the 30+ years of development in that stack that enables you to read these.

Xarray to read, numba for calcs in xarray, pandas to leave it sitting in a dataframe, numpy as pandas preferred math provider. You could write the api componentry from there, sure. Or you could use a library that has had the pants tested off it and covered most of the bugs you are likely to accidentally create along the way.

There's no compelling reason to write everything from scratch. If everyone was taking that approach then there would be no reason to have an ecosystem of libraries, and development would grind to a halt because we, as a collective of people programming, are not being efficient.

[shanebellone]

I see no compelling reason to implement a multidimensional time series for multiple files as a component of any backend API that consumes user (defined) data.

In what circumstance could that be profitable? Even if you batched data, any number of concurrent users would gobble resources at an incredible rate.

[anakaine]

Who said anything about profit? Not everything that exists to be solved, and for which their is a demand is driven by profit. Think: regulation, environmental, NGO, citizen science, academia, government agency, public service. All places where systems can exist that are not for profit, but do grant significant capabilities to their user base.

Also, it's a particularly arrogant point of view to assume that because you cannot see a reason for something to exist that its development is invalid both now and into the future. You've also assumed the data is user defined.

I can also guarantee you that user concurrency is not an issue after some recent load testing, with load capabilities surpassing expected user requests by several orders of magnitude whilst on minimum hardware.

[deredede]

Flask seems to be a very stable and feature-complete framework (I see about 3 commits per year for the last few years).

At this point isn't it easier and just as safe to manually review the code, pin the hash in a lockfile, and manually review the rare changes than it is to rewrite everything?

[shanebellone]

Definitely. There's nothing wrong with using Flask. It's actually quite pragmatic.

In my case, replacing Flask is purely preference.

[shanebellone]

Can someone explain why this comment is getting downvoted? I believe the statement is accurate. I'm not looking to justify or debate my position, but a clear answer might help me better approach this topic in the future.

[cmeacham98]

Your viewpoint, to be frank, is extremely naive and plain wrong.

Link me your reimplementations of tensorflow, numpy, and django (with similar features and same or better performance) and we can talk.

[shanebellone]

1 comment beneath.

"I will concede that there is no point in retooling ML. Web is an entirely different circumstance though."

Edit: I just realized the way I use votes isn't necessarily the same and no one is wrong in their understanding.

Your reply connected the dots. Thank you.