[gumboza]

The more I hear this stuff the more I write things in Go with no external dependencies pulled in. I can do 95% of what I need to do without involving a supply chain or downloading anything random off the internet other than the go distribution itself.

[shanebellone]

Anti-dependency mafia, rise up. I feel the same way. I code almost everything from scratch too.

[isoprophlex]

I like the sentiment and I'm usually first in line to ridicule the 'npm install left-pad' crowd, but this doesn't always fly. Python is a great glue language to mash high performance C/fortran components together. One does not simply write sklearn or pytorch from scratch.

[shanebellone]

"Python is a great glue language to mash high performance C"

This is exactly what I'm starting to work through. After 6 years of Python, I've finally hit the limit of what I can do with it. Now I'm working to rebuild an algorithm in C to reconnect to the Python application.

"One does not simply write sklearn or pytorch from scratch."

I also agree with this. Would either be in a product though? Personally, if it's not a product, I wouldn't mind dependencies.

[anakaine]

Yes, they are in at least one product I can think of, and likely more. That product deploys its own conda environment and includes a huge amount of spatial analytical tools. Governments and large private enterprise the world over use ArcGIS Pro, as do many NGOs and education institutions, which is a massive leap forward for both desktop and highly integrated Web GIS work.

I'd be prepared to be a bit of blind money that other industry tools use a similar setup where the python libraries permit an exceptional cadence of development and help place those vendors products at the pointy end of the market.

How they manage dependency security isn't super clear. They're always a couple of version behind, so perhaps it's a CI/CD QA/QC thing which also includes security.

[mschuster91]

I get the general idea, but at the same time, I don't have the time to write my own libraries from scratch - all modern web standards are complex and most libraries filled with years to decades worth of experience of all the edge cases that crop up, particularly as most standards don't carry a "compliance test suite".

It's one thing if I were paid by my employer to re-invent the wheel, but for personal projects... I don't have that much free time for them in the first place any more, I want to get shit done and not shave yaks all day. When I want a good grind, I'll pack out Factorio or one of the LEGO Switch games...

[shanebellone]

There's a difference in values between those who reinvent the wheel and those who leverage opensource. It sounds like you value time-to-product whereas I value ownership of said product.

There are always risks associated with building on other people's land, platforms, and codebases. However, there are also risks when reinventing the wheel. Both perspectives have advantages, disadvantages, and use cases.

[bigDinosaur]

A compromise is to audit and then pin exact versions, or even copy and paste the code into your project. Yes, this is a clear tradeoff in that you'll lose access to newer updates, but it's certainly worth thinking about. I do it with relatively trivial libraries for things that I know the package has solved various edge cases, is small in scope, and probably won't be updated again, for example.

[shanebellone]

I agree with you, but I'd prefer to reinvent the wheel rather than audit an existing code base.

[intelVISA]

It's reassuring isn't it? Every time something breaks you have easy access to the mfer who wrote it.

[shanebellone]

Exactly!

We're also talking about layers of dependencies. It's a ridiculous approach.

[pjmlp]

I always build my whole computer from scratch from NAND gates all the way up to the full OS, build my own switches, cut the network cables myself, dependencies be dammed. /s

[cuteboy19]

For python at least, most of the dependencies are very justifiable. The python stdlib is very huge and satisfies most regular programs such as glue code. But for web and ML it is not possible to include these libraries in stdlib nor is it feasible to write it from scratch

[shanebellone]

It's not difficult to write most of it from scratch. It just takes some time and attention.

[cuteboy19]

It's not possible. Even very basic numpy would be too slow to use, if you end up writing pure python equivalents.

If you import numpy might as well import the entire scipy ecosystem

[shanebellone]

It's absolutely possible. My only dependency is Flask and I'll be eliminating that in time too.

Why do you need numpy for web?

Edit: I will concede that there is no point in retooling ML. Web is an entirely different circumstance though.

[anakaine]

Let's say you are writing an API that works with some particular scientific file types on the back end, and you want to load that data into memory for fast querying and returns. Now, that data is a multidimensional time series for each file. You could spend the next months writing libraries and bashing your head against the wall, or you could leverage the 30+ years of development in that stack that enables you to read these.

Xarray to read, numba for calcs in xarray, pandas to leave it sitting in a dataframe, numpy as pandas preferred math provider. You could write the api componentry from there, sure. Or you could use a library that has had the pants tested off it and covered most of the bugs you are likely to accidentally create along the way.

There's no compelling reason to write everything from scratch. If everyone was taking that approach then there would be no reason to have an ecosystem of libraries, and development would grind to a halt because we, as a collective of people programming, are not being efficient.

[shanebellone]

I see no compelling reason to implement a multidimensional time series for multiple files as a component of any backend API that consumes user (defined) data.

In what circumstance could that be profitable? Even if you batched data, any number of concurrent users would gobble resources at an incredible rate.

[deredede]

Flask seems to be a very stable and feature-complete framework (I see about 3 commits per year for the last few years).

At this point isn't it easier and just as safe to manually review the code, pin the hash in a lockfile, and manually review the rare changes than it is to rewrite everything?

[shanebellone]

Definitely. There's nothing wrong with using Flask. It's actually quite pragmatic.

In my case, replacing Flask is purely preference.

[shanebellone]

Can someone explain why this comment is getting downvoted? I believe the statement is accurate. I'm not looking to justify or debate my position, but a clear answer might help me better approach this topic in the future.

[cmeacham98]

Your viewpoint, to be frank, is extremely naive and plain wrong.

Link me your reimplementations of tensorflow, numpy, and django (with similar features and same or better performance) and we can talk.

[shanebellone]

1 comment beneath.

"I will concede that there is no point in retooling ML. Web is an entirely different circumstance though."

Edit: I just realized the way I use votes isn't necessarily the same and no one is wrong in their understanding.

Your reply connected the dots. Thank you.

[throwaway_3850]

Standards and requirements will change, bits will rot, and im not expecting any ecosystem, to keep up with comming and going demands.

A better solution imho would be project level capabilities, so you can pull in a dependency but restrict its lib/syscall access, so it would not compile when it turns malicious.

Maybe it will solve at least something, maybe some day.

[gumboza]

Agree. I'd like to see an OpenBSD pledge(2) type system for libraries. So you can mask individual library capabilities rather than just programs. I don't want a web server that can write to the file system and I don't want a CSV reader that can talk to the network.

[saagarjha]

Doing this kind of thing at the library level is generally not very useful, because security protections between things running in the same process are hard to make very strong.

[LadyCailin]

This is a limitation of the particular language/ecosystem though, it feasible in a new language that has this security baked in to the language primitives.

[PurpleRamen]

I don't think the Go-stdlib is significant better than the Python-batteries. For normal stuff, you can build without dependencies in python too. The problem starts when you use more complex stuff, or want to save time by using a lib delivering certain benefits. After all, you can't build and maintain everything by yourself.

[gumboza]

Shipping Go is a hell of a lot easier.