Ask HN: What have you built more than twice and wish someone had built for you?

[andrezzoid]

For me, I'm building workspaces and sharing capabilities for the third time at a different company.

I would convince my boss to pay for a service that provided easy-to-use React components and implemented all the flows for inviting users to workspace, share by email or link, list who can access a resource, send invitation emails, etc, so I could focus on the damn app!

[boplicity]

A local dev environment that works.

Doing this again, and again, and again....

...truly sucks.

After not working on a codebase for a year, or two, I usually need to set up everything again -- and often run into very stupid/pointless issues that waste an hour, and often much more, just to get to the point where I can start the actual coding process.

This is pretty much why I've stopped coding. I don't have the time to waste any more.

[miloignis]

I think local reproducible dev environments is a great and less-intense way to get into Nix (the package manager)! https://devenv.sh/ is built on top of Nix as well to make it more approachable and user friendly for exactly your use case.

[lrvick]

I would note that while Nix is very well designed, and may be incredibly useful in untrusted dev environments, it should not be used to compile anything that touches production. Nix got where it is so quickly by mostly ignoring basic supply chain integrity like author package signing. It is always one compromised Github account or single dev workstation away from a massive supply chain attack.

Nix, NPM, Brew, Pip, etc all have basically the same blind trust security posture and should thus not be trusted. I generally suggest Debian in a container for a dual use dev/compile container made of signed/vetted/reproducibly-built dev/build/debug dependencies.

[jjj123]

As someone who knows nothing about package signing and security, are you saying that Nix is about as secure as NPM?

[lrvick]

Correct, and that is not a good thing.

[agent281]

I’ve been toying with the idea of using Nix for dev environments at work. What does devenv add on top of Nix?

Edit: And if you know, how does it differ from devbox?

[miloignis]

I haven't used devenv or devbox, I've taken the full Nix/NixOS plunge - sorry I can't be more helpful! I've just seen devenv on HN and glanced over its homepage and saw that it was based on Nix but supposed to be easier to use. If you want to jump straight to Nix itself, more power to you! There's a big learning curve, but I have found it very worth it.

[lrvick]

I ship a Dockerfile and Makefile with every single project that contains absolutely everything I need to build, test, and debug that codebase.

I cd to a project and run "make shell" and I am back in a docker container with every tool and the current working directory mounted in it with permissions all mapped 1/1, scripts in path, etc.

The only tools I need on a dev machine are an editor, a browser, a container runtime, and make.

[unwind]

At a previous employer, there was a basically full-time team member (contractor like me, but long-term presence so it didn't matter much) working with configuration management. It was basically his job to keep the development environment working, and kept reasonably up to date.

That was done using VM images that he owned and maintained, which was not optimal for me, but there was also documentation laying out the steps needed to create the environment so it was possible to maintain a bare-metal clone of the official environment and use that. Good times!

[MrDresden]

I have my whole Linux desktop machine, two laptops and a few servers fully codified now in Ansible.

Everything is in there. Every library and piece of software, every config and preference. User creation (minus any sensitive data). Everything.

Made a bootstrap script I run on a fresh install, and then I can push the ansible config onto it from a raspberrypi ansible server on my LAN.

Wanted a different file system in december. Wiped the system, changed filesystems, resetup. Took less than 1 hour.

[dloreto]

Tools like Devbox: https://github.com/jetpack-io/devbox are making this way easier. Devbox is powered by nix, but it hides all of the complexity and makes it easier to use.

[teekert]

We had a session here where someone explained how you can use a docker container with VS code to have reproducible local env (for the whole team).

Seem like you never got on board the declarative-infra-as-code boat, it's actually rekindling my software flame, with things like docker-compose and NixOS, everything you ever work on can be stored as code, and set up with 0 effort.

[plasticchris]

I think containers (docker/vagrant/etc) are probably what you are looking for.

[boplicity]

Just the other day I had to figure out an opaque problem with a vagrant image not working. It was very annoying, and wasted the entire time I had set aside for coding.

[lockyc]

Isn't this what Microsoft is trying to solve with dev containers? https://containers.dev/

[hyeomans]

Did this happen when you have more than 5 microservices each one with its own database?

[asim]

The programmable web. A singular gateway to a programmatic way in which you not only consume data but also services. It doesn't exist. We have a very fragmented ecosystem of http based services mixed with html, APIs and everything requires an account or credit card. I'd kill for one way to access everything in a programmatic way through one gateway. Tried to approach this in numerous ways and continually failed.

Edit: Last attempt https://M3O.com

[esquire_900]

You are quite right, it's something I've been considering for a long time as well. The cold start problem however is so large, that it has to be an open protocol supported by many of the biggest companies for it to have even a slim chance of succeeding. I admire your attempt, though I'd imagine it to be a very steep and long climb.

[asim]

I've come to the same conclusion. It only took me 8 years...but I'd hope it didn't rely on big tech to endorse it to work. I just don't know of another approach. Maybe publish a spec and see where it goes?

[dmak]

This is more likely to happen in crypto than any other ecosystem. The permissionless composabiblity is driving an insane amount of innovation.

[asim]

The issue with crypto is no one is really defining a protocol in the way we used to used to understand internet protocols. I'm all for blockchain and P2P but only where it makes sense. I've tried to understand how that would work here but it feels like it's going to be a natural and slow evolution.

[dmak]

And how is that? I see a lot of protocols that are being built on top of permissionlessly.

[Kinrany]

Decentralized standards for authn, authz and payments are a prerequisite.

[asim]

Yea you're right. Basically protocols or APIs for it.

[_huayra_]

A sane build system + package management solution for C++. I have to cobble it together via CMake + Conan every time and I'm praying I get put on a new Rust project someday, but my position and sizeable comp are mainly for being a "legacy janitor".

[dotnwat]

We've been a CMake shop from the beginning, but have recently been testing out Meson to see if it could work for us and we've been pleasantly surprised by how nice it is.

[PaulDavisThe1st]

It can't be done for C++ unless you massively constrain the problem.

Trivial example: libfftw3 has --enable-threads and --enable-float configure/compile time options. Your project may need/want both, one or neither of these options (it may, in fact, want two different versions of the library).

Rust "proposes" to deal with this by essentially compiling its libraries as a part of building your project (ie. "vendoring" every 3rd party library dependency).

Most C++ developers don't consider vendoring in this way to be particularly sane, even though many of us do it anyway.

[_huayra_]

I have tried to figure out how to use Nix as a package manager to deal with this, but it is still too much wizardry for me to wrap my head around.

[esprehn]

Have you tried gn or bazel?

[_huayra_]

I have advocated to at least try bazel (or even buck), but the higher-ranking devs all say that it's too much "organizational knowledge risk", i.e. if I force something unfamiliar onto the team and they only shakily know how to work with it with my help and then I leave, they're screwed. Therefore, it usually boils down to trusty old crusty tools.

Crabs in a bucket.

[Cthulhu_]

They're not wrong though; if you want to introduce a new tool - especially one as large and complicated as Bazel - you need to also take ownership of training and hiring not just yourself (assuming you're not already a Bazel expert) but all your current and future colleagues yourself. The management has to make a cost / benefit analysis and it's not in Bazel's favor.

Bazel is a tool by and for Google scale companies, which 99% of tech companies aren't. Make and co are good enough for most use cases.

[khaledh]

A SaaS for managing personal documents. The closest I have right now (not SaaS) is paperless-ng[0], but I have to self-host it, unless I missed a really compelling solution.

I have a sea of documents, both physical and electronic, and it's always a struggle to scan/organize/find them. I'd pay good money for a software/service that manages my documents, from scanning to archiving.

[0] https://github.com/jonaswinkler/paperless-ng

[teekert]

For anybody interested in paperless-ng, this is line 1 of their repo's readme:

This project is not dead! Rather, this project is now maintained as a community effort at https://github.com/paperless-ngx/paperless-ngx, a direct continuation of this project and drop-in replacement. Please see their project page and the documentation at https://docs.paperless-ngx.com/ for details.

I really like Paperless NGX, have been using it for some months now, mostly for scanned paper mail, I use [0]

[0]: https://apps.apple.com/in/app/scan4paperless/id1629964055

[dmak]

Are you saying you would pay a service to host it all for you? or are you saying you want to just send all they physical documents to someone to scan and upload for you?

[khaledh]

No, not to host paperless-ng for me. I also don't mean send them physical documents. I'll be scanning the documents myself of course. I mean a hosted service that I can use from my desktop or phone to upload documents quickly. It should also provide integration with my email with the ability to add rules regarding certain email senders and converting email content to PDF where possible (for the cases where the sender doesn't provide a PDF attachment already).

[gremlinsinc]

Can you list features? This sounds like a good/quick microsaas idea, maybe we can team up on it. Any idea if others would pay for something like this? Am I right thinking the features are:

- Ability to upload docs (perhaps w/ multiple integrations, drive, dropbox, etc. - Imap connection to only temporarily read/store email to check the sender or content (possibly using some ai as well?) to then convert specific mails to pdf, etc, and configure further rules on what to do w/ these upload somewhere, email them to someone, run through zapier? - Would the ability to transcribe or describe what's in messages be valuable? I've worked on something like that to basically tag emails for marketing messages when 90% of the emails were an image/ad of a competitor.

Any other features that could be fast/easy but make this reach a wider audience 'need'?

[dmak]

It would be better to continue to explore the problem than to begin solutioning.

[denysvitali]

My attempt a couple of years ago and never completed (Rust): https://github.com/DMSrs/dmsrs

[Traubenfuchs]

Deployment pipelines, CI/CD, (dev)ops. It's my biggest pet peeve ever that every company I have ever been to has written highly customised CI/CD/(dev)ops tooling that I had to fight with or adapt at some point. I don not understand why anything beyond heroku/dokku complexity is ever needed. The needs for teams to manage infrastructure is lunacy in this day and age.

[deckard1]

I really don't understand what has happened in the last few years in this space. At my company we went from stable but boring Jenkins to this Github/K8S hell. Everything is broke all the time. Literally every single day. It blows my mind. We are at the point that people have forgotten that things used to just work. It's like back in the day Windows98 users know you have to reboot the PC once in awhile. Because they know of nothing else. They don't know the past and they can't imagine a better, more stable future.

[Traubenfuchs]

I believe it‘s a conspiracy where sysadmins wanted to create their personal next generation job security.

Countless inane layers of overengineering no one can reason about anymore.

[nicolaslem]

Speaking of Dokku, I remember stumbling upon it a decade or so ago and dismissing it as a toy project that was unlikely to go anywhere.

Fast forward to the present, after noticing that the project is still going strong I did a deep dive in it and got really impressed. I think I will be moving my personal projects from the unmaintained Ansible mess I created to Dokku.

[Traubenfuchs]

With Dokku I can have a new dockerized webservice online behind https in <5 minutes. And I am confident I could set it up from scratch in <30 minutes.

All while not knowing a lot about devops at all.

[xakdog]

Yes! And some startups even set up k8s, manage db themselves, and spin up microservices. Even before having their first user!

So much hassle to support it when that shit breaks.

Also, to add to you list - Railway and Vercel are also super great

[ezekg]

A licensing system. That's actually why I ended up building my current business 7 years ago.

So you've found a problem -- now build a solution. :)

[dmak]

Could you elaborate? are you talking about something like MIT license? or something like 90s software where you get a key for an app?

[ezekg]

A license key system. Particularly one that also manages license entitlements and activations (e.g. what machines or nodes a license is allowed to be used on).

[probably_wrong]

My parent's computers.

In the ancient times known as "the 90s" you could find PC technicians who would run anti-malware, dig into the register, and overall tune your computer while leaving everything more-or-less as it was.

Those technicians are long gone, replaced by people who make easy money by wiping your computer clean and putting your old files in a folder called "Backup".

I'd gladly pay someone to do it right, but I just can't find anyone. So whenever I visit my parents I know there will be a parade of slow devices waiting for me to tune them.

[petiepooo]

You can blame the ever-increasing complexity of MS Windows for that. My parents are now on Linux, being that the only things they do on it, other than open the occasional email attachment, are all browser-based. It's easier for me to occasionally run an apt-get upgrade remotely than it is to try to keep Windows free of infestations.

[Cthulhu_]

I was one of those, but I was just a late teen going to school for it. TBF, in some cases a wipe was the better and faster option. Modern day PCs and laptops are (should be) set up so that user folders are on a separate partition, and a hidden partition to reinstall the operating system while leaving the user's files alone. Should be, anyway.

[cactacea]

Yup, and then rootkits happened. Better safe than sorry.

[h2odragon]

Postal Pre-sort routines. Many years ago, when the most reliable way to get the real walk lists was to go to the local postmasters.

There was fairly cheap commercial options even then but that shop had a bad "dont buy what you can build" attitude combined with a maddening shyness about making any other use out of the things they'd built. Having written our own presort software, we couldn't then sell it or even open source it because that would be "a burden distracting from the core mission of the business."

[PaulHoule]

It's a tricky area. There are strong reasons why systems like this tend to use dark patterns and it is your own call what you think is appropriate. Also making a system like this really work as opposed to pretend to work means effort put into email deliverability and things like that that other people can't really do for you.

I read this in 1999

https://philip.greenspun.com/panda/

and came to the conclusion that the basic need for a "web framework" was a system of authentication that did what most commercial sites do: let people create new self-service accounts with email verification and all of that. That was the essence of the tcl-based framework that Phil Greenspun was pushing but I didn't like tcl, so I wrote something in PHP that was meant to integrate with 'best-of-breed' PHP applications (install the authentication system, then modify various applications to use your authentication not then) as opposed to the "PHPNuke" approach which was popular in the industry which was "install some portal which had worst-of-breed implementations of most of the functionality you think you need".

What I found baffling was that nobody cared about authentication frameworks until they became something that worked "as a service" about 10 years later which is silly for so many reasons, not the least that a company that offers a service like that is going to either run out of money and shut down the service or get aqui-hired and shut down the service.

[simplify]

An open source lib where I can type up some text configuration, hook it up to a sql database, and get a nice looking generated form (for internal surveys in my case).

[dmak]

Airtable is the closest to what you are saying because it has data relationships. However, what did you find insufficient about Google Forms or similar?

[cabraca]

Check out https://lowdefy.com/ they even have a sample survey app as one of their examples.

[alanbernstein]

A photo deduplication tool. For some reason I always resort to writing small, scenario-specific python scripts. A couple of reasons I can recall:

- 3 or more input directories which have specific roles like "main archive, prioritize", "temp folder, remove from here first"

- multiple levels of equivalence test, including file name, exif tags, checksum, perhaps perceptual hash (e.g. for flagging downscaled images to be deleted)

[johnjeremy]

I found this library worked extremely well for finding and removing similar images https://github.com/elisemercury/Duplicate-Image-Finder Maybe this doesn't fulfill your main and temp folder requirements but worth a try.

[Diris]

I think this one[0] can do most of that.

[0] https://github.com/qarmin/czkawka

[ankit219]

An internal fix for attribution. I have worked with three companies - all three consumer - and they required a reliable attribution mechanism they could rely on to know which channel is required for acquisition of their users, and which one for conversion. Usually it's a very similar product too. We use Branch or Appsflyer api (or Facebook, Google Ads if on website) and then use the JSON generated to map it to our users (in a postgres table) and then run queries by joining that with our internal DB tables.

I usually value two things: one, who first made the user aware about the product (could be organic, FB, SEM, Organic, Instagram, reddit, hacker news, product hunt, anything) and then which channel was responsible for final conversion event we want. Most products in the market only care about last. First touch is important to figure out where to get more users from at a lower cost. In the products, they tend to overwrite that and just use last touch. So, you store the JSON at the time of generation in your own database, map it to userid, and use it for internal calculation.

[yardie]

Cheap offsite backups. I know AWS and BB are the bees knees. But when I read the terms they. Have so many escape clauses built in. My bosses already had a poor experience with our backup provider simply deleting our backup set over a missed payment. Although there are better services they are quite shy about doing it again.

[INTPenis]

But S3 compatible storage goes far beyond AWS and Backblaze these days, almost everyone offers an S3 compatible service.

So with that said, I prefer restic over borg for the S3 support.

[reidjs]

I just bought the Borg plan at rsync.net that may be what you’re looking for

It takes a day or two to set up though, if you are not super technical like myself

[ibejoeb]

Spreadsheet data ingestion. Someone did built it, but it was years after: https://flatfile.com/

Not a plug. I'm unaffiliated and just impressed by it. Should've thought of it myself.

[l-]

Regexstepper - takes an ordered list of regexs & repeatedly runs them against a corresponding ordered replacement list and ordered sample list until no further replacements are found. Used it to learn about stepping through PEMDAS simplification on various math iexpressions & attempts at combinatorial systems. Wrote it once years ago learning about PEMDAS then decided to delete it because JS cannot do at matching of arbitrary depth matching braces without parsing into an AST ala shunting yard, then learned about combinators & decided to try again to try and find systems that work without parentheses.

[mamcx]

Auth Code.

The only I have used before was django auth (great), but never get anything like that in other langs. I'm in Rust know and is my biggest desire...

[torte]

I kind of agree. Even now there are tons of third-parties dealing with the authentication for you, you will still have to write code to actually integrate with them and hook this up to whatever your user data is. I have written a few authentication layers now, and while I feel there are added benefits with third-parties, the amount of code I have to write is similar and strongly customised to the vendor offering the authentication solution.

Then there is also the problem you are facing and that is tooling not being language-agnostic across the board and you will have to re-implement similar patterns when changing the backend language or framework.

[pengo]

I wrote a web app that moved paper-based funding submission and grant management online. A second client wanted the same thing, so I wrote it again with newer tech. When the third client asked for the same thing I spent eighteen months creating a customisable framework, which is in use for increasing numbers of clients.

[dmak]

Is this basically upload, OCR, and storage?

[jjk166]

Every couple of months I wind up making a quick s3 static site which isn't too difficult but doing it manually there are several steps spread out across many pages and it's easy to forget one if it's been a while. Every time I do it I wish I had written a script to automate it the last time.

[nicolaslem]

Kind of related, I recently wondered why there is no such thing as a packaging format for static sites. It sounds trivial but hear me out.

There are many static site generators that spit out a directory with HTML files ready to be served. But getting that output in S3, netlify or your self hosted nginx is a very different process. Configuring things like cache-control, redirections, compression or error pages is done differently on all these platforms.

Wouldn't it be nice if it was possible to basically generate a tar.gz with some metadata that would automatically configure the web server and deploy the site? Kind of like a docker image but for static files?

[Cthulhu_]

Netlify is basically that, push to git and it'll auto deploy, along with serverless functions if you need something like a back-end. Github Pages works too for purely static sites. Cloudflare probably has a similar offering.

[alltheworlds]

A configuration management system. Actually I didn't build it, but have used different implementations at every job. Why can't this just be a commodity? It's not exactly doing anything very different in each case.

[lrvick]

Multi-party signed review tooling for git, CI, and secure enclave infrastructure. Built all of it several times. This time I found a client to let me open source it all though, so this is the last time I hope!

[INTPenis]

A password manager for teams of people that isn't dumb.

It's surprisingly hard to find. I contributed to one open source that a co-worker started, and I made my own in my own time. So I've almost made it twice.

[dmak]

What's wrong with 1Password or Bitwarden?

[INTPenis]

I haven't tried Bitwarden. A friend uses it and says it's fine for his personal use at least.

But when I say teams I mean it needs to be designed in a way that it can easily reflect a CMDB architecture, with a tree of objects, API and LDAP auth. Technical details aside, the goal is of course to integrate it with other systems.

On top of that, not all UIs are simple enough to be used by the entire organization either, but as long as it separates backend and frontend that can always be fixed.

Honestly my current plan, and this would be the 3rd time I build something like this, is to make a simplified frontend for Hashicorp Vault.

[kinggencha]

Not sure how easy self-hosting those are, but any paid-for and/or online offering by a third party is useless. This should have been clear long ago, but the LastPass breach should have made it obvious.

[dmak]

To each it's own, but 1Password security design looks pretty solid. Here is their white paper: https://1passwordstatic.com/files/security/1password-white-p...

[INTPenis]

No system is perfect. You're making a trade off by self-hosting but at least when something goes wrong you know who to blame and what to improve. When something goes wrong in someone else's environment you're lucky to even know what went wrong, and you have no one to hold accountable.

So it's not IF something goes wrong, it's WHEN something goes wrong. Going around thinking IF something goes wrong is delusional, even if you end up being lucky and right.

[slingnow]

You're right. I'm sure their security design is 100% bulletproof and none of my sensitive data will ever be leaked. And even if it were somehow possible for it to be leaked, I'm also sure the company would be completely forthcoming as quickly as possible.

[pwinnski]

A system which literally never sends my password information to any computers controlled by 1Password seems better than the LastPass nonsense, and more to the point, seems at least as secure as anything I would create. While LastPass was breached many times over the last few years, 1Password has never been breached.

Your sarcasm isn't helpful, and serves only to falsely conflate the insecure design and horrible history of LastPass with the best-in-class 1Password. Nothing is ever 100%, but 1Password is closer than anything else I know about, including most one-off systems used by people who don't use password managers.

[dmak]

It'll be better than rolling your own password manager and self-hosting it for your team.

[seanw444]

Self-hosting Bitwarden with the open source Rust re-implementation of the server (vaultwarden) seems like a good option. I've been looking into setting it up for my company.

[sofixa]

Or the official open source server, which has the potential to get support from Bitwarden if it gets to it. It's much heavier though, with SQL Server and similar, but vastly more enterprisey.

[cmcconomy]

I need bitwarden to let me log into my personal and work accounts simultaneously.

[steveruiz_sb]

Revenue infrastructure. For the amount of time and engineering resources it takes to build and maintain, it's just easier to spend 10% of it. Buy $10K > Build $120K

[francisofascii]

A developer laptop with all the required software installed, environment variables set, ready to go.

[domenkozar]

Have you tried https://devenv.sh?

[gt565k]

that varies per developer preference and per development environment, hence why it can't be generalized

it's custom to every user and every development environment

It can probably be configured by someone in your company as a docker image you can run, but usually that's only necessary for large teams where IT is provisioning development laptops and there's a lot of consistency and common dependencies, even then I don't see it being worth the effort as those preferences would still vary across team roles and projects.

[zabzonk]

making makefiles work correctly - happened every place i've worked

[lonelycompiler]

Installing matlab parallel server