Ask HN: What're the craziest hacks you've done to launch?

[msukmanowsky]

Lots of us have had to do some pretty crazy stuff to launch a minimum viable product but I think a lot of the stuff I've done personally was pretty mild (hardcoding things into an interface, running manual processes, etc.).

I want to hear some of the fun and crazy hacks. No need to mention the company or product to protect the innocent :).

[pcdoodle]

Young and hungry to impress a chain of restaurants:

Pandora just launched. I was installing a CCTV system at the first restaurant location. Wanted to show the owner something cool, launched pandora on the PC I just installed for him.

"That's awesome, can we hear it in the restaurant?".

The amp was about 50' from the office, I had pre made camera cables with BNC and found enough adapters to make it to the AMP.

Now he was REALLY impressed.

He then asked if he could make announcements over the restaurant speakers from the PC. Sure as shit, I'm going all the way.

Fire up VB6 and search google for "Volume Mixer VB6". Made a little tray app to mute the wave out and unmute the Mic. It works!

Then he asks: "We want to make the announcement from the guest check out counter". To get this done, I took some phone cable, soldered 2 wires to a paperclip i broke apart, then jammed the paper clip ends into the LPT port and searched google "VB6 LPT READ". Found a nice one liner that returned a different value when the wires were shorted and then adjusted the volume mixer for the announcement.

I didn't have a push button switch for remote control but had an extra IBM PS2 mouse, so I soldered the phone cord directly to the microswitch on the left mouse button clicker and jammed a sheetrock screw to mount the mouse directly to the wall. Now he can press the button and talk to all his customers.

This was crap I had sitting in my 1990 Buick Rivera. I think I was charging $40.00/hr. back then.

[risson]

You sir, are a legend.

[high_byte]

what's good $40/hr if you get the job and 5 others in one hour? :)

[0x706B]

Had to launch our AI product next day else customer would jump off and this would've basically killed the company. But AI was far from perfect and did only work for very small amount of usecases. Exchanged the AI with a queue and UI so humans could do the task instead in realtime. Worked. Went live. Completly smashed competitors.

--

Just at launchday our internal infrastructure was fucked pretty bad and we faced a severe outage on everything non-productive. Product failed at some place to do proper error handling and basically got stuck forever since it couldn't reach some monitoring endpoint we had still placed on our int infra. We couldn't build a fix, since basically CI/Signing was part of the outage. ETA to get infra back up was something like ~8h. Panic grew as it would've totally fucked our go live. Hacked together a dummy http server returning status code 200 and exchanged DNS entry to point to that instead of int infra. Worked. Went live.

--

Had to do an audit before we could go live, due to deadlines and planned vacation of auditor it needed to be spot on, no time for touch ups and re-audit. During the audit, he asked for a thing we forgot to do (some security alerting on a 3rd party tool which didnt support it), told him something along "sure, love to show that, yadda yadda, lets have lunch first." went to the most chatty guy in the office to join us for lunch (so he wastes some time), faked a "oh, shit, forgot i have a sync call, will catch you up there". While they were having lunch, I checked out the 3rd party code, added the alerting, wasted 20 mins with their broken buildsys. Managed to build the one lib necessary for the alerts, but not the whole image. Overrode the lib in the original img instead and hoped shit would work. They were back too early, couldnt test, just fired up the deployment, wasted some more time babbling with the auditor till it went through. We live tested it together. Worked. Audit went through without any remarks and we hit launchday.

[leros]

Ah yes, the classic AI startup with a team full of humans actually pulling levers behind the scenes.

[0x706B]

Fun fact: biggest learning was that this is perfectly fine. It's a $task startup solving $task for you.

$task means pulling 100 levers to the right level. Every lever has different precision requirements.

20 levers you can pull good enough with simple if/else and call it a day 20 more need a bit of statistics voodoo to reach the precision requirements 20 more can be pulled with AI to the approx. right position right away. 30 more could be pulled by AI in the future after going live when there's enough data on where the levers need to be pulled to. And the last 10 levers need such high precision that humans need to be involved (maybe for the moment, maybe for ever)

As long as solving $task is profitable enough and latency & error rate by humans low enough, there is nothing wrong with it.

Some competitors tried it with pure AI, but didn't hit the precision requirements and we could take over their customers. Others burned through all their funding trying to get the AI right before going live.

[leros]

Oh yeah, nothing wrong with it at all. Plus, you need the human team to be creating the training data set.

[akoncius]

third one is almost like in james bond movies when you barely make it but you succeed etc :D

[comprev]

I have a story of a hack from my time in the streaming audio industry.

I worked for a company who opened up their API to audio hardware manufacturers to allow integration with our streaming service. One particular company had a rigorous test: Query 20 songs via our API (one song per query - random songs selected so caching playlists would not work), retrieve and parse the metadata to display info on their hardware within 23 seconds.

The test had to use the same public API endpoint as our own customers (via our music app).

Their test lab was in Los Angeles. Our data and API endpoint was in Dublin. Using the public internet they managed to load 13 songs at most. I was tasked with reducing the latency.

To pass the test I deployed a temporary read-only copy of the app in a data centre in LA and used Route53's geolocation routing functionality to route the testers requests transparently to the local app. The DNS was live for 24hrs.

We passed the tests despite not reflecting any real world conditions and the hardware company promoted integration with our platform for years afterwards.

[xrd]

Working for a company that had one of the USA national league sports organizations as a customer.

They were super litigious. And the contract had metrics we had to hit with our services or they wouldn't pay us.

I was building the service. The customer was load testing it.

My manager had me on the phone in his lap while he was in the office with them. He had another back channel to the server guy who was monitoring the load.

He was whispering "Defcon 1" if the server started creaking and I needed to switch to a perl script that used cached data. If he said "Defcon 2" I was supposed to switch to sending a static file. Can't remember if we got to Defcon 2 or not, I think we did.

It was pretty nuts to deal with that customer. We passed the load test.

[Waterluvian]

What was the reason for doing that vs. shipping something that met specifications? Were they testing in bad faith (ie trying to DDOS it to not pay)? Was the contract poorly specified so you couldn’t test it ahead of time with double the load? Etc.

[tester756]

>Were they testing in bad faith (ie trying to DDOS it to not pay)?

Basing on "USA national league sports organization" it feels like they were reasonable

[xrd]

I'm not sure what you are saying here. Are you saying because they are a national sports organization they are probably honorable? I definitely didn't feel that way after working with them. They had no idea about how technology worked (granted, this was 20 years ago so they have probably fixed that deficit internally).

They did have a clear understanding of how to use litigation, and they played to their strengths, which I can respect.

[tester756]

I meant that I think that they're operating at really big scale (big events?), so their requirements and testing them feels reasonable

[xrd]

In my opinion, yes, they were trying to DDOS it and find a flaw and then renegotiate.

I saw this kind of behavior a lot those few years. A major university did the same.

In the case of the university, they weren't fulfilling on their end by providing us with raw video materials in time to process them. Then our software had bugs processing the files in time. We had to do it all within a few weeks. Who's fault is that, our bugs or their personnel issues.

Usually lawyers then come in to fight it out. And they will comb over those contracts to find one item that was not fulfilled and use that to the hilt.

And, lots of times, you sign a contract and THEN you figure out how to fulfill upon it. If you haven't done that, you've never really run a business, don't fool yourself.

[blitz_skull]

> If you haven’t done that, you’ve never really run a business, don’t fool yourself.

Whoa, okay Business gate-keeper. I didn’t realize you held the keys for bestowing “True Business Owner” status on us.

Some of us prefer to run our business in a sensible way that doesn’t include legal fees for signing contracts we know we can’t reasonably fulfill.

[sneak]

Does simple fraud now count as a hack? The bar should be higher than this.

[xrd]

Not a bad question.

They were super litigious and apparently often refused to pay once the slightest detail was missed in a contract. I wasn't in a position where I negotiated those things but it was conveyed as such in meetings I was a party to.

This was twenty years ago and the world of these contracts was different in that no one really knew how to structure them and inevitably all the parties involved were disappointed.

[Zircom]

Doesn't really count as a product launch, but my freshmen year of college in my first ever programming class we had to present an app we designed to the rest of the class.

I'd designed a small little casino game that had a few different games(roulette, poker, etc), and at some point in time a couple of nights before it was due I somehow managed to fuck something up and when it was displaying the results, the string somehow had the number repeated over and over and over with it running off the screen. Like "The wheel stopped on 66666666666666666666666". I had no idea what the fuck I had done and for the life of me couldn't chase the bug down, not that I put much effort into it because I didn't really have the time given the other 4 finals I had to be studying for.

So I changed the string generating function slightly to append the result + 200 spaces instead of just the result, so the repeating digits past the first one would all just get pushed off the edge of the screen and not be visible.

Got a 100% and a fun little email exchange with my professor when he saw the print statement, he thanked me for the laugh and wished me well hoping this would be the first of many last minute hacks in my career.

[mdwalters]

to be honest, that's a rather funny bug indeed

[SeanAnderson]

Mmm, best I've got is when the big G was mad at my Chrome Extension for not showing YouTube visuals inside of the pop-up UI. It's not possible to host the entire <video> inside of the extension's visible UI because the page is destroyed when closed which kills the audio. There is an invisible, permanent background page which can host the <video>, but then there's no way to show the visuals.

They told me they'd pull the extension if I didn't find a way to show the UI and that I had ~1 day to ship a new version. So, I found a way. :) Managed to read through the minified YT source and split the video from the audio, pipe the video to a separate page, and run the two in-sync. This meant that destroying the foreground page didn't pause the audio.

They didn't really care for my technical solution and it didn't change anything long-term, but I felt super badass in the moment as I launched what I thought was a clever, technical solution to resolve their concern with little notice.

https://medium.com/@MeoMix/beautifully-buffered-bytes-ff798e... I wrote about it here.

[mattpallissard]

Went to implement memory/CPU limits in the HPC batch scheduler we used. It turns out it had a fatal flaw, it counted cached memory as used. So servers would slowly become idle as cache was filled.

I piggy backed some code on the job validation interface to attach the users requests to the job's environment variables. Then I wrote a daemon to run on the compute nodes that walked the jobs process group/tree, grabbed the environment variables[1], and managed cgroups.

Super quick and dirty but it worked well enough that we kept it in place for years despite the fact the bug had been long addressed.

[1]: it was was safe to use environment variables as they were stored RO on disk by the batch system itself.

[greazy]

What schedular? Super interesting hack

[mattpallissard]

Univa Grid Engine, formerly Sun Grid Engine.

[seanhunter]

I had been working for a company in the early dotcom boom for about 2 weeks when they did a small public offering on an off-market exchange. Just before the investor roadshow we discovered a bug in the system (not the one I was hired to develop, their existing product that they thought was good to GTM) whereby if two people tried to access the website at once the database would become corrupted and the CGI process (yes it was written as a perl cgi) would just spin consuming CPU and not return and the only way to fix it was to kill the the process, dump and reload the db and then restart the process again.[1] So during the roadshow I was tailing the logs and keeping an eye on the website and if the problem occurred I would text my colleague who was doing the actual pitch “keep talking” if they needed to suddenly stop the demo while I did the dump and reload in the background and “good now” if they could carry on. Roadshow was an entire day of me doing this every couple of minutes (probably 8 hours or so but it felt much longer).

Offering was successful and I immediately ported the code to mysql straight afterwards which fixed the problem.

[1] Database was msql (no mysql, msql). I have no idea why somebody thought that was a good idea.

[TigerTeamX]

Reminded me of game developer who explained an null pointer exception. There was a function that kept getting a null passed and that broke the whole system. They searched for months trying to figure out why, and they couldn't. so at the end they just did a

  if (foo == null) {
     return
  }

worked like a charm.

[bitwize]

That reminds me of how they added a trap on the hardware invalid memory access exception to Sonic 3D Blast on Genesis/Mega Drive to take you to a "secret stage select" screen to comply with Sega's requirement that the game never crash. If you jiggle the cartridge in the console, you will likely be taken to this screen.

[TigerTeamX]

I never ended up there even though I played a lot of Sonic, but interesting. Reminds me a bit of how when something goes wrong on some of the tech websites it shows a cute animal. Okay at least they made sure I see a cute cat, somehow makes it nicer.

[Waterluvian]

Not sure if already well-known or maybe actually not true but I had a few ex RIM engineers tell me that the BlackBerry radio stack would reboot itself all the time. Like 100+ time a day. Because apparently fixing the problem was not an option.

[Our_Benefactors]

I had to integrate two conflicting style frameworks.

The architecture of the app had a different look and feel for the public/admin sides, but it was rolled into a single ball of mud.

The development pace of the team was glacial, because they had to restyle 50% of the framework defaults.

I dug through the code and realized that the reason some components looked so terrible was that they were built with different em basis. So I forked one framework, and wrote a script to find any lines terminating with ‘em’ or ‘rem’, extract the value, and multiply it by a const and then rewrite it in pixels.

This worked almost perfectly except for a few collisions with some other uncommon css property name. It allowed us to keep using those frameworks without needing to restyle things over and over again Which saved a ton of time in getting early pages shipped.

[mikewarot]

Long ago I was doing software, but on the other side of the room they were discussing a process step in making halogen lightbulbs, and the problem was that the bulb needed to be "seasoned" but if you didn't do things just right, it would sag and blow through the envelope.

I turned and said "why don't you just levitate the filament?"

The next day I had a DC power supply, some magnets and a long thin stretch of wire, and demoed how Lorenz forces work, moving the wire up and down at will.

A week or so later, I had a hacked up transformer for an electromagnet and my Pioneer stereo supplying the AC power to heat the filament, and the DC supply to levitate it. It worked until my stereo popped.

We rebuilt it with real industrial equipment and it runs to this day as far as I know.

[jskulski]

A company I worked at was in the business of classified ads in the early two thousands.

“We” started selling typeset ads at some point to customers unbeknownst to the tiny fresh grad three person tech team.

So we did what we had to do which was to run open office headless on a Linux box under our bosses desk.

It would accept type setting parameters, and we would simulate one key press at a time. We would send the print command keyboard shortcut and keys to print to a pdf and retrieve the files with rsync.

We didn’t know any better and I’m still proud of that.

[mablopoule]

Not really 'product launch', since it's about delivering a school project for my engineering school a long time ago.

I don't remember the subject, but I had to implement an image processing algorithm of some kind, it was all Matlab, and there was a particular function that just wouldn't work. I would read the same lines over and over again, but still not figure out where was the issue. The deadline to send the report with the code was at this point just a few hours away.

So I asked a friend in my class for his report, tried with his implementation of that particular function, and everything ran smoothly, which allowed me to write and generate the report.

Once the report was done, with the proper data, graphs and comments, I still couldn't figure out where was the issue with my implementation, but I also couldn't send my friend's implementation. The deadline approached fast.

Hence, the hack: since we needed to send the project code as a zip file, and this specific function was in a separate Matlab file, I had the last minute idea of opening the .zip archive with a hex editor, search for that file name, and edit the data following it with random values.

And just as I hoped, if you tried to extract the archive, everything but this file would be extracted, with a warning message about a corrupted file.

That way, I was able to send my projects basically 5 minutes before the deadline, while having a excuse to fix my function later if the teacher complained about the corrupted file (which didn't happen in the end).

It think it's the only time I cheated for an exam or a project, and, while I abhor cheating, I'm still damn proud of that hack.

[gargsms]

Back in my college days, we used to have YouTube blocked during the day by the IT department. This was a time when YouTube used flash for the main player. I figured out that YT embeds still worked because they used the HTML5 player instead. So I coded up a small extension which switched to HTML5 player all the time. Solved a real problem for tons of people. :)

[themodelplumber]

This reminded me of a story.

A friend of mine was once contacted to build an app for a respectable non-profit with a low-ish budget.

The developer friend thought "this app as described would be WAY better as a website with some app-like features, not an app-store app. Tons of people will want it as a website but not as an app...it'll just sit in the app store unused. Give people a way to find the pages of the app via third-party search engine and it'll probably be amazing." So he planned to start building it in a hybrid style, just in case.

The proposal to build an app went forward, and was accepted.

The non-profit then requested some significant new, scope-changing features during the project discussions.

The developer said, "those new features will cost extra, do you want an estimate?" and gave them an estimate.

The client said they didn't have the budget but still wanted the features. Possible?

So the developer said "we could include those features, and really beautifully so, if you decided to make it a website. But then it wouldn't be an 'app' in the traditional sense anymore, with the manifest and all that offline stuff, to say nothing of app-store presence, because of budget reasons X, Y, and Z. The best you could call it is a web app."

The client immediately said "it doesn't matter, a web app is fine as long as we let people know to download important data from the web version before they go offline! Really that's our only hurdle here!"

The developer said "WTF, OK" but made a note of the discussion, and built a lovely new website and it worked great. The website contained warnings--grab this data if you go offline, and so on.

People loved the web experience, it was one-of-a-kind and got lots of compliments.

Still, later on, organizational stakeholders tried to hold the non-profit responsible for calling it an "app"...meanwhile it was more like a website?! What is this, some kind of a scam? Show me where it is in the app store?

The NPO staff pushed back on the developer friend. <-- This was really not cool--a red flag for sure.

He said, "ah, but you remember our discussion...? You wanted extra features but didn't have the budget. So I gave you a choice and YOU made the choice to no longer build an app in that sense. And by the way--it's still technically an app...oh and also your organic search presence is so much better with this thing exposed to search engines..."

So: The app, that wasn't. But was? But also wasn't. He never expected the term "app" to require so much hacking.

[Our_Benefactors]

This sounds like a nightmare client.

[zaphirplane]

> organizational stakeholders tried to hold the non-profit responsible

What is an organisation stakeholder and why is the non profit as the owner beholden to then

[devoutsalsa]

I used to work as a QA Engineer at Transmeta, a low power semiconductor CPU company, and Linus Torvalds also worked there developing mobile Linux. Our first product launch was a Sony Picturebook featuring a Transmeta processor. The only problem was it wouldn’t boot Linux, and so when Linus said “hey, this won’t boot Linux” shortly before it went on sale, everyone (dramatic embellishment) was freaking out.

While everyone was scrambling to figure out the problem, I went to Fry’s Electronics. Fry’s had the Intel version of the Sony laptop for sale, and it did some QA work on their display model. I confirmed that the Intel version also didn’t boot Linux, and that seemed to be good enough for everyone one to launch our product, which was optimized for Windows ME.

The Linux problem turned out to be a BIOS issue, but I don’t remember what it was exactly.

[satvikpendem]

Reminds me of Tinder, the founders went to sororities and convinced them to sign up, then went to the fraternities and said, hey, there are a bunch of girls from sororities on the app, why don't you try it out? It worked exactly how you'd expect it to.

[iancmceachern]

One time we had to deliver 5 giant custom ultra bright digital signage enclosures (those news stands in times square) to NYC on Monday to avoid paying a large late fee Claus in our contract. We were done with the big boxes, but not the $20k custom monitors I designed. So we hired a "hot shot" semi truck to pick up the big enclosures on Thursday in order to get them to NYC by Friday. We then air freight-ed the monitors the next day to get them there in time, buying us about 12 hours to finish the displays, for $20k in air freight.