Show HN: Calibre Test Profiles – Test your sites under real world conditions

[benschwarz]

Hi HN! I'm Ben, the solo founder of Calibre Analytics: https://calibreapp.com. Calibre monitors websites and applications in a clean-room testing environment from all over the world. Up until January Calibre has been an "inbetween other things" / evenings / holidays kind of project. (Only, oops—I did that for four years.). Last year I knuckled down to get into the position where I could work on it full time.

I've spent the last 8 weeks working on "Test profiles" (The launch post is here: https://medium.com/get-calibre/announcing-test-profiles-and-...) — It allows you to emulate hardware devices (cpu speed, user-agents and screen dimensions), as well as limit the bandwidth (latency, download and upload speeds), and set cookies (you could use this to log into your app, test a staff-shipped feature, or maybe turn off advertising and see the performance impact.) I think I've finally got the basis of a really powerful tool, and I'm keen to share it with… everyone! I do everything myself, so if you've any questions, I'm here to answer them!

[benschwarz]

Hi HN! I'm Ben, the solo founder of Calibre Analytics: https://calibreapp.com. Calibre monitors websites and applications in a clean-room testing environment from all over the world. Up until January Calibre has been an "inbetween other things" / evenings / holidays kind of project. (Only, oops—I did that for four years.).

Last year I knuckled down to get into the position where I could work on it full time. I've spent the last 8 weeks working on "Test profiles" (The launch post is here: https://medium.com/get-calibre/announcing-test-profiles-and-...) — It allows you to emulate hardware devices (cpu speed, user-agents and screen dimensions), as well as limit the bandwidth (latency, download and upload speeds), and set cookies (you could use this to log into your app, test a staff-shipped feature, or maybe turn off advertising and see the performance impact.)

I think I've finally got the basis of a really powerful tool, and I'm keen to share it with… everyone! I do everything myself, so if you've any questions, I'm here to answer them!

[mikerg87]

I have been using http://www.webpagetest.org/ for similar investigations. csn you tell me what you see as advantages of your tool over webpagetest.org ?

Btw congrats. This looks very well done.

[benschwarz]

Web page test is a succhhh a great tool, and its really great for one off tests. Calibre will automatically hit your site daily (or, when you ping the snapshot api), and you'll have a full history of what happened and when.

With Calibre the aim is to make it a part of your toolchain (read: integrations) — so it should be visible to your entire team, or build system, or slack channel.

[palerdot]

Nice product. Can you share your tech stack? Do you sub contract some of the development like frontend or you do everything yourself?

[benschwarz]

Sure! Here goes:

The App itself is boring old rails, with postgres (get me started on how much I like postgres and I will literally bore you to tears.) The original views were all haml and sass delivered via the asset pipeline. I sprinkled in angular here or there as the application grew.

Today, I'm slowly tearing away the angular. I've moved to webpack for JavaScript, and React for all new UIs. I've been really enjoying both the speed of development, and how solid everything feels.

Over time, I'm writing less and less ActiveRecord, and more and more sequel… still learning, but getting better.

The app is run on Heroku, and its been A++ would recommend to everyone.

The Calibre agent is node and websockets talking to Chrome®, theres other bits of ruby, and even some golang now.

I use terraform and ansible (although, the ansible may as well have been bash scripts) through a buildkite (https://buildkite.com — the best fucking CI service on the planet) pipeline. AMIs are built, shipped to different regions, then based on some config are rolled out.

Papertrail for logs - omg, logging! Essential. I run thousands of tests per day (I don't know this number off the top of my head, but its a lot)… and the nature of the internet is THERE IS ALWAYS FAILURE. Having logging at my fingertips has always made my job building Calibre easier. I think I pay $20 per month right now, and its the most important thing I pay for.

--

Ok. Whew. Could probably keep going, but I'm not sure how much you wanna know. Happy to answer specific things though!

I do everything. No contractors. Although @fox has been recently helping me with my writing (her native language is Polish, and she writes waayyy better than me - and most others).

[pspeter3]

Is it only clean room metrics or can you also collect metrics from actual users?

[benschwarz]

Atm, the focus is clean room, but real time/user metrics are on the horizon—especially when the company is more than just me!

[techwizrd]

I totally mistook this for Calibre[0], the open-source e-book app.

The site looks really good. This is a very impressive product for an "in-between other things"/evenings/holidays kind of project. My only issue is that the animation of the application window coming up can sometimes be laggy.

0: https://calibre-ebook.com/

[benschwarz]

Yeah… that animation isn't what it should be for a performance product—luckily, after this (and the lighthouse integration I'm working on right now), I'll have all-new marketing too!

Edit: Thank you! :-)

[thomasfromcdnjs]

Calibre seems to be one of the only website performance testing tools that is keeping up with the new "user centric" website performance stuff coming out of Google aka Lighthouse.

I quite enjoyed watching this brain storm session with Paul Irish around Lighthouse and new metrics for measuring website performance in a meaningful way.

Edit: Forgot to include the link -> https://www.youtube.com/watch?v=IxXGMesq_8s

[packetized]

Test profiles is an especially interesting idea, something that I don't believe other tools in this space (Gomez/Compuware APN, ThousandEyes) have quite yet. Other than test profiles, what sets Calibre apart?

[benschwarz]

Without being super familiar with the tech that any of these companies are running —

The biggest thing that sets Calibre apart is that its entirely built using the Chrome developer tools APIs. I like to think about it as someone on your team who opened devtools after every deploy, and kept a neat spreadsheet of all the changes that had occurred. In short, all the data is ready and available for the taking.

This also has the added benefit that I have the power of the Chrome browser to play with. Projects like Google's Lighthouse (https://developers.google.com/web/tools/lighthouse/ — Which I'm currently integrating into Calibre!) have meant that a lot more powerful tools/metrics/resources/etc are coming along strongly, and this will only help make Calibre more powerful too.

Calibre is firmly focused on what your users actually experience, and helping you get to the bottom of 'why' quickly and efficiently. So while technically focused tools are imperative for any team, many of them aren't focused on the most important people in the equation.

Also, thanks — Good question.

[joshstrange]

I'd suggest a name change as calibre is already well established open source e-book management software. From my results you are the 6th result on google searching for "calibre".

[ianstormtaylor]

Congrats on the launch! This is really inspiring to see how polished a solo-founded startup can be :)

[adamgamble]

We've been using calibre app on Vecteezy and its helped us keep track (and improve) our site performance. Its a great tool and continues to improve!

[nlx]

Nice product, been watching this for a while. Looking forwards to using it in anger soon.

[benschwarz]

Please do! There's a fairly relaxed 14 day trial. (And you get a fresh trial if you create a new "team"/organisation