Hacker News new | ask | show | jobs
by SamyPesse 2209 days ago
GitBook CTO here:

Our production domains (gitbook.com and gitbook.io) have been blocked and locked by our registrar (Google Domains).

None of our infrastructure is impacted, all user content and databases are safe; our domains simply blocked by a heavy handed policy.

As mentioned on Twitter, we are all hands working with Google to fix this issues ASAP. We'll then share an in-depth post-mortem

https://twitter.com/GitBookStatus/status/1268554857411227648

8 comments

We've just published a postmortem: https://blog.gitbook.com/tech/post-mortems/06-20-gitbook-dom...

let us know if you have any questions!

Ok, we'll change the URL from https://twitter.com/GitBookStatus/status/1268528465990619137 to that.

I know it moves the rug under the existing discussion, but it's better than having two separate threads.

I'm curious how you feel about CloudFlare as a registrar not allowing GitBook to use an external root nameserver.

Being forcibly stuck on CloudFlare's own nameservers only sounds very nefarious, and isn't a limitation I've ever heard of with any other registrar. For instance, it would break my tooling that uses my host's APIs to control DNS records through their nameserver.

I'd be very appreciative if eastdakota or jgrahamc could elaborate on what possible reasoning there is for this restriction as well.

Cloudflare sells the domain at cost. I think the idea is that its an extra service meant for their customers, not a service for the general public. As they are a DNS provider, their customers will use cloudflare nameservers. If they didn't, they would no longer be customers.
That does make sense. If I were using Cloudflare I suppose it would be a no-brainer, and if I were Cloudflare and didn't want people not routing their traffic through me on my registrar, that would be an excellent way to discourage it. If they're forced to offer to everyone as part of being a registrar, then the combination of all of the above is my answer. Thanks!
Exactly, huge red flag. Google domains it's risky because they can ban your entire Google account including personal Gmail and any linked business ones. Can be pretty bad I'd say.
Seems like you should also move off of Google Domains, unless you have some compelling reason to use them.
In the postmortem it states that they are moving to Cloudflare
I'd like to interrupt all the sanctimonious blathering in this thread to note that Cloudflare domain registration terms and conditions are almost a verbatim copy of Google's, and includes the same unilateral cancellation clause for phishing.

Cloudflare: "Cloudflare and Registry Operator may deny, cancel, suspend, transfer, redirect or modify the Registrar Services or a Registration, or place any domain name(s) on lock, hold or similar status, as either deems necessary, in the unlimited and sole discretion of either Cloudflare ... for distributing malware, abusively operating botnets, phishing, piracy, trademark or copyright infringement, fraudulent or deceptive practices, counterfeiting or otherwise engaging in activity contrary to applicable law."

Google: "We may in our sole discretion, deny, suspend or cancel any registration or transaction, or place any domain name(s) on registry lock, hold or similar status if ... engaging in spam, phishing, or other deceptive practices."

I think the primary issue with Google Domains here is not that they have these kinds of terms, but that they enforced them in a particularly incompetent way. The original report of the phishing site was over a week old and had already been resolved, when google shut down the domain. Hopefully the people at Cloudflare are a bit better at their jobs.
Oh, I don't know that we have enough unbiased information to conclude what you concluded. One of the first comments posted in this thread today was "Is it related to the countless phishing pages hosted on your service?" from which we can deduce that the phishing problem on Gitbook is well-known to random members of the public.
Think you should investigate other options such as the known brand protection/domain asset management companies (MarkMonitor, CSC, easyDNS or their European equivalents)

EDIT: I see you're moving to Cloudflare, but I wish you the best of luck

How did you arrive at choosing Cloudflare? It's clear Google Domains has broken processes not conducive to running a business centered on user content. How do you know Cloudflare does not suffer from similar broken processes?
I doubt CloudFlare Registrar would be better in terms of customer support—unless said customer has an Enterprise plan—as their prices are just the registry + ICANN fee, no surcharge for them to make money.

Doesn't seem conducive to great customer support, but maybe I'm wrong cause I've never had to contact them.

Two years ago, support for paid (peanuts-level, $20/mo?) plan was... not great.
As a former domain registrar, I would get the authcode, unlock the domains, and transfer them away as soon as possible. It's been a while since I read the RAA (https://www.icann.org/resources/pages/approved-with-specs-20...), but it's rather extraordinary to put a domain on clientHold, which is what I assume they did to you, outside of non-payment or some kind of legal dispute.
Srsly. I would love to hear about what Google told you.

...I don't host any DNS at Google, and if this is actually not spin, now I never would.

Not that I trusted Google before, but good lord, this is egregious.

Google is the registry of some novelty TLDs as well, and you are pretty much bound to dance the way Google wants with those TLDs.
Is it related to the countless phishing pages hosted on your service? I’ve noticed an uptick.
Seems like it's sorted now? It resolves for me

    $ dig gitbook.com a @8.8.8.8 +short
    104.18.9.111
    104.18.8.111
Yes, after 6h without getting much responses from Google Domains support, we just got a notification that they unblocked our domains.

We are working on making sure that everything is correctly working.

Workaround that we've setup to allow our users to still access the platform through different hostnames will continue working.

Do you plan to stay with Google considering the experience?
No we don't, we were already planning on consolidating everything on Cloduflare, we are just going to make the switch sooner.

We'll share more details in the postmortem.

I'd be interested to know what this heavy handed policy was, assuming Google Domains gave you that information. I hope it wasn't something egregious or frivolous as I've seen with other parts of their organisation.
Ha, no way google will give any information/explanation
> consolidating everything

Are you sure that's the best strategy?

HN seems to be the last resort to get Google to help :/
You can do the same elsewhere, any sufficiently large or deep social networking would work.
gitbook.com is working but gitbook.io is having trouble redirecting.
This is unrelated, but you are the founder of codebox.io right? I always wondered why the service disappeared.
Yes, we've pivoted a few years ago to GitBook. Codebox was not working very well.
Do you use SPF and DMARC for your domain? You could have received instant reports to take actions sooner.
If you wanted good customer support you shouldn't have gone with Google. There are plenty of other more reputable domain regrestrars.
I don't understand why you got downvoted. Google's customer support is notoriously non-existent (perhaps except for stuff that brings in money like AdWords). They admit themselves that it's a business decision: https://www.seroundtable.com/google-support-staff-limits-139...
Because it's about as helpful as saying "you shouldn't have moved to Los Santos if you value safety" to someone who's bleeding on the street having just been mugged.

The same message could also be worded more like "once you get past this, I'm sure you're already considering moving registrars. But please let us know if the support you're receiving from them is as bad as (my experience / reputation / etc.)".

Or better yet, "here is a reputable site reviewing registrars for reliability and customer service" (I don't know if there is such a site, there really should, but it's unclear how it would make money).
Do you have examples?
According to whois, google.com, amazon.com, github.com, microsoft.com, netflix.com, reddit.com, baidu.com, youtube.com, twitch.tv and wikipedia.org all use MarkMonitor [1]

apple.com, twitter.com and ocado.com use CSC Corporate Domains [2]

I have no idea what such services charge, but they're all "call for pricing" and none of those companies would blink at spending $10k/year on their domains.

Not every well known brand uses such a service, though. bbc.com uses tucows, stackoverflow.com uses name.com and ycombinator.com uses gandi. facebook.com uses RegistrarSafe, a subsidiary of themselves, and almost every domain registrar is registered with themselves.

[1] https://markmonitor.com/ [2] https://www.cscglobal.com/global/web/csc//micro-domain-name-...

At my last job, we called MarkMonitor after NetworkSolutions' lack of admin security got our domain hijacked. I don't remeber the prices exactly, and I'm sure they've changed, but from what I recall, the per domain year prices were about 10x normal prices, like $100/year for .com, but they also had a mininum annual spend of I think $10k/year; to get the 'super lock' domain service was about $1000/year available on a small selection of TLDs. They were also pretty dismissive on the first call until they looked us up and you could hear the dollar signs spinning in their eyes. They were very easy to work with and professional after that though. This was while they were owned by Thompson-Reuters, they've since been sold to private equity.
> and I'm sure they've changed

How recent is your experience with MarkMonitor?

Google.com doesn't even use Google Domains....that is telling right there.
Google.com was registered long before Google Domains was created. Lots of other more modern Google domains---even .google ones---are registered with MarkMonitor as well. Google Domains doesn't compete with MarkMonitor for large businesses with extremely valuable domains.
Google Domains is not a corporate registrar. It's thus not targeted at the big corporate Google use case anyway, so that's not a surprise.
Not really useful when talking about companies like Google. They also don't use Kubernetes...
Apart from the most obvious examples, I would also consider Cloudflare's new registry service.

It's cheap, at-cost, and they support a lot of the new TLDs like .io which is also a lot cheaper.

> It's cheap, at-cost

That to me is a downside since that means that that is not a core part of their business. Financially, it makes no difference to them if I use their service or not.

I would rather pay a little extra to a company that has domain registration as a core part of their business and actually makes a profit from me.

And domain names are cheap. Even if you pay twice as much as the cheapest service, it still will not make any difference in your bottom line.

The counter is it's also risky to use a company that only does Domain Registration since it's a very low margin business and thus the risk for them shuttering is higher -- or they'll try to make it up with various erroneous fees

I know the concern of putting all your eggs in one basket is real, but since CF's business is literally to take over your domain DNS and slap on some add-on services, adding domain registration in-house seems like a good fit.

> The counter is it's also risky to use a company that only does Domain Registration since it's a very low margin business and thus the risk for them shuttering is higher

You can avoid this issue by going with a registrar that focuses on bulk domain sales (eg. internet.bs in my case, but there are more, like eNom I think?), as they have a high-enough volume that they can easily stay afloat even when charging reasonable prices and without aggressive upsells.

It's mostly the consumer-focused "$1 for the first year" registrars like GoDaddy that you want to stay away from. Those are the really problematic ones.

> but since CF's business is literally to take over your domain DNS and slap on some add-on services, adding domain registration in-house seems like a good fit.

Sure, if you want to send all the traffic of all of your users through a man-in-the-middle US-based company with a very dubious past and a questionable business model revolving around basically centralizing the internet.

It's not a great recommendation to make. It also raises the question of why they seem intent on killing off the registrar market by offering "at cost" (which honestly isn't much lower than what aforementioned internet.bs charges anyway).

How about mixing the two? Buy your domain at the cheapest registrar you can find. Pay for 9 years. Then as soon as you can transfer to some registrar you have more long term confidence in. You might have to purchase another year there to do this.

Net result: You get the domain at your preferred registrar, but you get 90% of the savings you would have got if you had it at the cheap register.

> they support a lot of the new TLDs like .io which is also a lot cheaper.

You might want to pick another example, .io is 23 years old [0]

[0]: https://en.wikipedia.org/wiki/.io

Ah yes that's true, I always seem to group .io in with the new crowd of TLDs in the sense that it became trendy "recently"; and I only mentioned .io domains since GitBook uses one, "gitbook.io".
For some reason people keep forgetting that io stands for indian ocean and is actually a regional tld like co.uk .net .de etc

Same with .ai fwiw.

.io isn't just "Indian Ocean", it is British Indian Ocean Territory. The location of the Diego Garcia military base (jointly operated by US and UK). The British expelled its indigenous population (the Chagossians) to make way for the US military. The territory is claimed by Mauritius, and the International Court of Justice in 2019 ruled (in a non-binding opinion) that the UKs separation of the territory from Mauritius was unlawful.

Some random British company convinced IANA to let it run the .io domain for their own profit. Their operation of it has nothing to do with the interests of its exiled inhabitants (the Chagossians), the British territorial and military authorities, or the US military presence which constitutes the the territory's raison d'etre.

I think it likely that, one of these days, something is going to happen to the .IO ccTLD operators. Their rights to it are very dubious, and someone else (the British government, the government of Mauritius, the Chagossians) could end up wresting it from them.

No one forgot. Everyone knows. No one cares.
What makes me uneasy about Cloudflare's registrar service is they force the use of Cloudflare's nameservers unless you have an "Enterprise" plan (paying a monthly fee for what amounts for some registry EPP calls?!) and given how they sell at cost I can't imagine the customer support in case of similar issues to this being good.
I'm a fan of Hover, personally.
Namecheap have been solid for me for several years now.
Namecheap dumping personal info without informing their customer (https://news.ycombinator.com/item?id=18063667), Namecheap threatening to shut down a site if the customer doesn’t delete two images posted there within 24 hours (https://news.ycombinator.com/item?id=14139288)
> Namecheap dumping personal info without informing their customer

Something similar happened to me – Namecheap dumped the wrong (private) information into WHOIS immediately after a redesign of their systems. It definitely was not user error.

Dealing with Namecheap's customer support to try to resolve this was possibly the worst customer support experience I've had in 20+ years in the tech industry. Lots of lies about getting back to me the next day, passing the buck, blaming everybody but themselves, extended periods of flat-out ignoring me, and eventually a complete inability to fix it.

I've been a happy user of Hover ever since, but I'm unable to recommend them – ironically because nothing has ever gone wrong with them. I used to recommend Namecheap until that nightmare happened, then I found out just how shockingly useless they are when it comes to customer support and privacy. Ever since, I only recommend services where something has gone wrong so that I know they are capable of resolving problems well. I regret ever recommending Namecheap and don't want to make the same mistake again.

Some more Namecheap horror stories here: https://news.ycombinator.com/item?id=18087862

Which registrar is recommended?
I have used hover for years and quite like them. The customer support was awesome when I had an issue with getting a .com.au domain setup for a business. Australia has some extra requirements for domains that I wasn't familiar with. I also like to have my domains separate from everything else so if I move hosts/email providers it's easy.
I like easydns - don't know who they resell thru, but it's awesome being able to call and get a real engineer on the phone and not a call center.

I've been using them for years, and never had any technical issues...

I personally prefer Gandi, but also use name.com and Cloudflare - haven't heard any horror stories about either of those 3 registrars.
I've been using Porkbun for a while now
I've been happy with my dedicated providers: IWantMyName.com (registrar)and DNSMadeEasy.com (DNS).
> Namecheap threatening to shut down a site if the customer doesn’t delete two images posted there within 24 hours

Better than what Google Domains did to Gitbook.

I haven't used them personally, but I've read a ton of rave reviews about gandi.net. Namecheap also talks a good talk, and Cloudflare has a good reputation.
GoDaddy has great support. It’s available via phone and you don’t need to be on some enterprise plan to get it.

(Disclaimer: I work there)

Wow, I’d use Google before I ever used GoDaddy, I mean that’s probably the most well-known "Do not use under any circumstances" registrar.
GoDaddy answers their phones with real people, but they are completely powerless to actually help you. You can escalate all the way to the office of the CEO (who doesn't answer their phones) and they won't lift a finger to help you. I had been a customer for over ten years with several domains, and they still wouldn't help me with a three-figure billing error. What kind of business fights a decade-long customer over an interest rounding error for them?
99% of people never need customer support, so I doubt most people consider it when choosing a service.
You're right.

Because 99% of people don't realise that their domain registrar holds the keys to their business's entire internet presence.

They can switch off your website/email at any time, with no real consequences apart from a little bit of bad PR if you have enough social media followers or post in the right forums where their staff hang out.

They can also do a shitty job of securing your domain, and let it get stolen/hijacked. The attacker then gets to set up their own MX records and collect all the password reset emails they triggered on every other important site, and pretty much own anything you doin't have 2FA set up on.

Anyone who doesn't think customer support from their business domain registrar is a thing worth paying for, most likely hasn't evaluated the risks properly.

That may be true for a given service, but I'd wager closer to 99% of people have used customer support for something in the past. It'd be foolish to disregard it when you know you've needed it before, even if not for that same service category.
I don't think it should be disregarded, just that it is not top of mind when making the choice.
You may be downvoted, but I think it is true. I think they consider price and convenience most of all (both of which happen to be google's forte)
Ever since you called my content "legacy", I've been leary of your platform.