[Lewisham]

Look at this chart:

https://4.bp.blogspot.com/-GLc5ve5_djc/V-ysZgW6uDI/AAAAAAAAD...

The worst case scenario was 5x, which was a factor of 10 off. If you are that far off when you do your capacity planning, you can be pretty sure you've got problems throughout your entire stack.

[roninb]

There is literally no way anyone could have expected it to do this well. Nintendo has steered clear of mobile gaming because they expected all of their IP to flop in the mobile world.

That said, what you said is still true. If you're thinking you're going to have 100k users, you might be willing to allow a lot more data to be transmitted and/or processed than if you had 10m users. Just looking at tracker alone, having to transmit and measure distance between a dozen points and ordering them is a lot more work than checking whether a dozen coordinates are within a range and listing them in any order.

[jandrese]

> Nintendo has steered clear of mobile gaming because they expected all of their IP to flop in the mobile world.

I don't believe this for a second. Nintendo knows where its goldmines are. IMHO it is far more likely that they steered clear of mobile because they didn't want to cannibalize their Game Boy sales. It's a classic case of a big corporation being slow to react to change or even trying to stop it because they hold a dominant position in the old system.

Nintendo would much prefer if "mobile gaming" still meant Game Boys but the world has already made their decision and they can't stop it. They can't ignore the sales figures on what would have otherwise been an obscure goofy spinoff fitness app on the Game Boy. They've ignored the mobile market for too long and now there is pent up demand. Without Pokemon Go I doubt you would have seen the sudden scramble to develop and release a Mario game for iPhone.

[kaishiro]

>> Nintendo would much prefer if "mobile gaming" still meant Game Boys but the world has already made their decision and they can't stop it.

I'm not sure this is entirely accurate. The Nintendo 3DS (I assume what you mean by "game boy"), has been a very successful product for Nintendo. And sales of the 3DS have actually spiked as a result of Pokemon GO.

[CannisterFlux]

While it may be a success, the 3DS has sold half of what its predecessor the DS did (~60 million vs ~130 million units worldwide). The NX will probably sell half again of the 3DS. The market for handheld gaming systems is dead or at least terminally ill. And I say this as someone who plays on a New 3DS daily (don't get into Monster Hunter if you want a life).

The reason Nintendo has avoided mobile gaming is probably down to fear of piracy. The DS was ruined (from Nintendo's POV) by flash carts, and with the changes they made on the next generation the barrier to entry for copying games is now lower on Android/iOS than on the 3DS. I think that when they do finally go mobile, the games will have a lot of server-side checks going on. The games will be a pain in the neck to play even if you are paying. For example, always-on internet required will only work on non-rooted devices, that sort of thing.

[kaishiro]

For sure - we're on the same page, although I'm a little more bullish on the future of some sort of hardware gaming platform, like the upcoming NX. I'll be the first to admit that could be wishful thinking, but I just can't go over the thought that my phone isn't a serious gaming device. Monster Hunter is actually a perfect example. When we get the crew together we'll rock for 5 or 6 hours, swapping out chargers. I can't see myself ever doing that comfortably holding a piece of glass.

[roninb]

That's a fair opinion and maybe it's true, but it comes from the horse's mouth that they were afraid Mario and Pikachu wouldn't do well on a phone.

[merb]

well Pokemon Go compared to the good old Pokemon Silver/Gold is just crap and that was a Gameboy Color Game.

[mtrpcic]

That chart is not at all accurate. If it were, it would be telling us that over time, they expected no change in data volume? They set a single number as "Expected volume", and a single number as "worst case", with no planning for growth at all? That's what this chart is showing. Either they were so poor at planning that this chart is accurate and the fault is on them, or the chart is inaccurate and we can't really trust any of the data it represents.

[itcmcgrath]

I can guarantee you the chart is accurate as I created it. Yes they did expect change, but the only number that matters for this graph is "how big can we get" and then "if we get that big by the end, is there enough capacity". The answer to to the first one turned out to history making ; no-one in their right mind would bet on happening beforehand, and the second turned out to be "Yes".

[mtrpcic]

Since you created the graph, could you provide more context as to the units of measurement and scale for the axes? Are you using the derivative of growth as biot mentioned below, or something else? This graph shows two lines that stay static over time (meaning that either Niantic started off on day one with their entire server infrastructure already running with no plans to scale up, or they did not plan to scale up as demand grew), and one upwards trending line that shows actual changes in traffic over time. I'm trying to discern what this graph is supposed to represent, and if it's supposed to represent the expected traffic over time versus actual, it's showing that there was no expected growth in traffic.

[itcmcgrath]

I cannot dive into too much more detail than is in the graph since it's still sensitive information. Y-axis is essentially traffic to Cloud Datastore (think: after layers of caching, etc), x-axis is date.

The 2 lines can be thought of as ceilings or upper bounds, hence why they are static - this are the numbers that traffic was expected to eventually reach at peak.So you can think of it as, "we thought we'd be looking at graphs that had this line as the top and traffic would be some curve underneath.

Obviously from the graph shown here, we/they needed a tall graph.

[mtrpcic]

I think that my issue with the chart (and it's such a minor issue to quibble about) is that you're effectively treating your single dataset (actual traffic over time) the same way you're treating your annotations (expected and worst-case traffic). Both of these different things are represented the exact same way in your chart, which is a confusing way to structure things. I would alter the appearance of the ceilings/bounds to not be represented in the legend, and instead be on-chart annotations that show where those expectations were relative to the actual traffic. I would also recommend adding even the most rudimentary labels to the axes.

[itcmcgrath]

No arguments there. There's a balance that needs to be struck between technical detail and marketing appeal. Not everyone is going to agree on where that balance is, less so when you're trying to share sensitive information without giving too much away.

[BinaryIdiot]

> The 2 lines can be thought of as ceilings or upper bounds, hence why they are static

I think this is the disconnect. This wasn't not obvious to me. It looked like everyone was expecting flat growth at either 1X or 5X. If there was a line showing what they thought would be the traffic that goes up (which would be expected) in addition to the ceiling lines then I think there would have been a lot less confusion.

I wonder if a single bar graph would have illustrated it better with overlaying colors for each ceiling.

[f_allwein]

https://xkcd.com/833/

[i336_]

> I cannot dive into too much more detail ... since it's still sensitive information.

Question: When should we start checking around for posts containing

A) High-level technical overviews with some basic implementational detail

B) In-depth analyses of the stack you built, the challenges you faced, what improvements you folded back into various open-source components, what you'd have done differently, etc etc

?

I'm thinking in terms of timescales - like n months or so. I suspect (A) will be a little easier (and quicker?) to publish than (B).

[biot]

If one interprets the graph as "derivative of transaction growth" instead of "number of transactions" then it makes perfect sense. The derivative of linear growth is a horizontal line, whereas quadratic (edited, thanks acomar) growth would be a line that has slope.

[acomar]

Quadratic, not exponential. d/dx(e^x) is e^x. That's actually why e is called the natural log.

[devilsavocado]

Do you know how those estimates are made? As someone who only passively follows some Pokemon news it was obvious that Pokemon Go would be absolutely huge on launch. It was all over facebook, youtube, reddit etc since the day it was announced. I don't think I've ever seen so much hype for a game, including major AAA titles. It seem strange to hear that no-one in their right mind would think it would happen.

[itcmcgrath]

Yes, and I can tell you even though I spend every day working with extremely large scale systems I wouldn't have told them "you should expect 10x larger", yet alone 50x. Their initial estimates would still have been an very large launch.

The Niantic team did incredible things given the instant historic success that became Pokemon Go.

[devilsavocado]

Sorry if I implied you did something wrong, not my intention at all. I'm just curious how someone comes up with an estimation at all for a game that doesn't have pre-orders, and where similar games don't already exist.

[itcmcgrath]

They ran a closed beta.

[BinaryIdiot]

> Yes, and I can tell you even though I spend every day working with extremely large scale systems I wouldn't have told them "you should expect 10x larger", yet alone 50x. Their initial estimates would still have been an very large launch.

The generic 1X, 5X and 50X are hard to understand in this context I think. Pokemon is so popular it's very difficult in imaging what the real numbers actually are. For instance on launch day in America and Asia I would have expected insane numbers (many hundreds of million).

I also feel like the 1X, 5X and 50X number placeholders are useless in this conversation because it doesn't give a sense of scale at all.

[itcmcgrath]

"For instance on launch day in America and Asia I would have expected insane numbers (many hundreds of million)."

Given no app had ever done this in history over the first few weeks, yet alone on launch day with no marketing, I doubt it.

[alasdair_]

The US has roughly 330 million people in it.

"Many hundreds of millions" implies at least three of them.

I would not expect the entire population of the US to be playing on day one, especially the roughly 50% of them that don't even own a phone.

[MBCook]

That shows their estimates were terrible.

The article says (in one of the only bits of real information) that they blasted past their estimates with only Australia and NZ just 15m after launch.

Whoever came up with those numbers must have had some serious methodology flaws. I know they couldn't predict that it would become the biggest online game ever for a while, but the initial demand prediction was clearly way off even before it started growing like a rocket.

[KirinDave]

No offense, but your comments are somewhat outrageous to me. I've been on the receiving end of one of those graphs (different scale) because suddenly things happen (we were placed above the apple logo as a feature with no warning AS WE LAUNCHED).

And you're sitting there as the engineering lead or staff going, "How do I even feel about this? Fortunately I have no time to feel because I am off to fight fires." I didn't go home for 2 days, I worked 82 hours that week and >70 the next.

Complaining that the estimates are bad for a product that literally broke everything we know about how to build a successful mobile game and has scaled to a truly unprecedented level is meaningless. Obviously no one expected this. Obviously the engineers wouldn't have wanted it. Obviously the world will respond the way it will to our work.

Show some compassion. But also some humility. None of us are qualified to make projections in the face of phenomena like this.

[MBCook]

> Show some compassion. But also some humility. None of us are qualified to make projections in the face of phenomena like this.

You can't predict it will be the biggest game ever, that's not possible. But I feel like their initial estimates were still too low and you could predict it would have been higher. I don't know if it was based on how well Ingres did but...

Pokemon is a huge property. It's had tons of games, movies, sleeping bags, an incredibly successful trading card game, etc. Just having the Pokemon brand on something makes it VERY big.

In the game, you live out the Pokemon dream. This isn't just Pokemon Puzzle League. This isn't Pokemon Mystery Dungeon where you navigate cute little Pokemon around and play a top-down rogue like. You FIND AND CATCH Pokemon in the wild. It's exactly what Ash did in the TV show or comics.

Also, Pokemon are cute as hell. That plus the novelty of the AR stuff meant this had a lot of potential. "Look, I found a cute Eevee over here on my potted plant!" Those pictures were EVERYWHERE. That's tons of viral advertising.

But there is also the in-person effect. You want to compare what Pokemon you have with other people, and that encourages you to get your friends into it. But people were walking outside with their phones playing the game, and they quickly got spotted by people asking "What are you doing?"

All these things make it clear to me the this game had a high chance of success.

There's no way to know it would go to 50x what they guessed or would top the charts. Given their numbers I wonder if the expected should have been closer to 7x and the worst case at 20-25x.

The popularity they got would have taken basically anyone down. That was going to happen. I'm just surprised the estimates weren't much higher.

[KirinDave]

> Given their numbers I wonder if the expected should have been closer to 7x and the worst case at 20-25x.

What numbers? We know that their estimates and their realities were quite different, but not knowing the real numbers we have no way of even beginning to judge what's reasonable and unreasonable here. For all we know, they modeled directly after the most successful game in the Android market at that date as a baseline and then said, "At the worst case we'd expect 5x THAT."

You can write a ton of paragraphs about how cute Pokemon are, but the truth is that the Pokemon AR game was a massive risk. AR games have had extremely limited update. It seems incredibly unreasonable for me to expect that those folks should have realized apriori that they were about to release the most popular mobile game ever created.

I, for one, will not throw stones. I don't get why you feel the need to assert that you (or anyone) could have done a better job by setting a 20x or 25x target. Or that you could have not only forseen it was necessary, but convince everyone around you that the capex was justified.

Why are you so keen on assigning blame and shame in this scenario? Some of our peers made history. Can we be happy for them for 6 months before immediately backseat driving about how much better we all are in our armchairs?

[bduerst]

>That shows their estimates were terrible.

Pokemon Go became the most-downloaded app in history within the first few days.

Are you saying that's the bar that people should estimate at launch?

[kmm]

It shows that they were over their worst estimate total volume in one day. Were they really expecting less than a few million user-days in total?

[bduerst]

From the article, it seems like a few million users/day is a typical expectation, but record-breaking isn't.

>Throughout my career as an engineer, I’ve had a hand in numerous product launches that grew to millions of users. User adoption typically happens gradually over several months, with new features and architectural changes scheduled over relatively long periods of time.

[jandrese]

It's hard to estimate with no data. Nintendo had never released any other mobile games and the few NES/SNES ports had modest sales (at admittedly high price points for mobile).

When it comes down to it, the game is closer to a fitness app than a Pokemon game and can be described as bare bones. The fancy accessory wasn't even available at launch. Many of the players are people who have not bought a Pokemon game in over a decade, or ever in some cases.

If there is one area I can solidly criticize Niantic for it is rushing the other region releases so quickly. It's clear the servers were already badly overloaded and they just started adding countries left and right. I know it sucks for people who live in those countries to have to sit out while the rest of the world is having fun, but it's really not much more fun to sit at the eternal loading screen because the backend is entirely on fire.

[petters]

No, it shows that successes like these are impossible to predict.

[BinaryIdiot]

I'm not sure why you're being downvoted into oblivion; you are completely correct. Yes, to address the other points raised, it's difficult to predict how popular something unreleased will be. But this is Pokemon. Their last game release, which is limited to the niche Nintendo 3DS console, sold over 3 million copies in just 3 days. And this was the Omega / Alpha games which were essentially just remakes of older games.

A mobile game, available to be downloaded onto hundreds of millions of phones, that is also FREE? I feel like they broke 100 million in the first day at the latest (counting totals from each region's first day). It wouldn't surprise me if it's significantly higher than that.

Media, users; everyone has been BEGGING Nintendo to release IP to mobile devices but they have kept it locked up in their own hardware. If no one had even a rough idea of the possible popularity they most certainly had a very, very wrong methodology.