I would be very happy if Niantec releases an in-depth write up on the how the backend was implemented and what measures they took to make it scale to handle this kind of traffic.
I don't care much for the game, but details of why the servers weren't (aren't?) able to handle the traffic interests me immensely.
I look forward to what they can further do to improve things.
For instance they don't cache pokestop images, my phone keeps redownloading the same images from pokestops and gyms, if it just cached the top 10 most visited pokestops and gyms that would be a decent reduction in data use.
Also it would help if they offloaded some of the work of the GPS onto the local phone steptracking/direction trackers it would help.
I can understand if they don't want to send coordinates of "nearby pokemon" down to the clients because that would inevitably get hacked, but the phone has to repeatedly poll.
If the phone tracked more locally it could not poll GPs until it detects locally that it's moved enough, so if you weren't moving you wouldn't be updating your position very often.
Also I read something that reckons that the game servers are currently all hosted in the US, which would explain the laggy battle experience in Europe. (Freezing at 1hp left, never being quite sure what's hit your character or good times to attack).
Offloading the GPS work to the step/direction trackers in the phone would also be great for those of us that work in big old buildings that do a great job of blocking my GPS position.
More on the images, since the images are just the ones that were popular during Ingress (I mean the same images, like some of the ones I took a few years ago in Ingress are now in Pokemon), they probably could have done even more aggressive caching for the very popular spots.
FWIW I'm in the US and I get the laggy/soft locking up experience you mentioned all the time. I think they just didn't write the app to be very reselient to and RPC failures.
If the phone tracked more locally it could not poll GPS until it detects locally that it's moved enough, so if you weren't moving you wouldn't be updating your position very often.
That's how the Android Location service works, unless you intentionally bypass it. You ask the location service to update you when it determines the user has moved X feet.
I'm pretty sure they had immense scaling capabilities, but were hitting upon bottlenecks which were unexpected. I'd be also really interested in what those were.
It's just not economically viable to scale your architecture to day-one or week-one traffic requirements. Look at Blizzard games for comparison, a new release every year or so and they're NEVER equipped to handle it. They know that by the end of the month, user count will drop sharply before stabilizing at a much more manageable level.
I have to disagree with regards to Overwatch, by far Blizzard's biggest IP in recent memory. The launch was perhaps the smoothest I've ever seen in a popular online game. A tiny minority of people had problems, in stark contrast with completely unprepared and utterly botched launches such as Battlefield 4. Blizzard did themselves a huge service with the open beta(which had over 9 million people play over the course of a week!) giving them an idea of where bottlenecks would crop up.
Another good example of this are the Steam sales, especially the big ones (summer and christmas sales render the store completely unusable for the first day or so)
It's "good enough" for the week of the sale, upgrading all the servers for a week of super heavy traffic just to see them idle the rest of the year isn't a great business plan
There are several firms that offer computing capacity online, at much more granular terms than "the rest of the year". It seems these "cloud" computing services are common topics of discussion here on HN.
The calculus has to be how much it would cost to temporarily bring in, run, maintain e.g. AWS boxes versus how much revenue is left on the table by the slow store. I presume Blizzard has run those numbers and decided it wasn't worth it.
Someone posted a list of protobuf definitions [1] used in Pokemon Go a few days ago. At the time I didn't think the submission was very interesting on its own right, but it seems relevant to your question now -- you can examine the names of the definitions to guesstimate what logic is performed on the server.
I suspect one reason for heavy traffic is that a lot of the logic you would expect to implement on the client, actually needs to execute on the server, in order to avoid client based hacks. For example the logic of whether the poke ball successfully catches the Pokemon. Sure you could implement that on the client but then you'd have third party apps cheating at the game without any verification from the server.
You can have both, if it's just randomness then you can have shared seeds which the server will verify but the local client can sample too. If they de-sync then the server wins, but the client can feedback quicker before the final response from the server.
I suppose that would enable further cheats such as not wasting pokeballs on ones you know will fail by peeking into the future.
The throw itself is client-side though, so I suspect it's already possible to cheat with "perfect throws" every time.
From my experience with the game, I believe their progression margin is quite large.
For example, they don't cache anything client side (even Pokestop assets), nor do they timeout/retry/prefetch after a network disconnect during a Pokemon capture (which is very annoying on spotty 3G networks). Last but not least, battery consumption is insane. Don't go out without a spare battery!
Backend side, I also noticed a few cases of "blinking pokemon" in the nearby grid. Surely symptomatic of synchronisations issues between instances of distributed systems (dirty reads?).
I enjoy the game a lot, but regarding the number of issues, it's still a beta.
One of the reasons is probably that people are using Pokemon Go in countries where it hasn't been released yet. For example here in The Netherlands there's 1.3 million+ players, even though the game is not even available yet in the Android or iOS app stores in The Netherlands.
I'm currently traveling in Hungary and was able to download the game on the first day of release. Is that because my Apple account has a US billing address?
I use Android myself but I've heard from colleagues that if you use a US account you can download it here. For Android it's just downloading executable through any channel that isn't Google's play store.
They also had a tonne of markets come online when they weren't expecting it - the softlaunch in Oz/NZ would have been to work through those, but once the world came online they couldn't hold it back, so might have had to accelerate the roll-out faster than they'd have liked? Technicalities of it will be interesting.
I may be naive, but Firebase may be the perfect fit for this kind of task scince it scales pretty good. Niantec would probably get massive discounts as well as an Ex-google company...
1. I tried playing this game, but really don't enjoy it. It feels to me it's not about strategy as much as it's "the one who puts in the most hours of playing wins". Felt actually like work to me.
2. I don't see this game being a long term super success, as it is now. I'd imagine cohort analysis would show the first cohorts waning in usage in the next several weeks. I imagine as more people begin playing, the overall popularity of the game will continue to rise for a while, but that hides the fact that the game has a limited shelf life with the masses.
1.) Agreed.
2.) The way I see it, they really brought a new game category into the mainstream. I foresee thousands of similar games coming shortly except instead of games, they will teach us about neighborhoods, history, architecture, health or figure out how to help us do more outdoor activities without walking around like zombies looking at our phones.
I speculate, perhaps groundlessly, that they could be racing to add the other traditional elements of Pokemon, e.g. battles and trading. So long as they begin to deliver those before people forget about the game, they can get a second, probably much longer lasting, wind.
I certainly couldn't see myself being a devotee even if they added those features, but kids the world over devoted countless hours to the gameboy games, and that's pretty much what they were.
P.S. When "playing" equates to "biking around town, walking around town, etc", I'm A-OK with play volume being an important element to "success"!
I'd be curious to see a graph of active users as time goes on. Anecdotally, I've noticed they have been losing a lot of users in the last few days (myself included) who are fed up with how absurdly buggy the game is (especially the bugs that manifest due to server issues), and the incomparable amount of battery that it consumes.
and as someone who has continued to play, thank you.
with less users, the issues have manifested less often. i was starting to wonder if they had addressed their scale issues, but i've noticed less people on the streets playing, so it might just be that the demand is much less.
Agreed. The most enjoyment I've gotten from the game so far is actually walking around and finding them and getting the best stats. As for actually using them in the gyms and such, I'm not yet interested in doing that.
I also agree, but a way I find entertainment on my quest to filling out my pokedex is by fighting gyms with the goal of adding a crappy pidgey or rattata after I've taken it.
I change the names in my replacements to something like "The King".
For a level 10 pidgey that's a lot of confidence.
I do this for a few reasons:
1) I can.
2) It's fun and keeping gyms is ridiculously hard.
3) It gives new players a chance to unseat my gym and get their daily bonus.
4) Powering up gyms is way under powered at this point.
5) This also gives other people on my team to power up the gym and reach their own goal if they like.
I assume the plan is to release new pokemon similar to the way they do throughout the generations. Also I assume they will add a battle system which is similar or exactly the same as the one on the Nintendo handheld systems since the one now is pretty lack luster. I do see the game coming down from this insane high pretty quickly but I bet it stays relevant for a long time.
Not having proximity based P2P battling tells me the implementors never played Pokemon. It's the other half of the game.
It's like directing a Batman movie without first reading the comics. It shows a complete distain for the culture, history, players of the Pokemon universe.
All games are supposed to be like this. A player has fun (hopefully) grok's it (usually) and then moves on (beat it, got frustrated or bored).
However, games that implement vegas like mechanics turn their games into an addiction. I can see that happening with this game as there are 720+ pokemon and only 150 in the game currently https://en.wikipedia.org/wiki/List_of_Pok%C3%A9mon
Though, this game is kind of like the iPhone, it's "cool" to be seen playing it (I presume, I've never played but do witness the pokehorde often). It may stick around for various reasons.
I have not seen a single mention if this yet, but it seems pretty clear to me that its popularity is partly enabled by the summer weather and will die off as soon as cooler weather kicks in.
Idly playing on your phone outside is not so bad in summer, but in winter? Forget about it. The new season of [xyz] is back on.
The competitive pieces aren't mandatory. And even then, even if you're weak, you can team up with other players, and you get to use 6 Pokemon vs. just one or a few.
The level differences will become more marginal as well as the experience required to level raises exponentially.
Seems obvious to me -- it's a game that takes time to play. Most people check Facebook for, like, 30 seconds at a time. It's like saying people spend more time playing Overwatch than checking their email? There's probably a better analogy. A bunch of interesting data in the article though.
Some people, for sure. In my experience, I usually see people checking it for 30 seconds or so but many times daily. Even if you're right, it still seems like a no-brainer that the average Pokemon GO player would spend more time playing than the average Facebook user is on Facebook.
One could argue that walking around - kilometers at a time, in the case of getting an egg to hatch - is serious exercise compared to sitting in front of a computer or on a couch.
Not saying walking is not good for you, but walking is very energy efficient. Your heart rate is probably higher reading HN, at least for me. Doing some sprints and other movements, like playing football, or moving the lawn, will give you a good workout though.
If you want to burn fat, walking is one of the best ways to do it. It you want to build muscle, there are better options. But long walks are absolutely good for certain kinds of exercise.
you also burn fat while eating pizza, but you put in lots of more calories. Walking doesnt use many calories. Its still healthy though, and better then nothing at all.
The success of Pokémon Go as the first breakout AR "belief circle" certainly seems like a victory for the Scooch-a-mouti. I wonder how the Librarians Militant will respond?
Do people really spend an average of 22 minutes/day on Facebook? That's astounding....I find it hard to believe unless that's including Messenger or something (which not a lot of my colleagues even use since there are better, more secure chat alternatives)
Not really, it seems just like any other fad. The technology involved has been around for a while, and nobody paid much attention to it before.
I'd be interested to see articles detailing technical difficulties that they may have overcome, but I've seen way too many articles just saying "Pokemon GO now has more X than Y".
You can discount it as a fad but it's also a resurgence of Pokemania. People who grew up catching Pokemon in elementary school are doing it all over again, along with a new generation of kids.
Whether you like it or not, it's become an overnight GLOBAL phenomenon and a unique one at that.
Not really, no, because I had already been an Ingress player. So the wow-factor of walking around to various spots for a geo-based-AR game is like 3-years-old.
On the contrary I think it's an extremely interesting phenomena, I just find the statements that it has more downloads than instagram, more daily users than, more miles walked then fitbit etc... stupid.
It is not comparable to any of these, the metrics do not contribute to the conversation and these titles are nothing but clickbait.
Pokemon Go is arguably the first killer app for collaborative Augmented Reality. It shows app developers that there is serious public interest in the 'right' type of AR and will no doubt open the doors for serious monetization attempts.
My prediction for the next big thing? The AR equivalent of snapchat filters that you superimpose over yourself, your friends or enemies, visible to people whose cameras are running the AR snapchat app. Unfortunately, adverts.
To be honest, I used AR mode for the first few days but I turned it off eventually. It had bugs on my Nexus 5x where it would do some weird stuff to the live preview from the camera, like stretching the video and other artifacts. It also drained my battery significantly faster. Finally, it's just plain easier to catch Pokemon with AR mode turned off, which kind of sucks since that's the killer feature.
I'd like to know how many users have AR mode turned off. Because while it is a fantastic part of the app, making it more immersive, if most users don't actually have it turned on it means that AR still isn't there yet for smartphones.
Although there's a toggle in the game labeled AR that turns on/off the camera as the background while catching Pokemon, I think Pokemon Go has a broader AR implementation.
My friends playing it have all discovered new things in a city they've lived in for years because of Pokemon Go. So even if they play the game with the camera turned off while catching a Pokemon, there's still a map always on showing how in-game places match up with the real world.
This translates to a weird phenomenon: the other day when we were standing outside a gym by a church, my friends and I noticed on our phone screen that a battle was happening at this gym. Instinctually, me and a few others took our eyes off our phone and turned our head to look at the spires of the church, since gyms are shown in-game as tall structures.
It sounds strange, perhaps even cheesy, but Pokemon is engaging our imaginations in a pretty interesting way.
There are reasons to turn it on sometimes, probably mostly once you get to higher levels. A tip I got from someone attempting to do "curveballs" was that with AR on you can adjust position to both keep the Pokemon on-screen and throw a curveball more easily.
Th 'AR' mode is terrible. I had hoped for at least an attempt at registration and tracking. It's not AR, it's just a gyro based 3D app that happens to use the camera feed as a background.
Isn't that how augmented reality works? I can't think of any other way to do it, unless you are tracking specific objects and overlay something on top of it.
It would be nothing without the IP! Any company with a backlog of IP has senior management in meetings right now asking themselves "what will be our AR story".
While it's one of the primary complaints (after purchase), the market would seem to suggest that people prefer smaller/thinner phones in terms of purchasing decisions.
Sounds like Pentagram Monster series. The one I have with 4100mAh battery lasts for over 20 days, if I'm not using it heavily. And while it is heavier than typical smartphones, it's only about 1cm thick. As far as I know there's also a newer model with 6000mAh battery.
Of course it's been a complaint forever but how many flagship phones with great battery life do you count? My point is that More battery-hungry widespread applications = More pressure on manufacturers to answer to those complaints.
It's also been a complaint forever that people's parents, friends, children, and selves keep dying. And yet somehow that hasn't translated into enough market pressure to make them all immortal.
Of course in reality the market pressure for protection from death is much greater than the market pressure for improved battery life while seeing weaker results in the market. Your battery life is not a market pressure issue.
As thaumasiotes has already pointed out battery life on phones has been a number one priority for years. In fact, the shortcomings of batteries are now one of the biggest bottlenecks in transport, energy, infrastructure and more. Some of the best minds/companies in the world are already investing everything they've got in fixing the problem.
Except through yesterday it was only released in five countries. Today it'll be released to three more countries [1], and it will continue to expand going forward.
I'm tired of those articles "Pokemon Go is bigger then X", they contain no useful information besides the fact that Pokemon Go is really popular, yeah Ok we got it, now please tell us something interesting that ain't obvious ;) ... Don't get me wrong, I'm happy Pokemon Go is such a success because it's good for Nintendo as it is their first mobile game (even though they haven't developed it inhouse) and this success will help them to stay afloat
I don't have the game, but it sounds like GPS power drain is the big problem. Is there not some way to improve that? Maybe only turn on GPS intermittently and use accelerometers, wireless signals or the camera to track your position from the last known point and any landmarks?
Just from using the app you can tell there are optimizations to be made that would dramatically save on battery life before even getting to the power draw from location services. Pokestops for example don't appear to locally cache their images and descriptions for long.
Aside from the assorted obvious optimizations that people have discussed, it might be possible to "offload" the GPS to a separate battery-powered Bluetooth GPS receiver. I didn't see any results for this on a quick search, so I'm not sure whether it's been tested or not.
Of course, if you don't already have one a Bluetooth GPS receiver appears to cost at least 5x as much as a low-end external battery and 2-3x as much as a pretty substantial one.
I know iOS and Android already use non GPS data to aid in location sensing, but I'm not certain to which degree they actively avoid the actual GPS sensor.
It's frankly stunning that Nintendo never attempted this in the mid 2000's when MMO's were the hot thing. I've seen the idea floated so often of the past decade.
Why? Pokemon Go is the most talked about topic in tech at the moment. Considering this a tech news site, it would seem more than appropriate to show this content.
"Please don't submit comments complaining that a submission is inappropriate for the site. If you think a story is spam or off-topic, flag it by clicking on its 'flag' link. If you think a comment is egregious, click on its timestamp to go to its page, then click 'flag' at the top. (Not all users see flag links; there's a small karma threshold.)"
Just hide if you're not personally interested. IMO flagging that would be an abuse of the facility and just wasting moderator time.
Do all flags get reviewed by moderators? I was under the impression that if some threshold number of people flag the same topic, it disappears automatically.
I don't care much for the game, but details of why the servers weren't (aren't?) able to handle the traffic interests me immensely.