|
|
|
|
|
|
by aprilledaughn
2893 days ago
|
|
|
- You’re absolutely right, not all the 100M bug reports are actually bugs. However, we highlighted this under the "Time to Close" section: "These are most likely not programmatic bugs, but could be support issues or spam." How do you think we should highlight this more to avoid confusion? - About bug resolution time, Instabug is used by many companies as their main bug reporting tool or they forward these bugs to another bug tracker like Jira and we have a two-way sync so whenever it gets resolved over there, it’s resolved at Instabug as well. That’s why we used the word "resolved" not "fixed" because each company has their own definition. I hope this makes sense. - About the quality and reliability of the data: Oh, we didn’t mean to be protective about this! On the contrary, we’d love to get your feedback. What would you like to know? - About your third point, I respectfully disagree. As the person who spent the most hours working on this report, I can tell you honestly that it was not for PR. We just wanted to put something out there that would hopefully be valuable to the people in our community. We initially shared this with our own users for them to have benchmarks. This is the first time we’ve released anything like it, so it was an experiment for us to be honest and I’m loving all these comments because it helps us know what to do better next time around. |
|
|
I think that this report is only useful in showing app developers that the patterns they encounter in their bug reports are common in the entire ecosystem, not special to their specific app requiring further investigation. Keep the patterns, keep the information about integration with external tools (customers might find it useful). Scraps the rest.
The main problem in the report is that you try to answer questions which your data and analysis is inherently incapable of answering. For example:
- "Which manufacturers have the most bugs?" - "Which UI orientation has more issues?" - "Which locale has the most bugs?" - "How does battery affect app stability?" - "Which OS has buggier apps?"
As other commenters have mentioned, your results could be just artifacts of the user demographics (or any number of other confounders). The answers are, at best, meaningless.
There are significant inconsistencies in figures 1 and 2. They definitely do not agree with "Errors discovered through Instabug are most likely to be resolved within 24 hours of being reported." (except in the narrow technical sense of the first day being the most likely day).
Even if the data was sufficient, there's no mention of statistical significance in comparisons. For example, Danish is the locale with the most bugs per user. However, you have quite a lot of locales and random variability is expected. Is the difference statistically significant?