[grvdrm]

I'll enjoy watching this thread evolve. Some thoughts from my experience:

- Everyone asks to translate simpler spreadsheets and Excel charts/graphs into dashboards in your BI tool of choice. As soon as it's there, they'll ask you why they can't export to manage the data themselves. This vicious cycle can sometimes be stopped but is a slow-motion drag on productivity in lots of orgs.

- Build in validations, and/or work on ways to check the dashboard. Dashboards sometimes put their builders and consumers on auto-pilot. The dashboard "must be right" but could easily have a bug or inaccuracy for weeks/months/etc. that isn't obvious without some external validation.

- The dashboard never has the "right" metrics - users will continue asking for changes. Be your best advocate and say no as a way of understanding the importance of the ask

- Related: always ask why about everything you're building into or modifying in dashboards. Business users often ask for things without an ounce of rationale.

- Related: taking away is harder than not doing at all!

Finally, I think most dashboards miss one fundamental point. Imagine you're the CEO/COO and you've got this beautiful 3 or 4-chart dashboard in front of you. What should you know about what you're seeing? What's the succinct summary?

I like building in spots to write 2-3 sentence executive summaries.

[conductr]

I work closely on BI projects but from a finance perspective. The concept I like to explain to the BI teams is that the dashboard is always just a snapshot of “what” is happening. But the underlying base level data is always needed to understand “why” it’s happening. And without the why, there’s no actual intelligence gain.

Take a metric like Average Order Value (AOV). It may be ; total sales / order quantity. But as that metric is used it’s often being compared to something like last year, last month, or a plan and anyone interested in that number is really interested in understanding the “why” it has changed from some other point in time/scenario.

For that, you actually need to bring in line item details behind orders as each order has multiple products/skus and they likely sold at different prices from a year ago or what was expected in a plan. An analysis of this has a name, price-volume-mix analysis or PVM.

I always seem to have to explain this to BI teams when I join a new company and am seeking data. I’m currently going through it with a BI team, that apparently the BI tool wouldn’t store this information. It’s like it only stores aggregate values so it’s not even possible to get base level data for analysis (without major architectural changes). I don’t know if that’s normal in BI or was an implementation decision at some point but I’ve come across this same thing on a handful of companies and as I said I really have to drive this concept for those teams. When I ask of it I’m usually met with a “why would you need that info / give us a use case”. Which means, the don’t even understand how un-intelligent their BI tool is or why the execs likely aren’t feeling like investing in BI has been worthwhile (eg. Ever build a dashboard that then goes unused? I probably wasn’t perceived as useful for some reason like this).

This could be more concise put as, understanding your end users needs. Understand the difference of what people ask for is often different than what they need. If they ask for AOV metrics, they’re really saying “I need to understand AOV” and that’s done via PVM analysis.

[grvdrm]

Similarly titled towards finance. I specialize in what I'll call decision analytics for insurance underwriters.

> Which means, the don’t even understand how un-intelligent their BI tool is or why the execs likely aren’t feeling like investing in BI has been worthwhile

And this relates to what I was thinking about in my first comment. I once was conversing the COO of my company (my last job), at a 1000+ person company, and asked him if he thought more concise requests for things would drive productivity. He, point blank, said: "sometimes I don't even know what I'm asking for"

I've remembered that moment for years. In so many situations, the actual BI/dashboard is the least important part of the puzzle. Instead it's all of the conversation and discovery to understand the real need(s)

[conductr]

> He, point blank, said: "sometimes I don't even know what I'm asking for"

Totally relate to that AND I'm often on the receiving end of those questions in a live setting (eg. board/exec meetings). Funny to stumble on this because just last week I told someone on our BI team, there is not any one "use case" I can lay out. The use case is this, assume I need to answer any random question that comes up. I need analytical enablement not a fancy dashboard in most instances. It's not to say dashboards don't have their place, but they're just the easily digestible summary of underlying data that's meant to highlight areas and raise those questions about "why..."

[SeriousM]

Oh thank you guys! I came to this post with the thinking of "dashboards are hard to build and when they exist it's hard to extend them". My boss is asking me from time to time to add yet another dashboard to grafana and all I can do is add another piece of specific inner working values because lack of traces - the "what" part. Up until now if was as if a car driver is asking for the average air volume intake yet in really he probably needs the "why" the air volume changes. The "probably" part is important here: if the driver is the test-driver then he needs the actual value, the what; is the driver the track performance guy he needs the why; and sometimes they need booth values because they feel more secure with more information. You guys just opened my eyes, thank you!

[proamdev123]

That’s a fantastic point. The unstated underlying request isn’t to see the same chart, the same way, every week/month/quarter. It’s actually: “Find any abnormalities in the data that could be threats or opportunities, and show me THAT — in a chart, table, email, or whatever medium make makes sense.”

[grvdrm]

Yes and to say it another way it’s often “tell me the story of something I don’t already know”

[clusterhacks]

"Business users often ask for things without an ounce of rationale."

Deserves extra upvotes just for this statement.

This has always been painful to me working in the data analysis and reporting space. When I get many requests for dashboards or reports that lack an answer to the question of "how will this be used?", I seem to find the requesting groups are cost-centers in the larger organization and are somewhat obsessed with processes and procedures.

This is rarely a good group to build a career with . . .

[proamdev123]

As a business user myself, often times we are simply responding to an edict from Leadership that amounts to: “You guys should have some dashboards!”

Other times it’s because a supervisor or internal customer likes to see certain things on a chart, and putting the chart in Power BI/Tableau/other tool will make them prettier than Excel charts.

Very few people, starting from the top on down, have a good understanding that dashboards mean very little in and of themselves.

[tailspin2019]

Brilliant summary - mirrors my thoughts and experience quite closely.

Validation/testing has always been a challenge, especially given that dashboards are by definition quite “full stack” implementations where testing just the front end or back end is not sufficient and testing both in isolation can also often be challenging due to the huge possible variations in input data.

Mocking data is also hard because dashboards may also lean a lot on database-side calculations/filtering.

All of this has lead me to take quite a full-fat approach to testing dashboards, by using a real DB populated with test data, and testing the full complete application stack (driven by something like Playwright or Cypress) as well as more granular unit tests where a mocked data layer may be used.

I’m also looking at introducing visual regression tests next time I work on this kind of thing. The visual aspects of dashboards can easily drift over time even if the data is correct. You’re often theming charting libraries for example and the compliance of the theme can drift slightly if you update the library without really checking every detail of the visual appearance/layout every time. Or you may not even notice the “visual drift”…

[grvdrm]

Interesting - great points

> Validation/testing has always been a challenge, especially given that dashboards are by definition quite “full stack” implementations where testing just the front end or back end is not sufficient and testing both in isolation can also often be challenging due to the huge possible variations in input data.

Constantly evolving but I've always tried hard to keep calculations away from the display tools. So, I put lots of things in SQL SPs, or in Python, or more broadly in tooling that allows me to recreate the summary data without the front-end. My nightmare is having to check a PowerBI calc that itself is based on an underlying SQL calc. Which one is wrong? Now spend twice as long figuring it out!

> The visual aspects of dashboards can easily drift over time even if the data is correct. You’re often theming charting libraries for example and the compliance of the theme can drift slightly if you update the library without really checking every detail of the visual appearance/layout every time. Or you may not even notice the “visual drift”..

Love it, very smart. Why I prefer tables for many things too - one less thing to maintain and check.

[tailspin2019]

PowerBI is a WHOLE other can of fish - I haven't spent long enough with it to figure out how you build a test suite around that - but it sounds tricky!!

[grvdrm]

I would like to use the Men In Blake eraser pen on my PBI experience. Can’t stand it!

[tailspin2019]

You have my condolences :)

[mushufasa]

> Finally, I think most dashboards miss one fundamental point. Imagine you're the CEO/COO and you've got this beautiful 3 or 4-chart dashboard in front of you. What should you know about what you're seeing? What's the succinct summary?

Having been on both sides of this, I think the challenge is that the CEO/COO's job is to figure out "what should we do about this?," which is the right approach to coming up with that summary (it's not just "here's a text version of the chart"). And the corollary challenge is that, in most cases, non-technical people with domain knowledge are the ones who need to produce the analysis: so any feature incomplete dashboard is going to stymie them and any general framework that requires a technical person to step in for code or configuration is going to slow the process to a crawl.

It's the rule (not the exception) that (especially if things are going poorly) the next step is asking more questions, which involves investigating something else in more detail. A dashboard, however pretty, is as useless as a doorknob if it doesn't have the needed information.

I have found that dashboards per say are always great as the high-level KPI trackers, like the things you would consider hanging on a wall in an office (e.g. "revenue growth this month" or "new customers acquired"). You'll always want to know that information, and many people of unrelated departments need to have that information shared to them.

The other helpful area is a deep-dive domain-specific analytics programs, like for example Google Analytics, where it has a very full featureset for non-technical marketing people to go in and drill down to answer questions. The UI/UX designers of that product have spent years honing and A/B testing which types of graphs to show where, and mapping out how to have people click around to find what they're looking for, to the point it is pretty easy for non-technical people. They even have courses and certifications on how to use the system.

Organizations that try to internally build a feature-complete system like google analytics for a specific domain need to consider it like building an entire software product (even if there's a general low/no-code BI SaaS to assist) because you'll need collaboration between general technical experts and non-technical stakeholders with changing and vague requirements. It can be done, but likely only with years of investment and UI/UX research, just like any other software product that solves a domain problem well. In practice: millions of dollars.

Technologists often forget that Excel *is* a turing complete programming language (and it's a functional programming paradigm too!). If an org is not committed to spend years and millions of dollars on deep dive analytics for a specific domain, the right choice is almost always using a commercial analytics system for that domain that costs less than the internal build, or embracing the trusty spreadsheet.

[itsoktocry]

>I think the challenge is that the CEO/COO's job is to figure out "what should we do about this?

Totally agree. I'd even go a little further and say the business is in trouble if the CEO doesn't know "what we should do about this". It's the CEOs job to know those things, and it's the data team's job to provide the tools to make those decisions easier, faster and better.

[grvdrm]

> Totally agree. I'd even go a little further and say the business is in trouble if the CEO doesn't know "what we should do about this". It's the CEOs job to know those things, and it's the data team's job to provide the tools to make those decisions easier, faster and better.

I agree with one modification. Also the CEO's job to empower folks to pitch what they think the CEO should know too. I've worked in plenty of successful shops where the CEO's answer to "what should we do" is "I'm not sure yet - what do you think?" - and that is a golden opportunity to show your chops if you've got the opportunity

[dgudkov]

People frequently overestimate the role of data in decision-making. Metrics, numbers and other quantitative information don't tell the full story. For a CEO to make decisions, the full picture must include qualitative information - risks, opportunities, market events, competitor's actions, etc. Metrics are just part of the full picture. Far less significant than many BI developers and "data teams" tend to think.

[yabbs]

It's my experience that you get easier, faster or better. Choose any 1, rarely 2, never 3.