| I agree with most of the sentiments in the article and am always keen on learning new meaningful, i.e. not individual but high-level measures. The notion of "measuring something is better than nothing" is not helpful, I feel. You not measure what is just easy to measure, but what is actually imporant and gives real business insight. This, as the article states, is typically very hard to define and execute. Also, I would widen the perspective of "CTO needs to provide data to CEO" / engineering effectivness a bit, as this is not the most important thing you need to know. Ultimately, as a company you wanna know if your business is going in the right direction or not. You want to spend more money on things & activities that add to the value of your product offering in the eyes of a customer so that the customer will pay for it and you want to spend less money on things customers will not pay for.
But what are the value-adding activities and which are non-value-adding? When will a customer buy your product or service? For me, this sets a certain order of importance of what things to measure/quantify to answer the following questions: 1. Deliver customer value. Are we building the right product? 2. Generate business value. Can we generate revenue? 3. High value product. Do we keep quality high and build the right next features? 4. Code quality. Are we building the product right so it is maintainable and exensible? 5. Team chemistry. Is the team aligned on the goal and healthy in their interactions and spirit? 6. Process efficiency. Is success repeatable? === (1) Answering that should be not too hard as you can measure any kind of customer feedback on your products, be it youtube likes, alpha tester feedback, support calls, surveys. (2) Once you know you have a product customers want and like, you need to know is the customer facing part of your organization (marketing, sales, training & education and support) connecting to said customer, so that they can sell and actually generate revenue? This is much harder to get data for. You can measure the number of licenses, lost and new customers, or the trends in the business volume of services, but what does that tell you about the ability of your organization to be able to sell a good product to customers? I am not sure here. (3) is to reflect on the business success (ROI) on new features that you implement. Do you keep building a high value product? Are we effective in communicating the new features value proposition to our customers? There is soo much to measure here. Feature level:
Measure via BI the business impact of new features, how often are they used?
Measure the resources needed to deliver a feature.
Measure number of support cases / bugs reported per feature.
Measure the estimated time vs the taken time for a feature. Quality of the overall product:
Measure the yearly mean number of support cases per week.
Measure the yearly mean number of bug reports per week.
Measure the yearly mean number of crash reports per week.
Measure number of major field incidences per year. Do we make the customer feel cared about after the sale?
Measure lead time to resolve support calls.
Measure customer ratings of support calls. (4) Code quality like compiler/sonar warnings or test coverage are the easy measures, but might not give you insight. I prefer to look more highlevel again at the product quality which ties into (3), like How often do tickets come back from testing to development?
Measure number of regressions reported by customers after a release.
Measure number of (real) hotfixes needed after a release. Answering the more fundamental questions of code quality like "how easy is it to add new features" is a bit more difficult. I have not found good measures yet. For (5) totally agree with the article, you should never measure how many lines of code were produced, how many tickets closed or the like. The question of team happiness is most important for team work. Ther are good tools for that like OfficeVibe or Glint. Measure the employee turnover.
Measure the number of uninterrupted hours of work time / total time present at work per week.
Measure the mean Office Vibe score.
Many further answers could be drawn off of Office vibe: Are people at ease, having a good time and enjoying interactions with their peers? Is there no sense that single individuals try to succeed in spite of the efforts of those around them? Was the work a joint product? Was everybody proud of its quality? Do they take enjoyment in their work? Is there trust and mutual esteem among the peers? For (6) you want to know: Are our processes such that success are repeatable? On a company level: Measure number of botched releases / roll backs. Measure number of failed audits per year. Measure number of open CAPAs per year. Per Team level: How long are compile/ CICD times? How long do code reviews lie around before taken up? How easy is on-boarding of new employees? etc |