Measuring and Trending - A Framework

Continuing on from an earlier post introducing some terminology around how to measure engagement within apps and sites, I'd like to continue with introducing a very basic framework for determining what to measure and considering how to display those measurements.

Let’s take a look at each of these four components:

Goals

You need clear goals to help you choose the right metrics. You really can’t start with the things you want to measure; you should select the things you want to measure based on any of these goals. We’ve got five categories… and they all interrelate:

Let’s look at some sample goals for a company's intranet called 'myCorptown' across these different categories:

Adoption

1. Ensure new employees and new users are engaging with myCorptown content for the first time, including the email newsletter
2. Build employee buy-in by telling our story through our vehicles
3. Reach our employees ‘where they are’ including via mobile and kiosks
4. ‘Give them what they want’ by being able to respond to the popularity of content
5. Get our employees to think of going to myCorptown first for company information

Engagement

1. Get our employees to give feedback on content including likes and comments, submitting questions, etc.
2. Have our employees be conversant in an informed manner about company news with fellow colleagues
3. Get our executive team to choose myCorptown as their communication vehicle
4. Get our employees to help each other (employees as communicators)

Retention

1. Get our users to return to myCorptown more often (more than 1x per week)
2. Encourage our lapsed users to return to myCorptown
3. Reduce stale content in areas on myCorptown where dynamic content is expected by our users

Task Success

1. ‘Give them what they need’ in addition to what our users want, including via search
2. Solve our users’ problems via self-service where possible (without a help desk call)

Happiness

1. Ensure our employees are able to articulate the company story...
   • What we stand for
   • How does my work fit into the story
   • How does my work touch consumers
2. Support a more transparent culture at the company
3. Decrease frustration during activities on myCorptown
4. Support our employees through times of change

Signals

Eventually, we want to think about mapping goals to lower-level signals which are things we might want to consider monitoring. These signals are what we might be able to pay attention to over time in order to determine progress toward or to what degree we've fallen short of our goals.

Here are some of the example goals and the signals that could be identified for them:

Goal	Signals
Get our users to return to myCorptown more often (more than 1x per week)	Providing content relevant to users Identify users who return to visit the site within a week period
Get our employees to give feedback on content including likes and comments, submitting questions, etc.	Identifying posts that have higher numbers of views Identifying posts that have high numbers of comments Monitoring submitted questions
‘Give them what they need’ in addition to what our users want, including via search	Identifying usage of tools and links that have typically been difficult to find Monitoring search results for successful and failed queries

Metrics

Based on our signals, we’ll decide on things to measure... the metrics... like the number of home page views and the site’s bounce rate.

The measurements themselves are the key to everything, but rarely do we want to take the actual metrics and present them for interpretation by themselves; we need to provide some context and a story around the measurement. Just installing a web analytics tool, turning it on, and looking at digits tick up will not lead to any understanding around your goals.

KPIs

This is where we get into how to display the metrics that have been chosen to track the signals and tell a story about our goal, and I’ll present a few varieties again using examples from above.

Trends

Here’s one that is a simple indication of home page stickiness with trending:

This KPI shows:

• Number of retained user home page views minus bounces as percentage of all views
• Over a time period of 7 days
• Indication with color of current metric against target w/in a certain window
• Trendline

Here's another example with trending showing off-hours visits:

This KPI shows:

• Number of views during off-work hour segments
• Over a time period of 5 days
• Indication with color of current metric against target w/in a certain window
• Trendline

Trends w/History

Here we have an example of displaying a trend over time, bringing history into the context:

This KPI is showing us:

• Number of posts related to influencers
• Over a time period of 30 days
• Indication with color of current metric against target w/in a certain window
• Performance with color against last window and previous year’s window

This happens to be an example of where a higher metric is better; year over year the increased percentage is a positive indication, so it is green.

Here's another example of trending with history showing submitted questions:

This KPI is showing us:

• Number of questions submitted to portal team
• Over a time period of 30 days
• Indication with color of current metric against target w/in a certain window
• Performance with color against last window and previous year’s window

The opposite of the previous example, here a lower metric is better; year over year the decreased percentage is a positive indication, so it is green.

• Example of where lower metric is better

Snapshots

These next two examples are presented in different ways, one simpler visually and the other with more context; they provide the same data in a snapshot for a specific window of time.

• Average visit length
• Segmented into groups
• Current metric for one window
• Shows spread/breakdown detail
• Does not show trend

• Average visit length in minutes
• Over a time period of 7 days
• Indication with color of current metric against target w/in a certain window
• Trendline

Here's another example of snapshotting and two different ways of presenting the data. Note these example visualizations include a metric that is abstracted (into three different categories):

• Rating of frustration level on portal from survey
• Current metric for one window
• Shows spread/breakdown detail
• Does not show trend

• Most common rating of frustration level from survey
• Over a time period of 1 fiscal quarter
• Indication of current metric against target w/in a certain window
• Performance against last window and previous year’s window

Hopefully this series of examples gives you some ideas for how to think about showing what's being generated from all your data! More important though you should have a better understanding of how to approach deciding what to measure based on what matters to track according to the goals you have for your site or app.