Content Precedes Design

Measuring and Trending - A Framework

Continuing on from an earlier post introducing some terminology around how to measure engagement within apps and sites, I'd like to continue with introducing a very basic framework for determining what to measure and considering how to display those measurements.

Let’s take a look at each of these four components:

Goals

You need clear goals to help you choose the right metrics. You really can’t start with the things you want to measure; you should select the things you want to measure based on any of these goals. We’ve got five categories… and they all interrelate:

Let’s look at some sample goals for a company's intranet called 'myCorptown' across these different categories:

Adoption

1. Ensure new employees and new users are engaging with myCorptown content for the first time, including the email newsletter
2. Build employee buy-in by telling our story through our vehicles
3. Reach our employees ‘where they are’ including via mobile and kiosks
4. ‘Give them what they want’ by being able to respond to the popularity of content
5. Get our employees to think of going to myCorptown first for company information

Engagement

1. Get our employees to give feedback on content including likes and comments, submitting questions, etc.
2. Have our employees be conversant in an informed manner about company news with fellow colleagues
3. Get our executive team to choose myCorptown as their communication vehicle
4. Get our employees to help each other (employees as communicators)

Retention

1. Get our users to return to myCorptown more often (more than 1x per week)
2. Encourage our lapsed users to return to myCorptown
3. Reduce stale content in areas on myCorptown where dynamic content is expected by our users

Task Success

1. ‘Give them what they need’ in addition to what our users want, including via search
2. Solve our users’ problems via self-service where possible (without a help desk call)

Happiness

1. Ensure our employees are able to articulate the company story...
   • What we stand for
   • How does my work fit into the story
   • How does my work touch consumers
2. Support a more transparent culture at the company
3. Decrease frustration during activities on myCorptown
4. Support our employees through times of change

Signals

Eventually, we want to think about mapping goals to lower-level signals which are things we might want to consider monitoring. These signals are what we might be able to pay attention to over time in order to determine progress toward or to what degree we've fallen short of our goals.

Here are some of the example goals and the signals that could be identified for them:

Goal	Signals
Get our users to return to myCorptown more often (more than 1x per week)	Providing content relevant to users Identify users who return to visit the site within a week period
Get our employees to give feedback on content including likes and comments, submitting questions, etc.	Identifying posts that have higher numbers of views Identifying posts that have high numbers of comments Monitoring submitted questions
‘Give them what they need’ in addition to what our users want, including via search	Identifying usage of tools and links that have typically been difficult to find Monitoring search results for successful and failed queries

Metrics

Based on our signals, we’ll decide on things to measure... the metrics... like the number of home page views and the site’s bounce rate.

The measurements themselves are the key to everything, but rarely do we want to take the actual metrics and present them for interpretation by themselves; we need to provide some context and a story around the measurement. Just installing a web analytics tool, turning it on, and looking at digits tick up will not lead to any understanding around your goals.

KPIs

This is where we get into how to display the metrics that have been chosen to track the signals and tell a story about our goal, and I’ll present a few varieties again using examples from above.

Trends

Here’s one that is a simple indication of home page stickiness with trending:

This KPI shows:

• Number of retained user home page views minus bounces as percentage of all views
• Over a time period of 7 days
• Indication with color of current metric against target w/in a certain window
• Trendline

Here's another example with trending showing off-hours visits:

This KPI shows:

• Number of views during off-work hour segments
• Over a time period of 5 days
• Indication with color of current metric against target w/in a certain window
• Trendline

Trends w/History

Here we have an example of displaying a trend over time, bringing history into the context:

This KPI is showing us:

• Number of posts related to influencers
• Over a time period of 30 days
• Indication with color of current metric against target w/in a certain window
• Performance with color against last window and previous year’s window

This happens to be an example of where a higher metric is better; year over year the increased percentage is a positive indication, so it is green.

Here's another example of trending with history showing submitted questions:

This KPI is showing us:

• Number of questions submitted to portal team
• Over a time period of 30 days
• Indication with color of current metric against target w/in a certain window
• Performance with color against last window and previous year’s window

The opposite of the previous example, here a lower metric is better; year over year the decreased percentage is a positive indication, so it is green.

• Example of where lower metric is better

Snapshots

These next two examples are presented in different ways, one simpler visually and the other with more context; they provide the same data in a snapshot for a specific window of time.

• Average visit length
• Segmented into groups
• Current metric for one window
• Shows spread/breakdown detail
• Does not show trend

• Average visit length in minutes
• Over a time period of 7 days
• Indication with color of current metric against target w/in a certain window
• Trendline

Here's another example of snapshotting and two different ways of presenting the data. Note these example visualizations include a metric that is abstracted (into three different categories):

• Rating of frustration level on portal from survey
• Current metric for one window
• Shows spread/breakdown detail
• Does not show trend

• Most common rating of frustration level from survey
• Over a time period of 1 fiscal quarter
• Indication of current metric against target w/in a certain window
• Performance against last window and previous year’s window

Hopefully this series of examples gives you some ideas for how to think about showing what's being generated from all your data! More important though you should have a better understanding of how to approach deciding what to measure based on what matters to track according to the goals you have for your site or app.

Measuring and Trending - Terminology

I find a lot of confusion out there when it comes to conversations about analytics for websites and applications. Here's a brief primer for how to use the right terms in your discussion...

Metrics

A metric is a something that can be
measured based on quantifiable data,
or at the very least,
estimated based on anecdotal data.

An example of a metric could be your heart rate or blood pressure taken by your doctor at a specific visit.

Key Performance Indicators (KPIs)

A key performance indicator (KPI) is
a visual indicator that tells a story
about one or more measurements, in context.

An example of a KPI could be the trend of your heart rate or blood pressure over a certain period of time. If the trend is upward, an up arrow might be used with the percentage of increase over that time period. It might be colored red if the implication is negative.

Web Analytics

Web analytics is the
measurement, collection, analysis, and reporting
of web data for purposes of
understanding and optimizing web usage.

Web analytics tools such as Google Analytics mainly provide metrics for consumption via reports and APIs. What they usually don’t do is put them together in a nice PowerPoint deck to present to your boss!

Web analytics tools may already provide some KPIs in dashboards. These are more sexy and ripe for executive consumption. However, most practical KPIs are usually either:

• Formulated automatically via additional software that process the metrics via APIs
• Manually compiled by a site administrator

This is because most businesses want to answer very specific questions and monitor custom KPIs that aren’t readily available as an include with a tool.

Past Performance *is* Indicative of Future Results

If our team/organization has taken on and sustained the practice of project retrospectives (see my earlier post), we're now faced with the opportunity not only to reflect and maybe get some clarity on the bigger picture successes and challenges, but also to memorialize those so we can actually do something with that information.

All of those lessons learned around capturing what propelled and confronting what dragged will forever be buried in those notes/outputs unless they are actually shared, understood, and considered next time around by other teams.

I'll repeat: Lessons learned die in PowerPoint decks… unless shared, understood, and considered next time around.

Arguably, there is a responsibility to generally socialize these lessons learned across other teams and beyond within your organization so some things actually start to take hold. These things can span what might need to be integrated into the way everyone approaches a plan, how they kick off a project, or similar challenges are handled at any point by anyone who encounters them.

This is true Knowledge Management at work, and as with any KM paradigm set up for success, will only be effective by combining the right tools, processes, and people responsible for sustaining this continuous improvement.

Project Retrospectives

Quite different from the iterative practice of sprint retros (see my earlier post), an overall project review has goals that are a bit different. This is also intended for a broader audience... time to explore how everyone feels about how the project went and what was produced. An important point: sometimes this can be done with a client and sometimes without. Of course, the focus may be substantively different in each of those cases. Just like a sprint retro, the idea is not to make a case for who to blame or ‘for the record’ airings of grievances—we want to capture the things that enabled successful outcomes and confront what encumbered quality and value.

Here are some examples of what could be good to discuss...

Capture what propelled

Try to determine what is repeatable and sustainable.

Confront what dragged

Consider what was detracting from delivering well.

Basically, we want to figure out what to adopt and what to avoid; can’t say it more simply than that.

The who and how is bit different from sprint retros...

Who

Involve everyone who contributed anything at any point—even if the team had a developer or designer a few months ago that is now on a different project, they need to be invited and make time to contribute to a complete picture.

• Project Manager
• Sales
• Product Manager/Product Owner
• Technical Lead(s)
• All team members involved in delivery throughout
  • Not just who was on the project at the end

Keep in mind that it may or may not be appropriate to do an internal and external project retro separately or combined.

How

It is essential to have an agnostic teammate facilitating (someone uninvolved with the project). Having the facilitator be impartial means they will be able to record things in a way that doesn’t assume a lot of the nuances and context that the project team has—because others who weren’t involved need to be able to understand the takeaways.

• Someone from outside the project should facilitate
  • prompt people
  • record +s and –s
  • encourage participation (not pass judgment or validate/invalidate the contributions)
• Artifact (deck, notes, etc.) produced with key summary points that can be approached and consumed by anyone without project context

The key here is assembling that artifact. I'll talk a bit more about what to do with those artifacts in another post soon!

Sprint Retro Guidelines

Continuing on from sharing some perspectives on sprint retros (see my earlier post), here are some simple guidelines to get you started on running better sessions with your team:

Celebrate what's great

Try to determine what is repeatable and sustainable.

Identify what to improve

Consider what (not who) is detracting.

Who

• Only the actual team members should be included in a sprint retro:
  • The tech lead(s)/developers, designers, etc.
  • The product owner and project manager/scrum master
  • If there are client people who are actively working on the project (contributing to the sprint goal and the increment/work product), they can and should be included as well.

How

• Come prepared with talking points
• Someone from this group (a peer) should be able to facilitate
  • prompt people
  • record +s and –s
  • encourage participation (not pass judgment or validate/invalidate the contributions)
• Try and propose a solution when identifying a problem
• Review past retro points and gauge progress

Remember... Team performance that continuously improves is the result of a team feeling accountable to each other!

A Perspective on Sprint Retrospectives

You've probably been in these meetings and on these calls with your team... "Anyone have anything to bring up?" Crickets... but your project manager can say they ran the sprint retro and move on. This isn't the way things should work and your team will never evolve without embracing this opportunity and substantively participating with courage. I’m going to briefly orient you with what the goals of doing this are—it’s not just a sprint ceremony to check a box on so we can say we’re doing Agile.

Your sprint has concluded and the team has held its sprint review. What we care about now is how the team thinks about working on that increment. Give everyone a chance to step away from the day-to-day and anything chaotic—debrief and decompress… amongst themselves. Feedback that we’re gathering here is from the team and for the team. People on the team are accountable to each other—this is not an external management review of the team’s performance or efficiency.

So with those goals understood, let’s review who we need and how to conduct these. Only the actual team members should be included in a sprint retro. This does not include anyone else—no exceptions. Others might review the output of the retro, but they are not invited to participate. In line with the team being accountable to each other, people need to feel comfortable to bring up concerns without anyone in attendance who doesn’t have an equal stake in what is affecting the team.

How to go about this? Come prepared—I can’t tell you how many retros I’ve been on where nobody speaks up. Self-select a facilitator from the group—they’ll record what is brought up and hopefully encourage participation. When you’re contributing a problem, try and propose a solution alongside it. And, have the previous retro summaries available to you can gauge progress from the previous sprints.

Instead of recording black and white feedback—what was good and what was bad, consider 'do less' and 'do more.' We really want to hold up and celebrate what went great! We want to identify those things that really propelled the team forward and maybe formalize some of them—get to a point where we can repeat and sustain. On the other side, we also want to identify what to improve—and important here to not point out who, but the things that held the team back... What is detracting from the team moving forward? What didn’t work so we can stop doing that thing and try a different thing that might be better?

I'll get into some details of how you can run better sprint retros in a follow-on post soon!

Sharing Responsibility for Client Satisfaction - Best Practices

Continuing on from an earlier post introducing how to think about an approach for paying attention to client satisfaction, I'd like to offer some ideas here on what I've found to be helpful aspects for bringing that approach to life. Monitoring and ensuring client satisfaction is a shared responsibility. For Delivery Excellence to be woven into the way we work requires collaboration, and thus client satisfaction is not just the job of your project manager or sales staff if something goes wrong. It’s a shared responsibility—all team members have a part to play to ensure that continual client satisfaction.

Client Satisfaction Best Practices

Here are some areas to consider in coming up with the approach that’s going to work for your particular client and project team.

Cadence & Constituencies

Decide who to involve and how often to focus in. I’d really encourage an internal steering call—set this up for as often as you have a client-facing status conversation and hold it just prior (e.g.: the afternoon of the day before or morning of). A common mistake is not inviting everyone you need into that tent for that steering discussion; every client touchpoint needs to be at least informed, if not involved, from sales through management, and that’s a key driver of being able to position ourselves to respond proactively vs. reactively. We want to discern and get out in front of risks before they become issues.

Questions to discuss sound like:

• Is the client happy?
• Do they like our communication style?
• Do they seem disgruntled and do we know why?
  • If we don’t know why, how do we find out?

Proactive vs. Reactive

Maintain a risk checklist. Our project managers are usually concerning themselves with quantitative risk management… and the classic risk register applies there. In this case, I’m talking about the notion of a qualitative risk management tracking tool. Develop whatever will work internally—it can be as simple as a wiki or notebook page or a more robust spreadsheet. Just make sure everyone has access to it and is informed of updates.

Transparency

As uncomfortable as it may be, we must strive for as much transparency as possible. We usually have the best of intentions and we need to make sure that comes through. During difficult conversations, maintain the context that we’re in this together. Any tough news should be relayed in spirit of partnership—it’s not ‘us vs. them.’ If there is a conflict, it is everyone’s problem as a unified team to work through. So yes, you may need to commit to solving the problem on your end, but you can’t just say it will be fixed in an unrealistic aspirational reassurance.

Relationships

Consider the relationships between your project team members and those on the client side. Stakeholders often gravitate toward and communicate better with different teammates on the service provider side. If you need to get a bit deeper and understand more about a problem candidly or you have to deliver a difficult message, think about who on the team might be the best relationship-wise to take on that conversation on behalf of your team. If someone has earned trust with a particular counterpart on the client side, you'll want to capitalize on that to everyone's benefit.

Tailoring

Lastly, tailor the messaging and interactions. Think about your audience. If you’re dealing with an executive stakeholder vs. a client’s project manager or product owner, think bullets rather than paragraphs. If they're not in the weeds, don't expect them to acclimate to the same level of detail. It’s going to go a long way to making the interactions and communication most efficient and beneficial.

Sharing Responsibility for Client Satisfaction - An Approach

I think we all get the importance of trying to maintain positive client relationships. It’s a key part of this concept called Delivery Excellence. Especially in a consulting relationship, there’s really a difference between work product check-ins and statuses vs. “how is the relationship?” and “is value being delivered?”

Thinking about client satisfaction means considering (and keeping a pulse on) how your clients feel about working with you and your organization; not just after the project is over, but starting where we first pick up the ball and periodically throughout.

Teams are usually pretty good at understanding if we’re on track with the work itself that we’re tasked with doing, but often we’re not as good at picking up on how the collaboration toward those technical outcomes is working well or if we’re falling short of expectations (and not communicating enough along the way.) So, trying to keep these goals in mind can really help to avoid issues and cultivate some positive capital in case something does start to go awry… so we can better address it together.

Client Satisfaction Approach

So let’s talk about what this looks like generally. Of course, every project and client is different, but you'll need a plan for trying to get that feedback and another for how to do something with the feedback.

In line with the idea that we don’t want to wait until the end to become aware of these concerns, think about the ways you could bake in the feedback opportunities across your engagement. That could involve being very deliberate about creating those opportunities. Sometimes we need to try and elicit both the good as well as the not-so-good feedback when it’s not-so-obvious. I’d really like to emphasize: if you’re asking for this information, you have to be prepared for whatever answers come back; not only for what you want to hear.

You’ll then need to be able to talk about what you’re going to do with the feedback. It’s not going to be enough to just nod and be appreciative for your client’s candor. As project teams and delivery organizations, we need to:

• track it
• agree and assign ownership for it
• follow up on it (amongst yourselves and with the client)

Following up will involve reporting back on progress towards change, improvements, or any other actions. These may be internally managed only, but in a lot of cases, there are going to be things you’re working on with client visibly.

My Slides - Making Secured Content Discoverable in SharePoint - SIKM Boston

Thanks for attending my presentation at this month's SIKM Boston meeting!

You can download the slide deck here.

My Slides - Making Secured Content Discoverable in SharePoint - ESDDC

Thanks for attending my session at Enterprise Search & Discovery (co-located with KMWorld) presented in collaboration with Cryptzone!

You can download the slide deck here.