Blog

Success Criteria

If product development is the exercise of coaxing behavior from your stakeholders that helps your organization to achieve its goals, then success criteria are the yardsticks you use to judge how well you’re doing.

Without understanding what success looks like, how can you know if you’ve achieved your goals? Success criteria are a set of objective measurements that allow you to chart that progress.

I’ll know it when I see it!

When I’m wearing my web design hat, I’ll frequently hear someone of importance exclaim, “I want to see it pop!” My response, couched as diplomatically as I can, is, “To whom? And how? And is that really what we want from this?” Thus begins the success criteria exercise, the result of which will be a set of measurements that tell us how well we’re doing. Are we achieving our goals?

Google Analytics Dashboard
Google Analytics Dashboard

Before you can begin this exercise, though, you need to understand the context. A process like that I described in my square milk jug discussion is a good start. With the context fully understood, you can begin mapping out your criteria.

Success criteria must be:

  1. Defined

    OK, I agree, that’s obvious. The point is that something is better than nothing. And chances are, the first time around, you’re going to get them wrong. That’s OK. You will learn over time what matters.

  2. Measurable

    You don’t want confusion when people consider your success criteria. They need to be expressed in objective terms.

    Rather than, “Searches must be fast and easy,” you should use, “Searches must return results in less than 2 seconds and the user will click a result on the first page 90% of the time.”

  3. Collectible

    There must be methods available to collect the data you need to evaluate your success criteria. This is not the issue it used to be, but, in cases where you are monitoring a large number of events within a single view, some analytics tracking software may balk. You must prioritize your criteria to ensure you are collecting the most important metrics.

    Also, while it may be feasible to collect all that you are asking for, the reports generated may be far too large. The important numbers may get lost in the sea of data. Start with a core set of high priority success criteria and expand as needed.

  4. Useful

    You need to consider what the metrics are really telling you. In the example above, “The user will click a result on the first page 90% of the time,” is certainly telling you that your search engine is doing a good job. But what if the vast majority of those clicks are for pages that are linked on your home page? Maybe the design of that page is flawed. Success criteria should be considered as part of a whole rather that looked at individually.

  5. Provisional

    The results do not necessarily connote success or failure, especially early in the life of your product. What matters is performance over time. As you accumulate metrics, ask your self: Were we over or under aggressive in our definitions? Does this even matter any more? And are we missing anything?

    As a product matures, your users will also change. They become more familiar, more experienced. They may have needed hand-holding in the early stages, but are now experts who find the hand-holding slows them down. Likewise, your success criteria should change along with this evolution.

Real World Application

We performed this exercise for a client who asked us to help improve their search engine optimization (SEO). This seems straightforward: Increase the amount of traffic driven to your site through organic search. This usually means speaking your visitors’ language in the content that you produce. The problem in this case is that the client uses a specialized language that cannot be easily changed. How do you bridge the gap? Let’s look at this through the steps we just defined.

  1. Defined: Yes, we want to see an increase in the number of visitors acquired through organic search. We also want to see people begin to use the specialized language, but that will take time. To prime this pump, we will salt our content with appropriate synonyms that tie real-world use to the specialized terminology.
  2. Measurable: Our criteria should be expressed like this:
    • Organic search traffic will increase, month-over-month (MoM), by 10%.
    • We will see our controlled vocabulary in search terms starting next quarter and their use, MoM, will increase 20%.
    • If a term from our list of synonyms is used as a search term and there is a hit for an article tagged with the canonical term that has an average position of 5 or greater, it will be clicked 20% of the time. (Phwew, that one is rather long.)

    This may seem aggressive, but we’re jumping a pretty low bar.

  3. Collectible: This was not a problem with these criteria. The first two were easily handled by Google Analytics (GA). For the last, we needed to use the Search Console from Google’s Webmaster Tools.We conducted this exercise 6 years ago. Since that time, Google, Yahoo, and Bing have encrypted submissions to their search engines, so most entries in GA are flagged as “(not provided)”. We would no longer be able to collect this data. In our case, however, this is no longer a problem since use of the specialized terminology has taken root.
  4. Useful: Our goal was to increase traffic to the site for people who would find useful the specialized approach our clients provided. What we discovered, though, was that it was not enough to focus on our content. While we kept these initial criteria, we added new ones.
  5. Provisional: Our criteria quickly revealed that our list of synonyms was not effective. Mining our internal search logs helped to refine that list.  We also needed to use our terminology outside of our site, so we focused on social media, which was just beginning to pick up stream about this time. We added criteria related to how often we were retweeted or mentioned (ping back) in others’ blogs.

This approach allowed our client to be heard in the world at large with their specialized language. While they need to continue to bring in fresh eyes, the processes put in place during this exercise will ensure that this is an integral part of their web presence.

Categories: Blog, Business Planning, Product Management

Tags: , , , , ,

Bill Lenoir
08 Dec, 2015