Web Analytics Management

Metrics for Smart Content Management - Metric #6- Task Completion ROI

This is the last post in my series on metrics for evaluating the effectiveness of web site content. Over the last five weeks I’ve presented five metrics that you can use to benchmark site performance basaed on content and navigation effectiveness. Today's metric, Task Completion ROI, is built on the first metric I presented: Task Completion Effectiveness (TCE). It is actually pretty easy to calculate, and it is a great way to introduce concepts of value to content in organizations where this connection isn't seen clearly, such as government, healthcare, non-profits or corporate intranets.

Task Completion ROI: Measures ROI in Web site operations, content, applications, and labor relative to the budget spent on these initiatives.

Calculation of Task-Completion ROI: Number of task completions multiplied by a dollar value given to each task-completion type compared with the total spent on creating the task application or with the avoided-cost alternative to offline.

Calculation Note: Total expenditures will need to be determined based on budget allocation, set-aside funds, hourly labor rates, etc. Dollar values for completion of tasks will need to be assigned based on the calculated cost of completing the task offline.  Find out the costs for developing the application, the marketing program, the event, the video development, and then estimate a value for the completed task based on a time and materials calculation if the task had to be completed offline.  Then add a “goodwill” value.

Goodwill, according to the Merriam-Webster definition is “the favor or advantage that a business has acquired especially through its brands and its good reputation; the value of projected earnings increases of a business especially as part of its purchase price; the excess of the purchase price of a company over its book value which represents the value of goodwill as an intangible asset for accounting purposes.” So, you’re adding in “goodwill” over the costs. Sound like a stretch? Maybe a bit…think of this as a conversation starter or straw man for valuing content that may not have a clear dollar value. Even better, find out from the sponsors of the application what their business and revenue goals are before it ends up on the web site.

Example: There were 130 submissions of grant applications during the month.  Each submission is valued at $200. This amount is based on the time saved by having the applicant enter the data directly into the database, rather than having staff transcribe the data from printed forms. This results in avoided cost or savings of $26,000. 

Another Example: Calculate and compare the cost and success rates associated with promoting research studies and grants in the offline channels and the online channels to determine which one was most effective at attracting qualified applicants.

So, that about does it for this series. If you come up with other metrics for evaluating content sites, let me know. I think this is an area that could use some additional development.

________________________

And lastly, hope to see you at eMetrics Toronto at Analytics in Real Life on Wed, 3/20 with Anuschka Van Dijke from KPMG and Nancy Duley, from American Society of Clinical Oncologists.

 

 

 

Metrics for Smart Content Management - Metric #5 - Navigation to Content Score (NCS)

NCS is a quick way to see if your visitors are spending too much time trying to navigate to content. The fewer pages it takes a visitor to content, or the more direct path, the better. NCS complements the Content Engagement Score (CES) which lets you know whether visitors are using the most important content on your site.

Navigation-to-Content Score: Compares number of navigation page views to all site page views.

Calculation: Percentage of navigation-type pages viewed compared with all content page views. This is derived from dividing navigation page views by all site page views. A lower percentage indicates a more efficient visitor interaction with the site.

Calculation Note: As with the other metrics in this series, you need to identify navigation pages and measure them separately. Pages include: 
o    home pages
o    portal pages
o    search engines
o    site maps
o    pages provided for navigational purposes

Example: There are 40,000 page views of navigation pages out of 150,000 total page views for the month. This equals a navigation-to-content score of 27%. Here’s how a report might look:

You can make this a much more valuable report by running this for different segments of visitors based on geolocation, domain, first time vs. repeat, as well as adding visit segmentation attributes such as acquisition source. If you use this with the segmentation, you’ll be able to see which visitor groups are having an easier time finding site content.
________________________

If you're going to eMetrics Toronto, be sure to stop by Analytics in Real Life on Wed, 3/20. I'll be moderating a panel with Anuschka Van Dijke from KPMG and Nancy Duley, from American Society of Clinical Oncologists. They'll be sharing their stories on how to get resistant management to come around to the value of analytics. Hope to see you there...

 

 

 

 

Metrics for Smart Content Management - Metric #4 - Content Distribution Index (CDI)

The Content Distribution Index (CDI) is another engagement measurement that you can use for benchmarking.  The CDI gives you a sense whether visitors believe content is valuable and interesting enough to share through Facebook, Twitter, email and other sharing links that you have on your site, such as AddThis.This is an easy to calculate metric and one that can help you rate overall relevancy of your site.

Content Distribution Index:  Measures whether content holds value and relevance based on visitor opinion.

Calculation: Content shares based on total number of sharing links usage divided by page views. A higher percentage of content shares indicates that content was distributed more successfully.

Calculation Note: Sharing events will need to be captured by tagging the article sharing links. As with the other metrics in this series, you’ll have more accurate results if you do not include navigation, search and index pages.

Example: During the month, there were 200,000 total page views and 12,000 article shares. This equals a content-distribution score of 6%.  Here’s how a report looked for one of our clients.
 

Taking a look at the results, you need to account for the number and type of articles that are incorporating the share function. A further analysis is needed to understand whether the article content is relevant and whether the position and presentation of the share link is clear to the visitors. Conduct this with a visitor/visit segment and you’ll be able to see which of your audiences is really engaging with content and which ones aren’t.

Metrics for Smart Content Management - Metric #2 - Mission Critical Content Effectiveness (MCCE)

Let’s return to the topic of content as being a measure of success. In my last post on Task Completion, I said that only sites that sell advertising can consider content viewing to be a goal. I admit that I overstated this a bit…if you’re a non-profit or government site, this could actually be pretty important. 
For non-profits, content that leads to donor contributions, program funding, volunteerism, and advocacy is highly important. Government agencies, branches and divisions have clearly stated missions against which they are funded.  I think of this as Mission Critical Content.

What is Mission Critical content? Content that directly or indirectly supports funding and revenue. For example, showing donors the readership of articles that they’ve funded.

Defining mission critical content can be a tricky exercise because you have to draw a line…or else all content can be viewed as critical. Here’s a few ways to go:

• Conduct an analysis to determine the pages viewed before visitors go to top tasks.  These pages are influential enough to bring a visitor to the task completion.
• Review the site to determine the pages that were designed to communicate mission. On a government site this could include: About Us, Our Mission, News, or posts from the head of the agency.
• Review the results of the top 100 pages and downloads on the site, cull out the non-content pages (home page, search, portals, index, and so forth). Review the resulting list to determine which pages are central to mission.

Once you figure out the content that is mission-critical, calculating the metric is pretty straightforward.

Mission-Critical Content Effectiveness (MCCE): Measures the number of times mission-critical content, including video, is viewed or files are downloaded.

Calculation: Expressed as a percentage based on total views or downloads of the top content pages of the site that most relate to organization mission, divided by the total number of all site content pages (do not include portal, search, or navigation-based pages). A higher percentage indicates a better MCCE rate.

Example: During the month, there were 25,000 total page views and downloads of mission-critical content and 150,000 total page views of site-content pages, including downloads. This equals a mission-critical-content effectiveness rate of 17%.
 

Based on the data, it would make sense that you could run this report for specific segments, such as visitors from targeted geolocation, organization type, content interest and branded keyword acquisition.  It would also be highly valuable to run an analysis showing the impact of mission critical content views on tasks such as donation, article sharing, membership and other top tasks.

Metrics for Smart Content Management - Metric #1 - Task Completion Effectiveness (TCE)

Management of content proliferation is important of course, but in evaluating content effectiveness it is also important to consider the goals of the site. All too often we hear that “reading content” is a site goal. While that may be true if you’re selling advertising, it is generally not true for content rich sites. (For more on this, you’ll want to read next week’s post on Mission Critical Content)The purpose of content is to drive an action or a task, such as a purchase, event registration, donation, newsletter sign up, lead generation form completion, using a selection wizard and so on.

Measuring Task Completion gives you an initial sense on whether the web site is effective in getting visitors to complete the tasks on your site that are associated with business goals. Once you establish Task Completion Effectiveness, you can use this as a benchmark to determine whether you content management actions are working. The goal: raise the site’s Task Completion Effectiveness percentage.

Task Completion Effectiveness (TCE): Measures completion of site tasks associated with your goals and mission

Calculation: Expressed as a percentage based on the total number of task-completion page views divided by total number of site page views. A higher percentage indicates more successful task completion.

Calculation Notes: The completion page for each task will need to be identified, such as a “thank you” page or a submission confirmation.

Example: During the month, there were 500,000 total page views. Of this total number, there were 10,000 task-completion page views. This equals a task-completion effectiveness of 2%.

In the chart below, you’ll notice that we’re using “visits” and not “page views” as a baseline metric. Why? This report was done in Google Analytics. GA bases its metrics on visits and can not correlate events to page views. While we could have created filters to obtain the page view data, we opted to go with what GA does best. Using visits as an alternative still provides a reliable measure of task completion effectiveness. If we had done this in SiteCatalyst, we would have been able to correlate the events to page views.

 

 

In this example, the TCE has gone down slightly over a 3 month trend. Next steps should include reviewing the page view totals of the task confirmation pages to see which ones are experiencing declines, and then determining what factors are influencing the drop. Is it because there has been more content added to the site, thereby “diluting” the completion page views? Less marketing and acquisition driving towards task completion? Or, perhaps tasks are seasonally driven? Like any good high level “health” report, the TCE will drive additional questions that require you to do some analysis and investigation.

As I mentioned in my first post, this metric becomes even more effective through segmentation of audiences by geolocation, subdomain, visit frequency, and acquisition source.

Metrics for Smarter Content Management

In your organization, what stakeholder group is most under-served by web analytics?

Certainly not marketers and product managers…they are an analysts’ best set of friends because the reporting can show real ROI.

The user experience designer or information architect? No, they get data that can really help with site optimization that brings visitors to the marketing and product focused site sections. 

I’m thinking it’s the content manager…the operations guy/gal whose job it is to manage the production, updating, archiving and retiring of content. Is there someone in this role where you work? Maybe they are a subject matter expert who has control over a particular site section. Maybe they manage a team of writers whose job it is to crank out the content. Maybe they are your intranet managers…a part of the site that generally flies beneath the radar when it comes to analytics. Maybe it’s the overall site manager. And maybe it’s you.

In much of my work with US Federal government agencies, financial services, pharmaceutical companies and non-profit organizations, the content managers and intranet managers tell me that they just don’t get metrics that help them determine when it’s time to clean up, spruce up or get rid of content that is old and in the way.  As one weary content manager told me, “The goal is to publish, publish, publish…not analyze whether anyone looks at the content or what they do with it.”

While I hear some content managers say with pride that they simply whack all the pages that don’t get page views, I don’t think this is the whole solution to the pruning problem.  I think it makes more sense to first get an understanding around relative site performance based on goals as a first step, and then take a more surgical approach that’s based on prioritization. This may indicate that certain areas of the site need more than just an elimination of pages and in fact may need a new content strategy. This seems like a more constructive approach.

Over the last few years, I’ve been working with clients who want to go this route.  I’ve developed six metrics that you can start using to evaluate site performance and direct your content management efforts.  I’ll be sharing them over the next six weeks in a series of posts that I’ll be publishing here. 

Let me say up front, that creating these metrics requires some prep work, like categorizing content pages vs. navigation pages. It's actually a good exercise...like doing an inventory. The goal: Do the analysis, create the metric and develop benchmarks that provide you with a good idea of overall site effectiveness that can be tracked and explained over time.

…But First a Word About Segmentation

While the metrics stand on their own, it makes sense to create a series of visitor/visit segments based on Semphonic’s two-tiered segmentation methodology. After an extensive review of site objectives, goals, top tasks and target audiences, you can create key segments. The metrics that you develop can be applied to these segments to understand whether each of these segments may use content that others don’t. This helps you understand immediately whether specific content types that do not get a high number of views, may in fact be content that is important to a highly valued segment. In the table below, we have some examples of segments developed for a site that reaches health care professionals, researchers and the general public.

As you can see from these categories, combining the known visitor data with site behavior gives you an opportunity to amplify meaning from the basic elements of data you’d be collecting.

OK…so categorizing pages and segmentation... two keys to making these metrics work. If you can do that over the next week, you’ll be on your way to applying the first metric I’ll be talking about next Tuesday…Task Completion Effectiveness. 

 

Has Digital Analytics Made it into the Digital Production Lifecycle?

Continuing on with Part 5 of this year's Profiles in Analytics - Does Web Governance Drive Analytics...or get the whole analysis, survey data and digital analytics profiles at Semphonic.

Has Digital Analytics Made it into the Digital Production Lifecycle?

For all of the aspirations on how to create digital cultures and data driven organizations, when it gets down to basics, simply making sure that data collection occurs is a back burner issue. While Web governance oriented organizations may have processes in place and top down commitment to ensure that analytics is part of the development cycle, that it is only at around 60 percent says that there is room for improvement.

While non-governance organizations have roughly the same compliance, that there is such a high level of dissatisfaction with the process of implementing code changes, tells you something about an underlying issue that can sabotage whether accurate data is available when senior leadership wants to use it for appraising the role of the digital channel.

The Details

It stands to reason that organizations with established Web governance have more formalized processes around production of new content and applications for the digital channel, than organizations that don’t have strong governance.

Web development process. Nearly twice the number of organizations with governance includes digital analytics in the process “all the time” versus those that don’t, as seen in the chart below. Still, at 25 percent, there is clearly more work that needs to be done in this area. Interestingly, once you combine “all” and “most” of the time responses, you get roughly 60 percent in all organizations.

The research shows that there is still a great deal of work that needs to be done in the area of integrating analytics to the digital development process. While there is an increasing amount of project management best practices aligned with enabling analytics, analytics itself is often implemented after a project has launched. With the advent of mobile applications and social media initiatives, there are new opportunities for using analytics, and at the same time a repeat of the poor practices that have left analytics out of the fixed Web development process.

Implementing data collection code changes. The research showed the biggest difference between governance and non-governance when dealing with the process for implementing changes to data collection code, as seen in the chart below.

Anecdotal research tell us that the application of data collection code, considered to be the least glamorous element of analytics, to be a universal and significant hold up in the analytics process.  Because there are often other projects in the queue ahead of analytics, or analytics is considered expendable if content or applications need to meet a deadline for launch, data collection coding may be held up for weeks and even months in certain organizations.  

The “not acceptable” level of satisfaction among organizations without governance is a situation that if addressed, could show immediate improvements to the availability of analytics and potential value of metrics created with the data.

Next week: So, will Web governance make your digital analytics program run faster, jump higher and make you smarter?

Get the entire research at Semphonic.

Are Web Governance Oriented Organizations “Better” at Using Digital Analytics?

Continuing on with Part 4 of this year's Profiles in Analytics - Does Web Governance Drive Analytics...or get the whole analysis, survey data and digital analytics profiles at Semphonic.

Are Web Governance Oriented Organizations "Better" at Using Digital Analytics?

Enterprise Web governance may contribute to high quality digital analytics for the enterprise, but it does not guarantee success. At the same time, lack of enterprise Web governance doesn’t necessarily doom an analytics initiative to failure. However, programs seem to stop short on delivering all that they can deliver. 

In both cases, analytics programs appear to be primarily centralized entities that focus on dashboard development, reporting, providing analysis and recommendations.  While there is service to multiple stakeholders, the lack of time dedicated to training, encouraging self service and creating analytics standards that may be used throughout the organization fosters a somewhat isolated digital analytics culture. So, while multiple stakeholders are potentially served through distribution of reports, the value of what’s being conveyed in the reports may or may not be understood.

The Details

If you serve more stakeholders more efficiently does this mean that you are providing a high quality analytics product? Perhaps, but not necessarily. This is like saying that a Super Size meal is better than a 4 oz filet mignon. Quantity doesn’t mean quality.

To determine the level of quality, the research focused on three elements:

• Digital analytics program priorities of digital analytics managers
• Whether the program priorities were in line with senior management
• Priority tasks of the digital analytics teams

Top priorities.  In looking at alignment in priorities between digital analytics managers and their management, it appears there is actually more alignment among those in organizations without Web governance councils.

 

 

It is altogether positive that there is fairly strong agreement between digital analytics managers and senior management that the primary mission is delivering recommendations and analysis, even if it is not completely equivalent.

There is a fairly strong sentiment among senior leadership around the importance of developing dashboards and reports in organizations that have Web governance. This suggests that there are a number of senior leaders who believe that success in analytics is based on getting reports into as many peoples’ hands as possible. 

But there is more to this story…

Top tasks. When asked about the tasks the analytics team spends the most time on, there is an interesting disconnect from the goals and perceptions described previously…the prominence of implementation and maintenance of the analytics tools, as seen below.

 

Neither Web analytics managers nor senior leadership wants to consider that this is a top priority. However, lack of recognition of the importance of this task is akin to driving a car and not wanting to get it serviced regularly. Negligence can catch up in a number of ways. In the scenario of mutual denial, analytics teams may not have sufficient resources to provide more valuable insight from analytics.  However, the question is whether they make the business case to request these resources. Conversely, senior leadership may not be aware of the amount of time and effort required to maintain and improve the Web analytics implementation.

Finishing among the top three tasks analytics teams spend their time on, and what also doesn’t show up as a priority is “conducting ad hoc analysis.”

Like implementation and maintenance tasks, there would be agreement that this would not be a priority; however, because so much time is spent on this task, it takes away from potentially other more valuable tasks. It is worth noting that organizations without Web governance indicated that their team spends more time on this than any other task. However, this activity actually occurs more often as a second and third task than non-Web governance organizations.  Reasons for this can range from positive to negative: a focus on conducting deep dive analyses, “drive by” requests, weak or non-existent request management and evaluation process, or a need to improve quality and distribution of dashboards.

Regardless of whether the organization is driven by Web governance or no not, the tasks that could potentially drive high value throughout the enterprise are given the lowest priority among both Web analytics teams and senior leadership. These tasks also get the least amount of analytics team attention:

• Train and educate on the use of analytics
• Provide strategic direction on the use of analytics
• Create organizational analytics standards
• Developing culture of self-service analytics

Next week: Has Digital Analytics Made It into the Mainstream of the Digital Production Lifecycle?

Get the entire research at Semphonic.

Improving Federal Websites and Measuring Effectiveness of Your Customer Service Plan

How far along is your agency in implementing the Customer Service Plan mandated in President Obama’s Executive Order 13571 – Streamlining Service Delivery and Improving Customer Service?

How are you addressing the activities outlined in the Appendix of Memo M-11-24 – Implementing Executive Order 13571? The Appendix provides the roadmap for how to create the Customer Service Plan with specific focus on:

• The need for establishing ways to receive customer feedback
• Developing a process for evaluating the entire customer experience
• Analyzing customer preferences for interactions from less preferred and more costly channels (print) to preferred, less costly channels (online, mobile)
• Reporting progress in meeting customer service standards

If you have an existing measurement strategy in place, you should be able to create the metrics to evaluate and analyze the services being addressed through your existing set of tools and methods.

However, what if you don’t have an existing measurement strategy in place, or tools to support a measurement strategy?

You can potentially develop something quickly to meet the Plan delivery deadline by the end of October. You can then develop a cohesive measurement strategy that takes a longer term perspective. This is something to think about if you want to use data to drive ongoing decisions about customer service, eliminate outdated and duplicative web sites and derive operational efficiencies on web site development…all objectives that the Executive Order is encouraging agencies to act on.

Short Term Measurement Plan

So what does a short term measurement plan look like? Let’s look at a minimal and inexpensive tool set that will measure three key aspects of online customer service that is addressed by the Plan.

 1. Web Analytics – Tool: Google Analytics – free; easy to implement, easy to derive value

To use Google Analytics, you need to be in compliance with OMB Memo M-10-22 especially on being clear about opt-out options on web measurement.  See my blog post from last year for what you need to do to be in compliance:

Uses: For measuring the number of visitors using the site, the number of starts and completion (such as for a form, or multi-step process), geographic location of visitors, where visitors originate.  Segment these basic metrics by first time and repeat visitors, domain name, content interest (professional, consumer, and researcher).

Sample Metrics

Task Completion Effectiveness: Indicates visitor interest in and successful completion of tasks

Definition: Measures completion of site tasks associated with NIH or IC mission.

Calculation: Expressed as a percentage based on the total number of task completion page views divided by total number of site page views.  A higher percentage indicates more successful task completion.

Return on Investment – Provides quantifiable cost/benefit analysis of budget and time expended on Web site operations.

Definition: Measures return on investment in web site operations, content, applications, and labor in context with budget spent on these initiatives.

Calculation: Task Completion ROI: Number of Task Completions multiplied by $ value given to each Task Completion type compared to total expenditure on creation of Task application or in Avoided Cost alternative to manual labor.

2. Post-Transaction Customer Survey  - Tool: SurveyMonkey, Zoomerang – inexpensive; easy to implement, easy to derive value

Uses: For understanding whether the service altered behavior after use, obtaining feedback on how to improve the service, as well as why service was improvement over offline alternatives.

3. Customer Satisfaction Survey – Tool: QuestionPro – inexpensive; easy to implement, easy to derive value

Uses: Provide insight into whether visitors found what they were seeking and whether they were satisfied with the quality of the content they viewed or the transaction that was completed. 

To use surveys, and a host of other voluntary collection mechanisms, you’ll need to participate in the new Fast Track Process for Collecting Service Delivery Feedback under the Paperwork Reduction Act. Your agency is likely participating in this initiative already. You can refer to OMB Memo M-11-26 for the details of the process and http://www.howto.gov/customer-service for supporting information.

Whether you have an existing web measurement strategy or need to come up with a short term plan, the key to getting value from the data that has been collected is to analyze each set in context with its purpose in evaluating the service. The data should be complementary.

In my next post, I’ll discuss how to “triangulate” data analysis from the different measurement methods at your disposal.

Web Analytics KPIs = BS

I’m starting to prepare for the tutorial I’m giving at the JBoye Conference in Philadelphia on May 3. The class is called Beyond Visits and Page Views…How to Develop Actionable Web Metrics, Reports and Analysis.

One of the areas that I’ll be going into in some depth is how you can build metrics that drive decision making. This would seem to be a clear goal for any Web site measurement initiative, but it is a huge struggle for many organizations. There are multiple reasons, but I think there is one in particular that causes considerable distraction from meeting this goal:

The popularized understanding of Web Analytics KPIs has led us down the wrong path.

Key Performance Indicators (KPIs) are a term that you hear a lot in Web Analytics practice. KPIs are commonly considered to be the metrics that you use for measuring your Web site with Web Analytics. For many, visits, visitors, page views, referrals trended month over month and year over year are the Web Analytics KPIs. This is not what KPIs are about; in fact KPIs are only a part of an equation that creates a complete and useful measurement.

Let’s get some historical perspective. In the 1960s, D. Ronald Daniel of McKinsey and Company came up with a methodology to help organizations define and measure their effectiveness against business objectives. In the 1980s, this methodology was clarified and evangelized by John F. Rockart of the Sloan School of Management. The result was a hierarchal system of evaluation that was based on:

• Definition of organizational mission
  • Identification of strategic goals that map to the mission
    • Determination of concrete and measurable objectives
      • Definition of Critical Success Factors (CSFs) that are used to evaluate the objectives
        • Developing a KPI measurement and target that determine whether the CSF is being met

In Web Analytics, the model could look something like this:

• Organizational mission – Be the industry leader
  • Strategic goal – Provide the most reliable and authoritative information
    • Objective – Publish a weekly email newsletter
      • Critical Success Factor (CSF) – Increase number of subscribers to the weekly newsletter
        • KPI- # of newsletter subscribers per month
          • Target – increase # of newsletter subscribers by 10% each month

The idea here is that you can have a number of CSFs per objective, and a few KPIs and Targets associated with each CSF.

As you can see, KPIs are only one part of a bigger picture and you can do a lot more to make your current Web Analytics numbers relevant, compelling and useful. It doesn’t necessarily have to take a lot of work or sophisticated tools…it just takes a different and more contextual way to look at the numbers.

So, I hope you’ll join me in Philadelphia for the tutorial. I’ll show you how you can use the real KPI methodology to improve your metrics and talk about other techniques that will really help you get value from Web Analytics.  I also suggest you consider the rest of the JBoye Conference. As always there is a great program and I’m pleased to see a big emphasis on Web Analytics this year, as well as online strategy, user experience, content management and lots more.

One last note: There's still time to apply for the X Change Web Analytics Challenge to Benefit Non-Profits...deadline is April 30.