Web Analytics Management

Metrics for Smart Content Management - Metric #1 - Task Completion Effectiveness (TCE)

Management of content proliferation is important of course, but in evaluating content effectiveness it is also important to consider the goals of the site. All too often we hear that “reading content” is a site goal. While that may be true if you’re selling advertising, it is generally not true for content rich sites. (For more on this, you’ll want to read next week’s post on Mission Critical Content)The purpose of content is to drive an action or a task, such as a purchase, event registration, donation, newsletter sign up, lead generation form completion, using a selection wizard and so on.

Measuring Task Completion gives you an initial sense on whether the web site is effective in getting visitors to complete the tasks on your site that are associated with business goals. Once you establish Task Completion Effectiveness, you can use this as a benchmark to determine whether you content management actions are working. The goal: raise the site’s Task Completion Effectiveness percentage.

Task Completion Effectiveness (TCE): Measures completion of site tasks associated with your goals and mission

Calculation: Expressed as a percentage based on the total number of task-completion page views divided by total number of site page views. A higher percentage indicates more successful task completion.

Calculation Notes: The completion page for each task will need to be identified, such as a “thank you” page or a submission confirmation.

Example: During the month, there were 500,000 total page views. Of this total number, there were 10,000 task-completion page views. This equals a task-completion effectiveness of 2%.

In the chart below, you’ll notice that we’re using “visits” and not “page views” as a baseline metric. Why? This report was done in Google Analytics. GA bases its metrics on visits and can not correlate events to page views. While we could have created filters to obtain the page view data, we opted to go with what GA does best. Using visits as an alternative still provides a reliable measure of task completion effectiveness. If we had done this in SiteCatalyst, we would have been able to correlate the events to page views.

 

 

In this example, the TCE has gone down slightly over a 3 month trend. Next steps should include reviewing the page view totals of the task confirmation pages to see which ones are experiencing declines, and then determining what factors are influencing the drop. Is it because there has been more content added to the site, thereby “diluting” the completion page views? Less marketing and acquisition driving towards task completion? Or, perhaps tasks are seasonally driven? Like any good high level “health” report, the TCE will drive additional questions that require you to do some analysis and investigation.

As I mentioned in my first post, this metric becomes even more effective through segmentation of audiences by geolocation, subdomain, visit frequency, and acquisition source.

Son of Nedstat: comScore’s Digital Analytix vs. SiteCatalyst, Google Analytics, Webtrends et al

I started following Nedstat when I wrote the first CMSWatch Web Analytics Report in 2007. Based in Amsterdam, the company never got much attention outside of Europe, even though it was the largest analytics vendor outside of North America. Over the years, the company built a loyal following of clients, driven by excellent, localized service, fair pricing and an interest in providing innovative features that were introduced either before or at the same time as its North American counterparts, such as video analytics, third party integration and an emphasis on providing “on the fly” segmentation.  

I thought Nedstat’s purchase by comScore last summer would make for a potentially interesting option in a market that was starting to witness a lack of choice in enterprise analytics solutions. I also liked the idea that an audience measurement company was getting into analytics and could understand the complementary nature of behavioral analytics to its core business. From an acquisition perspective, it seemed to make sense.

So, with this as a background, Semphonic was invited to attend a 3 day partner training at comScore’s Reston, VA headquarters to dive deep into the tool. My colleagues, Chris Meares, Ryan Praskievicz and Royce Fung joined me, and I asked them to share their takeaways on the “son of Nedstat” that is now comScore Digital Analytix (DA). With that being said, I was really impressed by the comScore’s team interest in trainee feedback, and commitment to acting on a range of recommendations to further improve the offering.  This is a long post, but I think you’ll find it a very thorough analysis of where DA stands today.  We’ll hear from Chris and Royce today, and Ryan later this week.  I think Chris sums up the general feeling in the class in his opening…

 

From Chris Meares, Senior Consultant, @chris_meares:

The first thing everyone asks me is “How does it compare to Omniture (Adobe) or Google Analytics?” What I am going to tell you is how it doesn’t compare to Omniture or Google Analytics and it can be summed up in three words, un-aggregated, segmentation and integration.  Obviously, the tools mentioned previously do all, or some, of these items also, but none do it as well as DA.  Here is a brief overview of each term and how DA differs from its competitors.

Unaggregated data

Digital Analytix (DA) stores all of the data it receives from your site in un-aggregated form which means you have the ability to slice and dice it anyway you like and you are not held to the constraints of how it is collected as is with other tools.  It also means there is no data sampling which is seen daily in Google Analytics and occasionally in Omniture Discover.  Omniture does have a data warehousing feature but the lag time on running reports using data warehousing is immense whereas DA allows you to run queries in real time with impressive speed.  Of course, DA doesn’t have a client base in the US yet and it will be interesting to see how the lag time increases (or not) once they begin collecting enormous amounts of data.   The other advantage of un-aggregated data comes into play when doing attribution to campaigns.  DA allows you to set up different attribution models on the same sets of data at the same time such as first click, last click, linear increasing/decreasing, equal share to all, etc., in order to see the differences in attribution models in real time.  This is something that cannot be done in other web analytics solutions.

Segmentation 

Digital Analytix (DA) has the best segmentation capabilities I have seen in a web analytics solution.  You can segment, in real time, on any of your data points and create reports and do deep dive analysis.  Also, you can have as many custom variables as you see fit, you are not set to a specific number, either per page or per site.  Because it is included “out of the box” with the DA package and because it can segment in real time, it has an advantage over both previously mentioned competitors.  It is one of the most flexible segmentation tools in the market that I have seen.

Integration

The last area that sets Digital Analytix (DA) apart at the moment is the integration feature that is included with the solution.   DA can integrate with your email service provider, your voice of customer tool, your multivariate testing tool and Google Adsense.  It also has the ability to integrate directly with your backend database either via Data Lookup (a SQL database hosted by DA) or through data merge (data that is uploaded nightly), which allows you to easily combine offline and online data. 

Segmentation is available on all of the integrated pieces because it all part of one interface.  It will also integrate with comScore’s audience measurement business which will allow clients to see demographic data based directly in their analytics solution.  This seems to be a huge hook for DA but was not actually shown during the training so I am not quite sure how it will exactly work. 

Final Takeaway from Chris

There are a few concerns with Digital Analytix.  The user interface is slightly clunky and is a work in progress.  It currently feels like it is more focused on the power user or full time analyst. Occasional users may find it difficult to get the reporting they need without some heavy training.  Like any enterprise level implementation, it will take extensive time with the client to figure out business needs in order to set out a road map that will allow correct data capture.   If DA can upgrade their user interface to the level of Webtrends or even Site Catalyst then I believe it can be a major player in the marketplace.

 

From Royce Fung, Consultant, rfung@semphonic.com:

While not necessarily groundbreaking as a whole, the tool is clearly catered towards those willing to invest time and resources into their web analytics tool and those who seek to drive their web strategy from a deeper understanding of user behavior on their website.

DA is striving for the high end of enterprise web analytics tools and is unabashed at saying it aims to wrest away Omniture’s dominance at the top end of the spectrum.  After seeing the tool and getting some insight into its development path, I believe, in time, it has more than a chance at doing this.

Reporting

The browser-based tool provides a range of metrics on par with the other top tier analytics tools.  Report Builder allows for building and saving customized reports.  There is also an easily accessible way to create custom, calculated metrics based on a mathematical formula.  Definitions tend to be very precise in DA.  This forces users to select between and, therefore, recognize nuances such as something that takes place within a visit-based scope or an event-based scope.  Ultimately this is a good thing -- it certainly raises the bar when it comes to the web metrics reporting – though I can envision it being initially frustrating to many.

Reporting Interface

The Reporting Interface is, all-around perhaps, one of the biggest drawbacks of the tool.  In order for reports to be displayed, they must be put into what most users might think of as a dashboard.  The interface tends to require users to work within sub-menus, making the overall experience somewhat tedious.  For example, to add a report to the Report display, users enters a rather small dedicated interface that requires a bit of navigation before performing even the simplest report.  From the Report display, several clicks are required to produce a simple visits report, even when using the default parameters, compared to just one click in SiteCatalyst.  Editing of reports suffers from the same type of syndrome.

DirectView

DirectView is a useful tool comparable to Omniture’s ClickMap.  Some of the common limitations apply, such as the reporting of flash-based elements as well as separating multiple identical links on a page.

Live Segmentation

Segmentation is very robust and can be applied to all or individually to reports.  Segments can also be applied in DirectView, which is a really nice feature (and a common theme of how well everything is tied together and integrated), though, we did not see this demonstration.  The ability to apply segments live at a top level to the Reports display, to individual reports, or to individual metrics is very impressive. 

Again, there are some shortfalls with the interface, but overall it is a comparatively strong feature.

Data Integration

A lot of thought has clearly been placed on integrating data sources.  And coming from comScore, owners of a treasure trove of data, it isn’t all that surprising.  DA has placed heavy emphasis on allowing clients to integrate data from other sources whether they come from comScore’s other range of products, third party sources or the clients own internal data sources.

By leveraging its RING integration clients can for example, not just track the success of an email campaign, but segment that population in DA to create a specific target audience for a follow-up campaign.  The integration with the email provider allows for specifying that target audience in the form of an email list.

Campaigns

On the subject of campaigns, comScore’s expertise in market intelligence and the media industry clearly shows through in the DA product.  Much thought has been given to areas such as campaigns and the concept of engagement, and DA clients can make use of this, particularly with the eCommerce add-on module.  How conversions can be applied to campaigns can be adjusted very precisely (again, scope comes into play, as well as other definition rules).  Clients can also assign the weighting of content contributing to engagement on the site, by making use of an engagement score which can be assigned to individual pages. As of now, point values must be tagged (that is, they cannot be set up in an interface), but this is set to be addressed by the end of the year.

Video Tracking

Video tracking can be finely tuned with the StreamSense add-on module, though standard DA seems to have much of the basic milestone-based measurements that I suspect many would find sufficient interaction with a video player can be tracked with StreamSense.

Final Takeaway from Royce

Overall, the tool is built with precision and deep measuring capability in mind.  It also attempts to get the most out of its data integration capabilities and from that stand point succeeds in not simply being a web analyst tool but a broader business tool.  Having an existing client base, many with the UDM tag already in place, will be a huge plus initially for comScore in getting a number of customers on board to DA.

 

 

Web Analytics KPIs = BS

I’m starting to prepare for the tutorial I’m giving at the JBoye Conference in Philadelphia on May 3. The class is called Beyond Visits and Page Views…How to Develop Actionable Web Metrics, Reports and Analysis.

One of the areas that I’ll be going into in some depth is how you can build metrics that drive decision making. This would seem to be a clear goal for any Web site measurement initiative, but it is a huge struggle for many organizations. There are multiple reasons, but I think there is one in particular that causes considerable distraction from meeting this goal:

The popularized understanding of Web Analytics KPIs has led us down the wrong path.

Key Performance Indicators (KPIs) are a term that you hear a lot in Web Analytics practice. KPIs are commonly considered to be the metrics that you use for measuring your Web site with Web Analytics. For many, visits, visitors, page views, referrals trended month over month and year over year are the Web Analytics KPIs. This is not what KPIs are about; in fact KPIs are only a part of an equation that creates a complete and useful measurement.

Let’s get some historical perspective. In the 1960s, D. Ronald Daniel of McKinsey and Company came up with a methodology to help organizations define and measure their effectiveness against business objectives. In the 1980s, this methodology was clarified and evangelized by John F. Rockart of the Sloan School of Management. The result was a hierarchal system of evaluation that was based on:

• Definition of organizational mission
  • Identification of strategic goals that map to the mission
    • Determination of concrete and measurable objectives
      • Definition of Critical Success Factors (CSFs) that are used to evaluate the objectives
        • Developing a KPI measurement and target that determine whether the CSF is being met

In Web Analytics, the model could look something like this:

• Organizational mission – Be the industry leader
  • Strategic goal – Provide the most reliable and authoritative information
    • Objective – Publish a weekly email newsletter
      • Critical Success Factor (CSF) – Increase number of subscribers to the weekly newsletter
        • KPI- # of newsletter subscribers per month
          • Target – increase # of newsletter subscribers by 10% each month

The idea here is that you can have a number of CSFs per objective, and a few KPIs and Targets associated with each CSF.

As you can see, KPIs are only one part of a bigger picture and you can do a lot more to make your current Web Analytics numbers relevant, compelling and useful. It doesn’t necessarily have to take a lot of work or sophisticated tools…it just takes a different and more contextual way to look at the numbers.

So, I hope you’ll join me in Philadelphia for the tutorial. I’ll show you how you can use the real KPI methodology to improve your metrics and talk about other techniques that will really help you get value from Web Analytics.  I also suggest you consider the rest of the JBoye Conference. As always there is a great program and I’m pleased to see a big emphasis on Web Analytics this year, as well as online strategy, user experience, content management and lots more.

One last note: There's still time to apply for the X Change Web Analytics Challenge to Benefit Non-Profits...deadline is April 30.

An “Under the Hood” Look at SiteCatalyst 15

On the fly segmentation, additional variables and custom events, display of multiple metrics on a single graph and presentation of unique visitors, visits and page views on all reports are the upgrades that got most folks excited about last week’s preview of SiteCatalyst 15 at the Omniture Summit. My colleague Gary Angel has a nice summary of these features in his recent post.

I thought one of the more interesting upgrades is about what you don’t readily see…how data in SiteCatalyst is processed. In the new version, you will be able to have what appears to be significant more control over how your data is processed as it goes into SiteCatalyst than you do currently, and will likely need to depend less on s_code customization and VISTA rule implementation to achieve this flexibility. This data processing rules control will be available from the Admin interface. What this means in “on the ground” terms is still a bit murky based on the conversations I’ve had with Omniture team, but I like the fact that this is being introduced. It’s definitely a step in the right direction.

SiteCatalyst Data Processing Today

Today, to really control how data is processed in SiteCatalyst, you need to either do a lot of customization on the s_code, as well as pay for expensive VISTA rules. If you’re a SiteCatalyst customer you know that there are roadblock issues associated with both methods.

For s_code modifications, not only do you need to unit test your code, but if you’re in a large enterprise or one that has established IT change control processes, you may need to have the new code go through additional QA as it moves through test and production environments. If you live in this world, you know that this can often take a lot of time and impact your ability to collect data in a timely manner.

VISTA rules allow you to populate multiple variables based on a keyed value in another variable. Internal IP filtering, for example, is a good candidate for a separate VISTA rule. Sites with complicated page hierarchy or intense usage of URL parameters often will use VISTA rules to decipher information in the page URL and populate more user-friendly or analytically useful variables. VISTA can also be used for visitor segmentation purposes, populating visitor information based on source or page URL. Not only are VISTA rules expensive, but you need to contact your Account Manager to set them up.

First Look at SiteCatalyst Admin Based Processing

On first look, the new Admin based processing control will mitigate reliance on both VISTA rules and s_code customization. It isn’t completely clear at this point how much, but it is certainly a step in the right direction for enabling easier implementation and maintenance. This addresses a key sore spot in SiteCatalyst and has certainly been a contributing factor for some organizations to just give up on using it.  

If you are a bit concerned about having so much control over your data processing at basically the flip of a switch, you should. If you implement a rule incorrectly you could seriously and irrevocably compromise your data collection and accuracy.

Omniture has considered this as well. Prior to allowing any clients to use the data processing rules dashboard, you will need to be certified by Omniture as an administrator. This will be a new certification. Sure, it’s another money making opportunity for Omniture, but I think this training and certification makes sense. 

I think there’s a bit of irony here as well if you have an historical perspective on web analytics. In the “olden” days of enterprise log file analyzers, you really needed a systems administrator to manage the tool for tasks such as log file importing, stitching, and processing (and of course, those that use systems like Webtrends, Urchin and Unica still do). As web analytics migrated away from IT and into Web and online marketing teams (due in large part through analytics vendor changes in sales direction) there has been an emphasis on not having IT involved. Now, we have another example of analytics coming back to having a resource involved who understands how data works and what occurs under the hood with data. Perhaps it isn’t someone in IT, but it must certainly be someone who is more in the weeds than a typical online marketing or web content manager or even a web analyst.

This will be something new to those who have relied on a software as a service model for web analytics. I think it speaks to the need to view digital analytics as a program with its own leadership, governance, and process. If you’d like to read more about how to develop this in your organization, I invite you to take a look at my white paper on managing digital analytics.

Related Posts:

Omniture Summit

Is Omniture Stepping into a Privacy Morass with Demdex Acquisition?

Getting a Grip on Digital Analytics

P.S - I'll be giving a tutorial on developing business-focused web metrics at the JBoye 11 conference in Philadelphia on Tuesday, May 3 and a conference presentation about the work we've done with The Nature Conservancy. Hope to see you there.

P.P.S - If you're a non-profit, I invite you to take the X Change Web Analytics Challenge.

Is Omniture Stepping into a Privacy Morass with Demdex Acquisition?

The opening session at an Omniture Summit is a mix of pep rally, vision and celebration. Today’s event did not disappoint.  To start the show, VP and GM of Omniture, Brad Rencher, tossed out some interesting stats about the audience.  The one that struck me was that 40% of total online ad spend was represented by the companies in the room. That’s a pretty big number. So, it was not surprising that a big part of the “vision” aspect of the session was all about online advertising and the potential value offered to online advertisers by the January, 2011 acquisition of Demdex.

Online advertising is where the money is…$109 billion by 2013 according to IDC… and Adobe Omniture is placing itself squarely in the middle of trying to enable both advertisers and publishers do a better job of targeting ads. Demdex is a platform that allows you to aggregate all information you can get your hands on about online visitors. It’s like a CRM system, but allows you to put data in from external sources, such as Demdex’s data partners Acxiom, Datalogix, Exelate and TargusInfo.  It’s a pretty powerful concept, no doubt. Vivek Shah, CEO of Ziff Davis, went into a surprisingly detailed description of how they’ve created BuyerBase, a targeted platform focused on 40 million tech buyers, with the Demdex technology.

Well, as one of the opening session speakers quipped, “the Omniture Summit is like Woodstock for geeks”…Or something to that effect. So if all of this talk about how to use data for online advertising gets folks feeling as heady as the contraband passed around at Woodstock, is there also a parallel to the “brown acid” warning in the form of privacy?  Demdex seems like just the type of company that the FTC was talking about in its Privacy Report.

Demdex claims that it is a good citizen when it comes to providing consumers with a view into all the data that is collected about them and be able to opt out. This is accomplished through the Digital Advertising Alliance (DAA) opt out program.  According to Politico, FTC Chairman Jon Leibowitz was questioning this effort, "We commend industry's effort to get a broad group of industry leaders on board. However, the effectiveness of this effort will depend on how, and the extent to which, the opt-out is actually implemented and enforced - all of which is yet to be seen. We also urge industry to make sure that the opt-out is easy for consumers to find, use, and understand."

If visitors don’t accept third party cookies, they won’t be tracked by Demdex.  However, the issue for Adobe Omniture is to both make sure they provide clear and effective options into opt out so it provides Demdex customers with both powerful marketing and something that isn’t going to get regulated out of existence. Maybe this will be covered in next year’s Summit opening session…probably not as exciting as today, but certainly something that will make all potentially interested purchasers be able to rest more easily.