Web Analytics Management

Are Web Governance Oriented Organizations “Better” at Using Digital Analytics?

Continuing on with Part 4 of this year's Profiles in Analytics - Does Web Governance Drive Analytics...or get the whole analysis, survey data and digital analytics profiles at Semphonic.

Are Web Governance Oriented Organizations "Better" at Using Digital Analytics?

Enterprise Web governance may contribute to high quality digital analytics for the enterprise, but it does not guarantee success. At the same time, lack of enterprise Web governance doesn’t necessarily doom an analytics initiative to failure. However, programs seem to stop short on delivering all that they can deliver. 

In both cases, analytics programs appear to be primarily centralized entities that focus on dashboard development, reporting, providing analysis and recommendations.  While there is service to multiple stakeholders, the lack of time dedicated to training, encouraging self service and creating analytics standards that may be used throughout the organization fosters a somewhat isolated digital analytics culture. So, while multiple stakeholders are potentially served through distribution of reports, the value of what’s being conveyed in the reports may or may not be understood.

The Details

If you serve more stakeholders more efficiently does this mean that you are providing a high quality analytics product? Perhaps, but not necessarily. This is like saying that a Super Size meal is better than a 4 oz filet mignon. Quantity doesn’t mean quality.

To determine the level of quality, the research focused on three elements:

• Digital analytics program priorities of digital analytics managers
• Whether the program priorities were in line with senior management
• Priority tasks of the digital analytics teams

Top priorities.  In looking at alignment in priorities between digital analytics managers and their management, it appears there is actually more alignment among those in organizations without Web governance councils.

 

 

It is altogether positive that there is fairly strong agreement between digital analytics managers and senior management that the primary mission is delivering recommendations and analysis, even if it is not completely equivalent.

There is a fairly strong sentiment among senior leadership around the importance of developing dashboards and reports in organizations that have Web governance. This suggests that there are a number of senior leaders who believe that success in analytics is based on getting reports into as many peoples’ hands as possible. 

But there is more to this story…

Top tasks. When asked about the tasks the analytics team spends the most time on, there is an interesting disconnect from the goals and perceptions described previously…the prominence of implementation and maintenance of the analytics tools, as seen below.

 

Neither Web analytics managers nor senior leadership wants to consider that this is a top priority. However, lack of recognition of the importance of this task is akin to driving a car and not wanting to get it serviced regularly. Negligence can catch up in a number of ways. In the scenario of mutual denial, analytics teams may not have sufficient resources to provide more valuable insight from analytics.  However, the question is whether they make the business case to request these resources. Conversely, senior leadership may not be aware of the amount of time and effort required to maintain and improve the Web analytics implementation.

Finishing among the top three tasks analytics teams spend their time on, and what also doesn’t show up as a priority is “conducting ad hoc analysis.”

Like implementation and maintenance tasks, there would be agreement that this would not be a priority; however, because so much time is spent on this task, it takes away from potentially other more valuable tasks. It is worth noting that organizations without Web governance indicated that their team spends more time on this than any other task. However, this activity actually occurs more often as a second and third task than non-Web governance organizations.  Reasons for this can range from positive to negative: a focus on conducting deep dive analyses, “drive by” requests, weak or non-existent request management and evaluation process, or a need to improve quality and distribution of dashboards.

Regardless of whether the organization is driven by Web governance or no not, the tasks that could potentially drive high value throughout the enterprise are given the lowest priority among both Web analytics teams and senior leadership. These tasks also get the least amount of analytics team attention:

• Train and educate on the use of analytics
• Provide strategic direction on the use of analytics
• Create organizational analytics standards
• Developing culture of self-service analytics

Next week: Has Digital Analytics Made It into the Mainstream of the Digital Production Lifecycle?

Get the entire research at Semphonic.

Does Web Governance Encourage More Efficient Operation of Digital Analytics?

Continuing on with Part 3 of this year's Profiles in Analytics - Does Web Governance Drive Analytics...or get the whole analysis, survey data and digital analytics profiles at Semphonic.

Does Web Governance Encourage More Efficient Operation of Digital Analytics?

Digital analytics teams in organizations with Web governance appear to be more efficient in serving a larger stakeholder base than those that are in organizations without Web governance. While staff size appears to be somewhat smaller in Web governance oriented organizations, there appears to be slightly more management expertise.  The experience of digital analytics team management coupled with overall organization Web governance could contribute to a more process oriented approach to serving stakeholders. This would enable more stakeholders to benefit from the use of digital analytics services.

In addition, stronger management of digital analytics teams could also lead to a tighter definition of roles and responsibilities within the team. This could foster greater specialization among team members that could be manifest in more efficient completion of tasks.

The Details

Digital analytics teams in Web governance-oriented enterprises appear to serve more stakeholders than those organizations without Web governance.  As indicated in the chart below, digital analytics teams in organizations with governance appear to serve more stakeholders in all categories except one.

Of course, some of the difference in stakeholders served can be attributed to organization size…larger organizations have more stakeholders. However, the research suggests that the influence of Web governance on staff size and management experience also plays a role in how many stakeholders are served.

Analytics staff size. Organizations with Web governance had twice the percentage of large staffs with 10 or more people. However, these enterprises actually had smaller staffs than did organizations without Web governance in the mid-range (2-10 staff size).

Digital analytics management experience. From a title perspective, managing a digital analytics team is largely a middle management function, with Director and Manager the most frequently indicated titles.

However, it is in the area of both analytics and management experience where there are potentially significant differences between teams in organizations with and without Web governance.

Organizations with Web governance tend to have managers with more management experience than analytics experience. Organizations without Web governance tend to have managers with more analytics experience than management experience.  A significant plurality of organizations, 30 percent, has managers with both ten and more years of analytics and management experience.

This overweighting of technical and management experience appears logical in context with the characteristics of the types of digital analytics teams one finds in large and Web governance-oriented organizations vs. those that are smaller and not inclined to Web governance. Managers in large organizations appear to be more involved with  strategic and operational functions; in smaller organizations with less staff, managers are in a more practitioner/manager role.

Is there an approach that works better? If there is an organizational strategy to scale analytics across the enterprise, than having more experienced managers who know how to manage programs and staff and are comfortable interfacing with senior leadership makes sense.

 

Next week: Are Web Governance Oriented Organizations "Better" at Using Digital Analytics?

Get the entire research at Semphonic.

Does Web Governance Drive Digital Analytics?

Continuing on with Part 2 of this year's Profiles in Analytics - Does Web Governance Drive Analytics...or get the whole analysis, survey data and digital analytics profiles at Semphonic.

Larger Organizations Invest in Web Governance

By their nature, large organizations recognize the need to invest in operations governance as a means to manage processes and people efficiently.  Web governance is a natural extension of a governance culture that already exists.  Smaller organizations tend to have less process and a flatter management structure. The concept of Web governance may run counter to how the organization approaches overall operational processes. In addition, as smaller organizations tend to run leaner from a staff perspective, they may not have the resources to dedicate specifically to Web governance.

The Details

There is a relationship between investment in Web governance and an organization’s size and annual revenues. Larger organizations tend to be more inclined to have Web governance councils than smaller organizations, as seen in the charts below.

51 percent of the organizations with over $501 million in annual revenue have Web governance councils compared to 17 percent that do not. On the other end of the spectrum, 77 percent of the organizations with less than $500 million in revenue do not have Web governance councils compared to 24 percent that do.

The trend towards Web governance is also borne out based on number of employees. 66 percent of the organizations with over 1,000 employees have Web governance councils compared to 25 percent that do not and 44 percent of the organizations with 100 or fewer employees do not have Web governance compared to 11 percent that do.

 

Next week: Does Web Governance Encourage More Efficient Operation of Digital Analytics?

Get the entire research at Semphonic.

 

Improving Federal Websites and Measuring Effectiveness of Your Customer Service Plan

How far along is your agency in implementing the Customer Service Plan mandated in President Obama’s Executive Order 13571 – Streamlining Service Delivery and Improving Customer Service?

How are you addressing the activities outlined in the Appendix of Memo M-11-24 – Implementing Executive Order 13571? The Appendix provides the roadmap for how to create the Customer Service Plan with specific focus on:

• The need for establishing ways to receive customer feedback
• Developing a process for evaluating the entire customer experience
• Analyzing customer preferences for interactions from less preferred and more costly channels (print) to preferred, less costly channels (online, mobile)
• Reporting progress in meeting customer service standards

If you have an existing measurement strategy in place, you should be able to create the metrics to evaluate and analyze the services being addressed through your existing set of tools and methods.

However, what if you don’t have an existing measurement strategy in place, or tools to support a measurement strategy?

You can potentially develop something quickly to meet the Plan delivery deadline by the end of October. You can then develop a cohesive measurement strategy that takes a longer term perspective. This is something to think about if you want to use data to drive ongoing decisions about customer service, eliminate outdated and duplicative web sites and derive operational efficiencies on web site development…all objectives that the Executive Order is encouraging agencies to act on.

Short Term Measurement Plan

So what does a short term measurement plan look like? Let’s look at a minimal and inexpensive tool set that will measure three key aspects of online customer service that is addressed by the Plan.

 1. Web Analytics – Tool: Google Analytics – free; easy to implement, easy to derive value

To use Google Analytics, you need to be in compliance with OMB Memo M-10-22 especially on being clear about opt-out options on web measurement.  See my blog post from last year for what you need to do to be in compliance:

Uses: For measuring the number of visitors using the site, the number of starts and completion (such as for a form, or multi-step process), geographic location of visitors, where visitors originate.  Segment these basic metrics by first time and repeat visitors, domain name, content interest (professional, consumer, and researcher).

Sample Metrics

Task Completion Effectiveness: Indicates visitor interest in and successful completion of tasks

Definition: Measures completion of site tasks associated with NIH or IC mission.

Calculation: Expressed as a percentage based on the total number of task completion page views divided by total number of site page views.  A higher percentage indicates more successful task completion.

Return on Investment – Provides quantifiable cost/benefit analysis of budget and time expended on Web site operations.

Definition: Measures return on investment in web site operations, content, applications, and labor in context with budget spent on these initiatives.

Calculation: Task Completion ROI: Number of Task Completions multiplied by $ value given to each Task Completion type compared to total expenditure on creation of Task application or in Avoided Cost alternative to manual labor.

2. Post-Transaction Customer Survey  - Tool: SurveyMonkey, Zoomerang – inexpensive; easy to implement, easy to derive value

Uses: For understanding whether the service altered behavior after use, obtaining feedback on how to improve the service, as well as why service was improvement over offline alternatives.

3. Customer Satisfaction Survey – Tool: QuestionPro – inexpensive; easy to implement, easy to derive value

Uses: Provide insight into whether visitors found what they were seeking and whether they were satisfied with the quality of the content they viewed or the transaction that was completed. 

To use surveys, and a host of other voluntary collection mechanisms, you’ll need to participate in the new Fast Track Process for Collecting Service Delivery Feedback under the Paperwork Reduction Act. Your agency is likely participating in this initiative already. You can refer to OMB Memo M-11-26 for the details of the process and http://www.howto.gov/customer-service for supporting information.

Whether you have an existing web measurement strategy or need to come up with a short term plan, the key to getting value from the data that has been collected is to analyze each set in context with its purpose in evaluating the service. The data should be complementary.

In my next post, I’ll discuss how to “triangulate” data analysis from the different measurement methods at your disposal.

Time to Contribute Your Profiles in Analytics

I've just posted up the new Profiles in Analytics survey. Like last year, I'm interested in collecting and sharing collective wisdom about how people are managing their digital analytics programs...what's working, what's challenging, how to meet challenges.

I published everyone's  responses from last year and plan to do the same this year as well. There were tons of great ideas and insights.

I'd appreciate it if you could fill out the survey and pass the link to others who you think would be interested.

Link to survey: http://svy.mk/pz0IYp

Link to last year's Profile in Analytics results: http://bit.ly/acMePk

On another note, I'm looking forward to seeing you at X Change in September. I'll be teaching pre-conference ThinkTank on Managing a Digital Analytics Program, a class I introduced last year and also teaching a class I introduced at the recent JBoye conference: Beyond Visits and Page Views...How to Develop Actionable Web Metrics, Reports and Analysis. Hope you'll be joining me.

 

 

 

Web Analytics KPIs = BS

I’m starting to prepare for the tutorial I’m giving at the JBoye Conference in Philadelphia on May 3. The class is called Beyond Visits and Page Views…How to Develop Actionable Web Metrics, Reports and Analysis.

One of the areas that I’ll be going into in some depth is how you can build metrics that drive decision making. This would seem to be a clear goal for any Web site measurement initiative, but it is a huge struggle for many organizations. There are multiple reasons, but I think there is one in particular that causes considerable distraction from meeting this goal:

The popularized understanding of Web Analytics KPIs has led us down the wrong path.

Key Performance Indicators (KPIs) are a term that you hear a lot in Web Analytics practice. KPIs are commonly considered to be the metrics that you use for measuring your Web site with Web Analytics. For many, visits, visitors, page views, referrals trended month over month and year over year are the Web Analytics KPIs. This is not what KPIs are about; in fact KPIs are only a part of an equation that creates a complete and useful measurement.

Let’s get some historical perspective. In the 1960s, D. Ronald Daniel of McKinsey and Company came up with a methodology to help organizations define and measure their effectiveness against business objectives. In the 1980s, this methodology was clarified and evangelized by John F. Rockart of the Sloan School of Management. The result was a hierarchal system of evaluation that was based on:

• Definition of organizational mission
  • Identification of strategic goals that map to the mission
    • Determination of concrete and measurable objectives
      • Definition of Critical Success Factors (CSFs) that are used to evaluate the objectives
        • Developing a KPI measurement and target that determine whether the CSF is being met

In Web Analytics, the model could look something like this:

• Organizational mission – Be the industry leader
  • Strategic goal – Provide the most reliable and authoritative information
    • Objective – Publish a weekly email newsletter
      • Critical Success Factor (CSF) – Increase number of subscribers to the weekly newsletter
        • KPI- # of newsletter subscribers per month
          • Target – increase # of newsletter subscribers by 10% each month

The idea here is that you can have a number of CSFs per objective, and a few KPIs and Targets associated with each CSF.

As you can see, KPIs are only one part of a bigger picture and you can do a lot more to make your current Web Analytics numbers relevant, compelling and useful. It doesn’t necessarily have to take a lot of work or sophisticated tools…it just takes a different and more contextual way to look at the numbers.

So, I hope you’ll join me in Philadelphia for the tutorial. I’ll show you how you can use the real KPI methodology to improve your metrics and talk about other techniques that will really help you get value from Web Analytics.  I also suggest you consider the rest of the JBoye Conference. As always there is a great program and I’m pleased to see a big emphasis on Web Analytics this year, as well as online strategy, user experience, content management and lots more.

One last note: There's still time to apply for the X Change Web Analytics Challenge to Benefit Non-Profits...deadline is April 30.

The FTC Privacy Report - A Web Analytics Perspective

Last week’s Federal Trade Commission (FTC) online privacy report draft got a lot of attention for a call to provide a “Do Not Track” option for consumers. While that might have made the headlines, it’s important to note that the report was released by the FTC for the purpose of receiving public comments by January 31, 2011.

After this period, the FTC will finalize and release the report. What happens with it after that is a good question. The Commission hopes that Congress will use the report as a reference upon which to base privacy legislation, an initiative that may get fresh attention in the U.S. House Energy and Commerce committee.  

As someone who works in the field of web analytics, I wanted to understand potential impacts to what we do. Here’s what I found.

The 118 Word Summary

The report and the recommendations are aimed squarely at ad networks, ISPs, carriers, operating system vendors, software application companies and other organizations that collect Personal Identifiable Information (PII) and sell it to third parties for marketing and behavioral targeting without identification of the collection. The report calls this third party marketing.

If you collect customer data and use it to market products from your own web site, this is considered “acceptable practice” and called first party marketing. If this is the case, the Do Not Track option would not necessarily apply to your site, nor would visitors to your site need to be given an “opt out” option.

However, all sites would benefit from plain language privacy policies that clearly disclose how the collected data will be used.     

A Picture is Worth 1000 Words

As a lot of the report speaks to companies that really deal in “trafficking” personal information, it’s important to see where your organization fits in the Personal Data Ecosystem. See if you can find it in the graphic that’s included as an appendix in the report. (These graphics don't reproduce well because they are so detailed. I'd recommend you download the report.)

  

 Chances are, your organization falls somewhere in the Data Collectors circle.

  

In presenting this report, the FTC is trying to move companies who deal with PII into a place where they give people clear options for control of their data. This initiative has less to do with analytics and more to do to with addressing companies that are in the business of capturing, security and access to PII.  

It isn’t that none of this occurs today, FTC would like to see this institutionalized and standardized. Google (Dashboard), Yahoo (Ad Manager) and Microsoft (IE9) have all been taking preventive measures. I've described the Google and Yahoo privacy initiatives in more detail in my last post on privacy.

So, how can your site take initiative to improve transparency?

• Let’s start with your site Privacy Policy. I agree with the FTC that most Privacy Policies are a mess of legalese. So, take another look and re-write them in plain language. Make it clear and transparent. Tell people what you’re doing. If you are using the data collect for marketing and product development purposes, then just say so and get it over with. Don’t bury it.
• Give people a choice to opt out…using browser settings or analytics tool options. Make it clear how to do so.

The Federal Government recently released its own guidelines for how Federal agency web sites should deal with privacy and PII. These might serve as a good template for your organization. For more information, you might find my posting on the new Federal Government web measurement policies a good starting place.

The Good Part Starts on Page 53

If you’re not in the data collection and reselling business, the report may not seem that interesting until you reach page 53. This is where the FTC discusses “First-party marketing.”

“Companies do not need to provide choice before collecting and using consumers’ data for commonly accepted practices, such as product fulfillment.”

 Among those activities considered to be commonly accepted practices include;

“ First-party marketing: Online retailers recommend products and services based upon consumers’ prior purchases on the website. Offline retailers do the same and may, for example, offer frequent purchasers of diapers a coupon for baby formula at the cash register. Some of these practices, such as where a retailer collects a consumer’s address solely to deliver a product the consumer ordered, are obvious from the context of the transaction, and therefore, the consumer’s consent to them can be inferred."

 “Staff proposes that first-party marketing include only the collection of data from a consumer with whom the company interacts directly for purposes of marketing to that consumer."

^"134 If a company shares data with a third party other than a service provider acting on the company’s behalf … the company’s practices would not be considered first-party marketing and thus they would fall outside of “commonly accepted practices,” …. Similarly, if a website publisher allows a third party, other than a service provider, to collect data about consumers visiting the site, the practice would not be “commonly accepted.”^135"

 Described in additional detail in a footnote on the same page:

 “¹³⁴ Staff also believes that online contextual advertising should fall within the “commonly accepted practices” category. Contextual advertising involves the delivery of advertisements based upon a consumer’s current visit to a web page or a single search query, without the collection and retention of data about the consumer’s online activities over time. As staff concluded in its 2009 online behavioral advertising report, contextual advertising is more transparent to consumers and presents minimal privacy intrusion as compared to other forms of online advertising. See OBA Report, supra note 37, at 26-27 (where a consumer has a direct interface with a particular company, the consumer is likely to understand, and to be in a position to control, the company’s practice of collecting and using the consumer’s data).”

The FTC is asking for comment on these recommendations and trying to determine if they are too broad or narrow. I think what they have currently does make sense and provides the level of flexibility that we’d like to have as web analysts.

I’d suggest putting in your comments to ensure that this recommendation goes into the final report. It’s really critical that policy makers understand the difference between what most of us do, and what most of our organizations do with respect to using data vs. third party ad networks and the use of “deep packet inspection”…the real target of the privacy initiatives.

Questions? Comments? Love to hear them.

Google Analytics and Yahoo! Analytics Privacy, Data Ownership and Data Expiration

With the Office of Management and Budget providing a framework for enabling the use of persistent cookies for web site measurement (see post on New OMB Guidance for details) there is lots of interest among Federal web managers and communications staff in using Google Analytics (GA) and Yahoo! Analytics (YWA) the predominant “no cost” solutions on the market. At the same time, I hear lots of hesitancy about using the solutions because of concerns and questions about:

• Whether Google Analytics and Yahoo! Analytics use analytics data for marketing
• Their data collection opt out policies for both site visitors and site owners
• Data ownership

If you’re in the private or non-profit sectors and are considering using these solutions, or currently using GA or YWA, you should know that both solutions have recently modified their privacy policies and opt-out capabilities to address the increasing concern about site visitor control over their data.

What I find interesting about the GA and YWA options is the paths each company has taken over the last year. I think they are quite divergent, and they may reflect business goals that go well beyond the sphere of web analytics.

Google Analytics

Google Analytics provides you with a number of options for controlling data, and the acceptance of Google Analytics among site visitors. In fact, as a site owner, you will need to think about the options you’d like to present to visitors. Google is becoming increasingly interested in addressing privacy issues raised in the US and Europe and the promotion of control over data is one avenue that they are taking.

Using Data for Marketing

GA provides options that are fairly nuanced, so you really need to consider what each one provides.

• Data sharing

When we think about sharing data, concerns often focus on the issue of “I don’t want my site’s visitor information given to Google for their marketing efforts” or “I don’t want my site’s visitor information going to a third party for their marketing efforts.”

GA covers this pretty well. If you go to your Account Settings, you can choose how you want your data shared, if at all. Selecting “Do not share my Google Analytics data” effectively prevents Google from using the data for marketing purposes.

• Opt out

Personally, I’d always recommend to your site visitors that if they don’t want to be tracked on your site, then direct them to manage this through their browser settings. However, if you want to give them another path, you can direct them to download a GA plug in that will disable GA tracking on any site they visit…not just yours. While this may be a civic-minded option to offer, I don’t endorse it because you’re encouraging visitors not to be tracked. I wrote about this in some detail in an earlier blog post. You can check out the plug-in directly at http://tools.google.com/dlpage/gaoptout?hl=en. I affectionately call this the Nuclear Option.

• Anonymize IP

GA already obscures IP addresses so that you cannot report on visitor level data.Now, you can further anonymize IP addresses through the addition of additional code to your GA tag. This will remove the last octet of the IP address prior to its storage, and slightly reduce the accuracy of geographic reporting, according to Google.

Just keep in mind that these options apply to your site visitors and the data you collect on their activities.  They don’t apply to you, the GA account owner.

Let’s start with the Terms of Service for GA <http://www.google.com/analytics/tos.html> and go down to point #6. INFORMATION RIGHTS AND PUBLICITY. This passage explains that Google won’t share your data with third parties. Now, go to the link within #6 to Google’s Privacy Policy < http://www.google.com/privacy.html>.

This describes privacy for use of all Google services. I’ve summarized the items that will be of most interest if you’re using Google Analytics:

Information you provide – describes how Google uses the data you’ve provided when you sign up for a Google Account. So, if you’re going to open your GA account, you’ll need to provide your name and email address. Google can use this information to, in their words, “provide you with a better experience and to improve the quality of our services.”

Cookies – Google uses cookies to track your visits. If you are using GA, they will track your usage. This is also used to “to improve the quality of our service.”

Log information – When you access GA, their servers record your visit, as well as information such as, “Internet Protocol address, browser type, browser language, the date and time of your request and one or more cookies that may uniquely identify your browser or your account.”

And in case you didn’t get it from the “Information you provide” section, the following statement concludes the list:

“In addition to the above, we may use the information we collect to:

• Provide, maintain, protect, and improve our services (including advertising services) and develop new services; and
• Protect the rights or property of Google or our users.”

So, the bottom line is that if you open a GA account and provide usage to others in your organization, Google can use that data for marketing and product development purposes.

Recommendation - If you’re a Federal government site, select the “Do not share my Google Analytics data” and direct people to use their browser opt out. Go to USA.gov if you need a template to work from. Remember that you must update your privacy policy per the OMB directive before you start using GA. See NIH and NCI privacy policy examples as templates.

If you’re a commercial site, consider your site visitors’ privacy vs. the advantages of using Google’s benchmarking services and select your data sharing option accordingly. Remember to be up front and transparent about this in your privacy policy.

Data Ownership and Other Considerations

Don’t let “free” obscure your vision. Read the Terms of Service and know what you’re signing up for. Also notice what isn’t covered.

• Google indicates in that it “may retain and use, subject to the terms of its Privacy Policy (located at http://www.google.com/privacy.html), or such other URL as Google may provide from time to time), information collected in Your use of the Service.” Yes, they provide the opt out and data sharing options described above so that appears to mitigate unauthorized usage. Of course, they could make this a bit clearer. You’ve got to keep drilling down to get the whole story, and then you end up back either back in the TOS or mucking about in the Privacy Center.
• From #2. Fees and Services: “the Service is provided without charge to You for up to 5 million page views per month per account, and if You have an active Adwords campaign in good standing, the Service is provided without charge to You without a page view limitation.” If you have a  a site with more than 5 million page views, plan to do an Adwords campaign.
• Data retention: With fee-based tools, there is generally a default length of time that data is kept and this can be negotiated. There is no such guarantee with Google Analytics. Even though we’ve seen data kept for indefinite periods, there is nothing in the Terms of Service to ensure that this policy will continue.
• Data sampling: GA samples visits on reports with 500K visits within a selected date range to return reports within a reasonable time frame.

Recommendation - Google Analytics may be a good choice for your Federal agency web site, especially if you have little or no budget for web analytics, have less than 5 million page views per month and are comfortable with archiving your reports on a regular basis.

 

Yahoo! Web Analytics

YWA is distinctly different than GA in that its goal as an analytics tool is to provide the potential for delivering data that can be attributed to specific visitors. This is consistent with all fee-based options.

Using Data for Marketing

Yahoo is pretty up front about using YWA data for marketing and product development. This is described in the Terms of Service <http://info.yahoo.com/legal/us/yahoo/ywa/ywa-4750.html>. According to the TOS, under Analytics Program Terms, you must state this in your Privacy Policy:

NOTICE AND ABILITY TO OPT-OUT. You will prominently display within your website’s privacy policy a notice that includes each of the following disclosures in close proximity: (i) a statement that expressly identifies Yahoo! as a provider of third-party web beacons on your website(s), (ii) a statement that describes your use of the data gathered by the Analytics, (iii) a statement that expressly identifies Yahoo! and its use of the Analytics data to improve Yahoo!’s products and services and to provide advertisements about goods and services that may be of interest to end users, and (iv) a statement that describes how the end user may learn more about the choices offered in connection with the use of Analytics information, including an active link to the following URL: http://info.yahoo.com/privacy/us/yahoo/opt_out/targeting/details.html.

This is followed in the TOS by sample language that you can use.

Enabling visitors to opt out of YWA is a bit convoluted. The company recently removed the stand alone opt out for YWA and combined it with general opt out options for the ad network. Unfortunately, there’s nothing on the ad network opt out page <http://info.yahoo.com/privacy/us/yahoo/opt_out/targeting/details.html> that indicates anything about YWA.

Visitors need to know to select “Opt Out” as shown in the screen shot. This opt out is a cookie-based solution, and visitors must keep the third party cookie on their machine for it to be effective. Like the Google Opt Out plug-in, I don’t recommend this direction, and would much rather you push the browser based opt out from your site.

Unlike GA, but similar to fee based tools, such as Omniture, Webtrends, Unica and Coremetrics, there is no anonymizing of IP addresses.

Recommendation - If you’re a Federal government site, I could see some issues about using YWA. While the transparency is all good, you may cause some dust ups by telling visitors that their data is being used by Yahoo, as well as indicating to the public that you are passing their data to Yahoo. True, the data passed to Yahoo is an aggregate number, and there’s no personal information being transferred, but try explaining that nuance to your boss or the hundreds of thousands of visitors coming to your site.

Data Ownership and Other Considerations

Pretty much the same as GA…no mention of data retention in the TOS, although YWA assures me that it’s their policy to never expire your data. I believe their good intentions, but if it’s not in the TOS, then it isn’t guaranteed. YWA does not sample data, but you can make arrangements for data sampling if you’d want to explore this option. Like GA, the TOS is written clearly as a document that gives you the right to use YWA as long as and under the terms laid out by Yahoo, and like GA, these terms can change at any time. Remember…it’s free.

Recommendation - YWA may be a good choice for your Federal agency web site, especially if you have little or no budget for web analytics, and you don’t mind that visitor data is going to Yahoo to provide them with info to be used for their marketing and product development efforts. Like GA, I’d recommend that you archive your reports in case there comes a time when data isn’t kept indefinitely.

Summing Up

While I’ve spent this post describing the data capture issues for both of these “no cost” tools, these are just a few of the requirements to consider in selection. They both have differences in data collection, reporting and analysis that you should look into as well before deciding which one works best for your overall analytics requirements.

 

 

 

What the New OMB Guidance on Web Measurement Means for Federal Agencies?

I was overseas when the guidelines on the use of web analytics and measurement on Federal government web site was published on June 25 and then spent a few days tromping in the woods with my son, so I haven't had a lot of time to digest the OMB's final memos. Yet after reading through the memos pertaining to web site measurement (M-10-22) and use of third party websites and applications (M-10-23), like Facebook and YouTube, I feel optimistic about options and opportunities if you're running a Federal agency web site or using web analytics on a Federal web site. When I last wrote about this on May 4, I expressed doubts as to how friendly the policy would be for Federal agency web managers and web analysts. 

Here are the biggest wins for conducting web analytics and measurement on Federal agency web sites:

1. Using persistent cookies is ok. This had previously been a major sticking point for Federal web managers and web analysts who had only been able to identify visitors at a session level. Remember, unique visitors in web analytics means only that you are identifying the computer from where the visit originated...there is no Personal Identifiable Information (PII) being made available. Now you have the opportunity to use your web analytics tools to create segments based on unique visitors, such as first time and repeat visitors. You can apply these segments to referral sources such as search engines, search terms, geographical origination and domains. Now, you'll be able to observe how your content, applications and marketing are used by groups of visitors. With this information, you'll be able to understand more clearly the requirements and usage that visitor groups have based on behavior and be able to develop new content and applications that are more focused on specific user groups. For example, if you can now tell that first time site visitors are most apt to exit the home page, you could consider providing a web site tour specifically for first time visitors and promoting this on the home page; or if you find that you get multiple visits to your search engine from visitors who have been to your site more than 2 times within a month and are using the same terms, you could perhaps use this information to provide more visibility to the desired content. What you need to do: You can not start using persistent cookies today or tomorrow, or next month for that matter. You'll need first to alter your Privacy Policy (see discussion below) to enable visitors to opt out.
2. Using free software that depend on persistent cookies is ok. You can now consider using Google Analytics and Yahoo Analytics! to be viable site measurement options. Use of these tools has been considered off limits or in a grey area at best because of their reliance on persistent cookies. While both provide methods for disabling or minimizing the use of persistent cookies, neither gained much traction within the Federal government.  What you need to do: The same rules apply as described above with regard to Privacy Policy. If you do want to go with the free software approach, please, please, please, spend time considering how you are going to enable access to the web analytics, who is going to use analytics, how are they going to use it, how to train and educate folks on the use of GA or YA, as well as how to interpret analytics data. I strongly suggest too that you determine the important metrics to track and how you'll use this to support decision making.   I can assure you that if you don't plan your deployment of GA or YA with any sort of plan you'll create more headaches for yourself than you have now without any web analytics in place and quite possible damage your management's and peer's buy-in to the value of web analytics for a very long time.
3. You'll be fine as long as you don't collect PII.The guidelines are relatively easy to accommodate as long as you don't want to collect PII, such as names, email addresses, postal addresses. This is classified as Tier3 - multi-session with PII data. Most of you will be concentrating only on Tier 2 data - multi-session without PII. What you need to do: If it is only Tier 1 and Tier 2 data you are collecting, you need to be in compliance with what is outlined in M-10-22. If you do want to collect Tier 3 data, you'll need to work with your Senior Agency  Official for Privacy (SAOP)and then put out for public comment as to why you need to collect such data and then a review by the CIO. This will likely to be a long process.The only exception is getting written permission from your agency CIO to this process if you can make a case for a delay causing serious public harm...which in most cases is not going to happen.
4. You need to evaluate the data collection practice of third party sites, such as YouTube and Facebook.This is one of the most interesting aspects of the guidance...as long as a third party site does not share the PII that they collect with your agency there appears to be no issue, and as long as the third party site does not use their data on behalf of a Federal site, there also appears to be no issue. It also appears that you can use baked in analytics tools, such as YouTube Insight, or blog packages such as WordPress, as these collect Tier2 data. What you need to do:If you are using third party social media sites, you'll need to consider risks as outlined in OMB’s memorandum providing Guidance for Agency Use of Third-Party Websites and Applications that can be found in M-10-23 and prepare a Privacy Impact Assessment (PIA). You will need to work with your agency Senior Agency  Official for Privacy (SAOP) for approval of the PIA, so find out the process and expectations of documentation you will be required to provide.
5. Data retention is flexible.There are guidelines for retention of web data, but it is not an explicit mandate. According to the policy, agencies should limit the retention of data to one year or less. However, it would seem that there can be reasonable arguments made for keeping data for longer, such as wanting to understand year over year trends.What you need to do: Consider your current data retention time. Is it necessary to keep analytics data beyond a year? While many agencies keep data for years, it may not have business value. I'd suggest that  you start planning your data archiving and expiration strategy, if you don't already have one.
6. Web site measurement opt-out options focus on Privacy Policy communication. You need to make it easy for site visitors to opt out of web site measurement. Fortunately, there are a number of options at your disposable, and they do not require you to work with your web analytics software vendor to make them happen. The good news is that as long as you're in compliance with what is outlined in M-10-22 and make your privacy policies very clear and easy to find, you will have fulfilled the OMB requirements. Attachment 3 in the M-10-22 document provides a check list of items that need to be addressed in the privacy policy. The caveat here is that it may take multiple rounds of revisions and approvals to get privacy policies approved. What you need to do: You should understand the process for revising your agency privacy policy and who or whom provides final approval. While the OMB policy sounds relatively straightforward from a content perspective, the actual process in your agency may not be quick nor easily deciphered. On the content side, here's what you can do to provide the opt-out measures:
   

• Use web analytics software that enables visitors to opt out, such as Google Analytics, Omniture, Yahoo! Web Analytics and Webtrends. The caution here is that if you allow the GA or YWA opt-out, you'll be providing visitors with the option to opt-out of Web wide measurement for these solutions. Read my post on CMSWatch for a more detailed description of this issue.
• Provide clear instructions to visitors on how they can opt out of web site measurement on your site. From my perspective, this is clearly the better option to take. USA.gov provides a nice central location with instructions on opting out of measurement on popular desktop and mobile browsers. This is a great reference.

So with all of this seemingly good news are there any downsides to the policy. Well, it may depend how you read it. I think the biggest grey area is around the Privacy Policy discussion in Attachment 2. The memo states that you can use web measurement and customization technologies for Tier 1 and 2  as long as the agency is in compliance with the Memo, there is clear and obvious notice in the Privacy Policy citing the use of technologies and there is compliance with internal policies governing the use of such technologies. Some may see this as needing to get permission or needing to amend the Privacy Policy to maintain even their current web measurement. I don't think this is the intention of the policy and I certainly wouldn't recommend approaching implementation of the policy in this way. However, a quick review of about a half dozen Federal agency web site Privacy Policy statements tells me that there are no policies currently in full compliance with the new additions described in Attachment 3 of the memo. What you need to do: From a practical perspective, it wouldn't make sense to shut down web analytics across the government, however, to really beat home this message, if you are in charge of web analytics at your agency or are using web analytics within your agency, you need to make it a priority to determine how your Privacy Policy needs to change, the process for enabling the change, and who you will need to coordinate with in order to make this change possible.

Final Take: While the policy itself looks like it is fair and provides Federal web site managers and web analysts with long awaited flexibility, we know that implementation of the Memorandum, may not be smooth sailing. As the cornerstone to much of the flexibility is derived from rewriting web site Privacy Policies, swift realization of the Memorandum's benefits could get hung up in long rewrite and review processes. I suggest taking a pro-active approach to determining how your site Privacy Policy needs to be revised and understanding who is involved in approval and the process for obtaining approval. These will be the most challenging aspects to making the Memorandum guidance real.

Waiting for the new Federal Government Cookie Policy

Is waiting for the new OMB guidelines on a new Federal Government cookie policy like waiting for Godot?

You might say that it is...public comments were taken in the summer of 2009, there was expectancy that an announcement would be made this past April 7 during the Open Government Initiative fanfare and Vivek Kundra said last week at a Federal Government web managers meeting that a decision could be made next month.

While it is expected that persistent cookies will be allowed, there is equivalent expectation among Federal web managers whom I speak to that there will be accompanying "opt out" protocols, multiple tier for usage definitions, data expiration and approval rules, that will actually make it as, if not more, difficult than it is today, to get anonymous visitor data.

After the public comment period over the summer, I'd expect that OMB had more content to sift through than they expected (classic data deluge scenario) before they could figure it all out. 

I've been a sometimes participant in the process...I've added my 2 cents during the commenting period and have provided opinions, clarifications and definitions when asked. But for the most part, I feel pretty much like an observer watching how Washington works and how policies get made. I guess just like any analysis, the more answers you find the more questions arise. Here are some of mine:

• What is the role and influence of the Center for Democracy and Technology (CDT), a non-profit public interest group with a mission shaping government policies on Internet issues? Ari Schwartz, the organization's COO is credited with developing the paper that kicked off the discussion in May 2009. He sits on the board of the Information Security and Privacy Advisory Board (ISPAB) of the National Institute of Standards and Technology, the the federal technology agency that works with industry to develop and apply technology, measurements, and standards. Google also sits on the ISPAB as do other consulting, IT firms, universities and government agencies.
• CDT co-wrote a set of recommendations with the Electronic Frontier Foundation in May 2009 that outlines the disclosure, data retention, opt out measures described above, as well as requiring that Federal agency Inspector Generals verify their privacy compliance. Is this the blueprint for the new cookie policy?
• Mr. Schwartz wrote in a blog post on January 9, 2009 "that government should be creating policies to encourage this innovation, rather than railroading the issue with an inflexible mandate (i.e., the current policy) or eliminating the government-wide policy altogether (a possible alternative)" He goes on to say:         

"1) There are a growing number of cases where information about an individual may not be directly personally identifiable, but where the individual has a privacy interest based on the use of the information. IDs of all kinds (including those used in state management mechanisms) and location information are two prominent examples. Today, there are few privacy rules in government to cover these kinds of information."

"2) There are clearly some instances where federal Web sites could be greatly improved through the use of monitoring aggregate and individual usage for diagnostic and analytical use. The feasibility of conducting such analysis in a privacy-protective manner deserves further exploration."

"A study panel should consider what the appropriate policy guidelines should be for these situations. This panel should assess how policies specific to federal Web sites can allow beneficial uses of cookies and other state management mechanisms while protecting privacy, taking the differences between the types of data commonly collected online into account."

My questions: Has this panel ever been convened? Who participated? What was determined? What was the outcome? Where is the report available?

• How does Google influence this discussion? Well, they serve on the ISPAB, they are probably the private sector lightening rod for all things related to privacy, and they have a rapidly spreading footprint within the Federal Government for services and applications. I don't think its a coincidence that they announced Google Analytics availability on the GSA schedule, nor that they are working on opt out options. Although the company has pointed out to me that opt-out will also help them do business in other parts of the world where privacy is a big issue.
• Where is the web analytics "lobby"? Oh right, there is no web analytics lobby. There's the Web Analytics Association, but haven't been hearing too much from them on this issue outside of some meetings. That's probably because the interest among web analytics software vendors and the WAA membership is in "online marketing optimization." Sure there's interest in selling to the government, but its a tough sell, and a lot more limited in upsell modules like testing, behavioral targeting, and so forth. Maybe they'll start paying more attention to what's going on in DC after the introduction of the privacy bill in the House by Rick Boucher. I haven't seen the bill, but there's curiously familiar terminology being  put out there like "it would require firms to allow consumers to "opt out" of having such information collected." Maybe everyone is waiting for Google to carry the banner on this. So, I wonder what the WAA, its corporate members and all those who are invested in online are going to do about this. Then again, we all know how long it can take any bill to get through Congress. Maybe its not something to worry about....too much...for this decade.

Getting back to the Federal cookie policy, I wonder if by the time this new policy comes out and folks figure out what to do with it if it doesn't become the Internet version of the famous Maginot line. You remember your WWII history, right?

The French built a line of defense on the German border thinking it would keep them from being invaded. So what did the German's do? They marched around the line and completed their invasion.

I wonder if this is how the new cookie policy will viewed after it's published, if it indeed does come with the baggage outlined in the CDT/EFF recommendations.

The larger and simpler issue in my mind is in making government web sites more valuable to the citizenry. Making them more valuable means making them easier to use - better and personalized content, finding what you need more quickly, applications that save time. Using cookies to enable personalization, get a better idea of unique visitation, first time visitors, repeat visitors, where visitors are from, what campaigns brought them in and so forth all help in guiding decisions towards that end.

Making it hard for Federal government agencies to use persistent cookies is akin to "throwing out the baby with the bath water." In the name of privacy, the main objective of Federal web sites...to provide useful information and help us interact with the government, is hampered and diminished. The Federal government already has all of my information and if they wanted to tie it together (which they do already if there's a perceived security threat or criminal act) then they can. Amazon, Google, Microsoft and Adobe already have a boatload of info about me and everyone else in the US. State governments and non-US governments have been using persistent cookie based analytics solutions like Google Analytics to successfully improve their web sites for their constituencies.

So, if Federal Government web managers are given a mandate to improve their web sites to serve the citizens and do it in a cost effective manner and need web analytics to guide them perhaps they'll just start using Google Analytics and Yahoo! Analytics...both free, both with ways already to disable persistent cookies, and take their chances that a constituent or enterprising reporter is going to look at their site and see that there's web analytics code on their pages and collecting information about their site usage. So, it will be in the papers for a few days and just go away...remember, its already happened to the White House...and life just goes on...and perhaps more Federal government web sites will start moving to having online presence that's really useful. 

Related Posts:

Waiting for the new Federal Government Cookie Policy
Can US Government agencies really use Google Analytics?
Whose Data is it Anyway? Part 1
Whose Data is it Anyway? Part 2

Whose Data is it Anyway? Part 3