Jan 04 2010
Web analytics limitations … and a bright future
I recently read Joseph Carrabis’ fantastic posts – The Unfulfilled Promise of Online Analytics part I and part II. If you haven’t read them already – do so. They are so thought-provoking and written like no other analytics posts you’ll ever read. Really great stuff.
For an analytics evangelist like myself, reading them was both a breath of fresh air and a punch in the gut.
It got me thinking about web analytics in general, but also in our little corner of the internet – the higher education websites. There are so many limitations as far as online analytics is concerned and, as Mr. Carrabis points out, most of it is because of people. Yes, people. People in the analytics industry. You. Me. All of us.
Why am I bringing all this up you may ask? Why not? I’m still an analytics evangelist, but I acknowledge the fact that there are holes … big holes that we need to work together to fill up. I also acknowledge the fact that higher education is *years* behind other industries when it comes to analytics – all analytics. Sometimes I think this may work to our advantage.
If you really think about it, we aren’t unlike other industries in what we want to get out of our websites. We also aren’t unlike companies that have issues with web analytics – numbers don’t add up, leadership doesn’t take the recommendations of the analysts (that are based on *facts*), leadership doesn’t *trust* the numbers, etc.
They do have a point. The web analytics industry has yet to have any real standards. Definitions, tagging, metrics, are different from tool to tool. Because there are so many variables involved, numbers will never tie – heck numbers often don’t tie between tools from the same darn company because they gather the data differently between those tools. From the same darn company! It’s enough to drive you batty.
There isn’t a ton we can do about that right now, though. Question for all of us – have we done all we can to get the most out of these tools, even with all their limitations? We complain all the time (why can’t I do this or this or this) – I know I complain. I’m very guilty of this.
At most our institutions, the current times have given us the opportunity (yes, opportunity!) to scrutinize what we do more. To look at the numbers more. Website analytics is a small part of the larger analytics picture, but it is a part. It’s a very important part and it will only get more important.
Where can we do better? Let’s talk about training. How many of you in higher education have been trained in your analytics tool? I’m talking real training, whether it’s self-taught or classes, or consultants, or anything. I’m also not talking about, “training in your spare time” because we all know how that goes.
From Mr. Carrabis:
… the use of any tool is going to require training across the usage spectrum. The use of new tools definitely so. This training can be self-training and the user should be prepared for scraped knuckles, smashed thumbs and lots of cursing. Self-training is great when the user has lots of time and patience. Otherwise, take a class or let the experts (”consultants”) in.
He goes on to say:
More training is the answer only if the training results in well-reasoned and understandable business actions*. Tools and trainings are worthless without knowing what one wants to build (”It reminds me of the development of web sites themselves ten years ago – everybody had to have one, still not being absolutely sure what to use them for. Of course the free tools have done their part in this evolution.”).
So, although the analytics industry has to get there … eh em … stuff … together, we do as well.
Bright Future?
So let’s get to the “bright future” part of this post before everyone ends up throwing in the towel and going back to the homepage hit counter. Yikes!
I really do think that web analytics has a bright future, especially in higher education. Some thoughts:
- Web analytics is young. Avinash Kaushik has said it’s a toddler. When it comes to higher education, I’d say web analytics is an infant – still wearing 0 – 3 month clothes. We’re trying to figure all this out. We’re still at the joke-forward stage. Remember when you were first introduced to e-mail? You’d get jokes or stupid chain e-mails and forward them to your entire address book because it was something that was new and cool and you weren’t quite sure what to do with it? Today, if you get joke forwards or chain e-mails you either reply to sender cursing him/her out or just delete it without reading. Higher education is still there with web analytics. We spit out visit reports, see an increase after a certain marketing campaign and say, “cool, a job well done” and nothing more. This is good news for us, though. There is nowhere to go but up! But, if we don’t change and evolve, this can be bad news. 2 years from now we shouldn’t still be forwarding jokes!
- Higher education is slowly but surely coming to respect the power of analytics (not just web analytics!). With a little centralization, some training, and some resources, we can do this. Should be easy, right? : )
- Web analytics vendors are starting to feel the pressure from end-users to improve their products, both functionality and customer/technical service. Google Analytics alone has come out with a ton of new features just in the past few months. Remember, however, that we must practice (training!) using them or new features don’t mean a thing.
- The web analytics industry is trying to standardize definitions. They’re not there yet, but they’re trying.
I hope that I’m not wrong on #2 above. With some of the conversations I had at HighEdWeb in October I may be. What we have now is better than what we had a year ago. It’s better than what we had 4 months ago.
Let’s face it – with the economy and layoffs, and everything else that’s going on in higher education these days, we are being asked to scrutinize more than ever before. For those of us that came from another industry, this is nothing new. Even with all the limitations, analytics has a lot to offer, but we must educate ourselves and leadership. We must continually ask for training and show leadership why it’s so important. Secondly, we must educate others on what web analytics can and can’t do. It cannot tell you how many *people* clicked a certain button while standing on one leg wearing a red t-shirt. It is not the be all, end all. It can tell you how many times that button was clicked, and using segmentation, what other things during those visits. to your site. If you’re using both quantitative and qualitative analytics, you can start to know not only the “what,” but the “why” as well. To me, that alone can tell you so much. So much so that it’s definitely worth the time and effort. As for the limitations – I’ll take them … and move on.
PostScript: My apologies for a stream of consciousness post. The 2 posts mentioned really got me thinking and this post is an attempt to articulate those thoughts – it may be a poor attempt, but it’s one nonetheless.
FYI … coming up within the next month - Look for a survey on the state of web analytics in higher education. After reading some analytics posts recently, I’d love to gather some data from higher education as to what we’re using, how we’re using it, etc. I really want to look in-depth, especially into how people are using the data (if at all).
*My emphasis
Thank you for your kind words and thoughts, Ms. Thayer. I’m amazed at the response those posts are generating. Many people have contacted me to provide a part 3 and I was pretty much content to close the arc in part 2.
Regarding your post, your points are well taken. My original research did include some higher ed people and definitely there were some higher ed folks who contacted me after the posts went public.
Like you, I believe that analytics has a bright future. Will that future analytics be in its current form? That I’m not sure about. Everything evolves over time. It’s the environmental/ecologic pressures that determine what products survive. The current economy, population dynamics, ease of use (read “required education and training”), support infrastructure, user satisfaction, …, all play roles in shaping those pressures.
Please feel free (either yourself or your readers) with ideas for continuing that arc.
Thanks again,
Joseph
Great post Shelby,
based on my involvement with a couple of educational institutions, I totally agree with your assessment of falling behind in terms of analytics capabilities in higher-ed. One of the very interesting comment I got about my work on the Web Analytics Maturity Model was the following: “The words “maturity” and “sophistication,” used in the paper, are also value-loaded words. My belief is that a model works best if the model user can figure out which stage is best for them”.
Pressure is high within our industry to “move up”. Rather than trying to climb the web analytics maturity ladder, maybe higher-ed needs a baseline, a framework to get a good control on what’s possible (addressed by defining good objectives in the first place!). Instead of trying to do all the great stuff more capable (in terms of resources) companies can do, higher-ed could be better served by a set of well defined analytics practices – practices that can only be defined by collectively sharing our knowledge and conducting peer review… some would say it’s an academic exercise and a waste of time… I say it’s a necessity and an invaluable exercise.
For your survey, I would avoid a simple tools inventory and would recommend looking at the key practices of online analytics. Of course, I’m biased, but allow me to suggest the Web Analytics Maturity Model assessment method to gather this information. Please don’t hesitate to contact me if you need help in conducting this survey.
Regards,
Stéphane
A very nice post Shelby,
We’ve been working on reaching out to the academic community through the WAA Research Committee.
To my mind, it makes sense for academic institutions to practice a lot of what their researchers are preaching. (Though, I know full well the barriers between the institution and researchers
)
You’re right to be optimistic Shelby.
So much is possible.
Thank you, everyone, for such nice comments.
@Joseph - What I loved most about your posts is that rarely did any *tool* come into the conversation. Can a certain tool do this or that. We’re missing the point if that is our biggest complaint.
It’s my opinion that the biggest thing that higher education lacks right now in web analytics is training (for various reasons). Regardless of limitations, there are so many things we could be doing right now and don’t – either because the HIPPOs don’t think it’s necessary or because we don’t know how to do it (or both).
You’re right, it will definitely be an interesting couple of years in the analytics industry and I’m curious to see where higher education will go regarding it. I didn’t even bring up the whole issue of privacy, which is huge in higher ed.
By the way, I agree with everyone else, you should definitely do a 3rd post. It’s such a fantastic conversation and one that is desperately needed. I wish some vendor representatives would chime in.
@Stéphane - Your WAMM paper was great and I have to admit that across the key areas, higher education would be at a 0 or 1 level. Governance, for example, is a nightmare. I would guess that most websites are decentralized (especially at larger institutions) and the “champions” will come from different levels for all units. For large institutions we’re talking 100s of websites (with different owners, processes, etc.). Obviously this is a generalization, but I’d argue that more often than not, it’s true.
I would love your input on the survey and will definitely be in touch about it. I’m in total agreement with you that it needs to be more practices oriented and less tool-oriented. In higher education, staff are so hungry for training that those are the bulk of the questions that are asked (how do you do X with a certain tool). They are all great questions, but I’m of the opinion that we all need to understand *why* we need to do X with a certain tool and how it fits in and/or has implications for things elsewhere. We need to get away from the task-taking and move more toward a holistic strategy. Training is so lacking that that is the squeaky wheel now.
@Christopher – I completely agree with you as far as the research aspect is concerned. What we lack as a whole around web analytics is a champion. We just don’t have that right now. When I say “we” I mean the collective we – higher education.
I don’t know if you are all based in the U.S or not but I can only speak from how it is in the U.K and would say that Web Analytics in the U.K is still at the ‘newborn’ stage.
Knowledge, importance of data driven action, inspiration and drive are all attributes which are scarce in the U.K when it comes to web analytics. All the great speakers, evangelists, educators..heck even the events and associations are all U.S based.
For me, as someone inspired by and using web analytics to improve online marketing is pretty disappointing to read that you guys are already thinking about higher education and the sort. I would be happy if web analytics got the recognition it needs within businesses for a start.
@priteshpatel9 – Thanks for the comment. I work for a university based in the US. I think others who have commented do not live in the US, but please chime in if I’m wrong.
The level of analytics use/expertise/etc, in my opinion, really depends on the industry. There are a lot of companies in the US that are using it extensively. There are also a lot of companies that aren’t at all.
At universities, however, we are at the infant stage as well. I’m thinking of higher education only because that is the industry I am in. I’m sure, even here in the US, there are other companies in your boat, though.
Keep up the good fight.
Great post Shelby. Lots of good points and agree with you about web analytics having a bright future. However, I am going to take the minority stance on analytics in HigheEd and disagree that analytics in HigherEd is lagging behind
. I know I might be gheraoed and bashed by you, Joseph, Stephane and Chris but I will take the chance and be honest with my thoughts 
Web analytics is a young field – but analytics is not. I grew up navigating through a wide array of disciplines in academia and over the years I have seen some brilliant analysts in fields such as Marketing, Economics, Decision Support and the like. In fact, many of them have used advanced analytical techniques for years that only now the Web Analytics community is even hearing about (One of the reasons I am surprised when folks go ga-ga over Predictive analytics – the stuff has been in use for decades). Now having said that, I would agree that what is lagging behind is the application of analytics in the web arena. For years web analytics was like a mantra known only to a few chosen ones and as such it never took off in the academia or with the masses (that is changing though). Secondly, for years we (the WA community in general) have put ourselves on a pedestal and looked down on other domains. If we really want to progress, we should admit our lackings and learn from the various other domains — including ‘non-analytical’ domains such as cognitive science and in the process show how our side of the world (web analytics) can help them with their work. Third, I do agree with you that there needs to be a structure in place in terms of standards, processes etc. We argue about ‘Art vs Science’ in web analytics – it does not really matter. You will not find a single field of study that is successful that does not have a ‘guiding principle’ and certain stakes on the ground in terms of what-it-is and what-it-is-not etc.
Enjoyed the reading.
Regards,
Ned
(try #3. If it doesn’t get in this time I’ll just email it to you)
Hello again,
Yes, we (NextStage) is vendor/tool agnostic. We prefer it that way.
Yes, HiPPOs. It might be helpful to remember that each culture has a version of its water horses. They are always helpers, monsters or chimeras. Perhaps that’s another bit of research; how do groups employing recognized HiPPOs think of them?
Yes, training training training. It’s the new location location location.
Yes, privacy. Something I’ve been concerned about since before NextStage began.
3rd Post – I did ask vendors to take part based on responses to P1 and there were no takers. Sigh.
I’m still open to suggestions and guidance there, by the way.
Joseph
@Ned – Thanks for the comment. You are right. Obviously there are those in higher education that are analytics experts. Point definitely taken.
I completely agree that we need to be accepting of other techniques. I think that’s why I found Joseph’s post so fascinating.
@Joseph – Sorry about your trouble posting. If it was on my end, please let me know what happened and I will troubleshoot.
It’s sad to hear that vendors would not participate. Hopefully because of the response you’ve received some will change their minds. Then again, they probably won’t.
About post #3 – I like Christopher Berry’s comment in post #2 – where do we go from here? We can talk about it until we’re blue in the face – not that the conversation isn’t important and interesting – but what do we do about it? You state that, “Time to mature is culture dependent so the online community as a whole must do the work.” Agreed. So is that what we’re doing right now? How do we “do the work?”
A couple of clarifications
On the analytics vs HigherEd, realized that there are a couple of aspects mixed in here and so wanted to clarify. First, there is the what I call ‘core knowledge’. This is what I was referring to in my earlier post. Academics have excellent knowledge of analytics, analytical technique, and you will even find great examples of applications to various problems from Market Research to Economic Forecasts to predicting Consumer Behavior.
The second is what I am going to label as ROK (Return on Knowledge) vis-a-vis that institution — in other words how well an institution of higher learning applies the core knowledge to improve their KPIs. In this I would tend to agree more with Shelby and Stephane that academia is lagging behind – especially in the application of web analytics, and that they can do a lot more to leverage analytics to boost enrollment or increase grants etc. However, even here (based on my experience) I see a big variation across various institutions depending on whether they are a focused Teaching school (no Ph.d programs or even Masters in many cases) or Research school, size and diversity, city (or State), proximity and collaborations with surrounding practioners and/or companies etc. I know of schools that use analytics and web analytics — but then I would not be totally surprised if there are schools with barely a website.
As to where we go from here – well, I will leave you with my Tweet to Stephane:
It is time WA #measure is stripped of the cocoon of factions and emerge as an independent yet collaborative field.
Regards,
Ned
I love it when such interesting conversations sparkle from a blog post. If anything, true social media and engagement is demonstrated when a thread of comments enrich an original point of view
For surveying the “state of the union” in higher ed, I’ll be able to offer a draft survey in a couple of days and we can work together to improve it.
I don’t want to reveal too much for now (got scooped once with a great idea, don’t want to risk it again!)… but I’m working on something that is blatantly lacking in our industry and once the background is established, I will be honored if you can help out.
Stéphane
Hello again,
What Ned wrote in his tweet to Stephane – yes.
Joseph
PS) This conversation is greatly helping me figure out what P3 will be about. Thanks.
[...] came up: Shelby Thayer’s “Web analytics limitations … and a bright future” (here). From there I found a link to Jospeh Carrabis’ excellent post “The Unfulfilled Promise [...]