“How do you measure success?”
It’s the question you get from your hiring candidates, from your peers, from your boss, from the people you serve, from the finance department, and probably even your family. It may be the most frequent question of the many you field each week.
For a lot of professionals I know in the developer evangelism game, the question raises a lot of apprehension. For some, measurement feels tied to their value inside their company. For others, measurement is how they secure future resources to continue to serve developers well. And for everyone, it is a needed source of intelligence to improve the craft and deliver better work.
If you don’t have a great answer for this question today, you should know you’re not alone. Everyone in developer relations starts measurement from zero at some point – here’s how you can get started.
Why Is Measurement So Hard?
First, a word on why so many in developer evangelism is uneasy about measurement. It isn’t because the discipline doesn’t want to measure success and it isn’t because the discipline can’t be measured.
Developer relations isn’t measured frequently because it is expensive to measure. All the software to link offline activity to online engagement to financial results costs a goddamn fortune in annual licenses and custom development to get right. The packages you need all start at ~$120k per year. Why so much? The pricing model for is oriented around how much it would cost to hire engineers to build a fraction of the functionality for you. “For the price of one engineer,” they say, “you get so much more.”
Further, those integrations are a struggle to get going as they touch nearly every part of the organization. Even if you fork over the money on the subscriptions and the consultants, without a quorum of invested stakeholders across the company, it’ll get bogged down into integration hell.
This is important for both you, your team and your reporting chain to understand as you start the measurement journey. It’s pretty easy to talk about CAC and LTV after you skim some Business Insider piece on the way into work.
It is a long, expensive walk to getting to the point where everyone in your organization is armed with that information and it requires investment from everyone to achieve it.
Why Measurement Matters
“You ever see the movie Groundhog Day?”
My colleague and good friend Carter Rabasa were in a heavy conversation as he was coming in on a year into the gig. Being a fan of even the curious missteps in Bill Murray’s catalogue, I replied affirmatively.
“That’s what my job is like. I say goodbye to my wife and my daughters, I get on a plane, I go meet with a bunch of developers, I come back home a week later and I have no idea whether or not anything I just left my family for matters.
“Then Monday comes and I go do it again.”
As he’s done many times of the course of knowing each other, Carter put the real source of the problem in stark relief. Measurement wasn’t about the rest of the company knowing developer evangelism matters.
Measurement is about developer evangelists knowing that developer evangelism matters.
Much is written in the community of professionals serving developers talking about burnout. Measurement is at the core of that problem. If you’re putting any meaningful amount of yourself in any endeavour without knowing – not feeling, but knowing – your impact, that investment will never sustain.
Why should you measure success in developer relations? So you can be happy doing it.
A Framework For Understanding
We’re not happy with our measurement story and we understand why it is important – how do we fix it? First, we need to take stock of where we are. One compass to help triangulate your location on the measurement journey is the DIKW Pyramid of Understanding.
The DIKW Pyramid is a hierarchy for identifying the degree of insight gained from your measurement at the front line of your organization. For our purpose, we need to identify where each frontline evangelist in our devrel crew sits on this hierarchy.
Here are characteristics of insight for an evangelist at each level in the hierarchy:
- Data – quarterly report cards of work and uncorrelated company-wide top line results
- Knowledge – monthly updates correlating online work with easily measurable non-revenue results like signups
- Information – weekly huddles linking work with revenue by product line or revenue type
- Wisdom – daily notifications showing work’s progress against medium and long term goals
To start our journey with measurement, we need to assess candidly where each of our front line individual contributors sits on this hierarchy. With the understanding of where we sit on measurement, we’re ready for step zero.
Starting the Journey
With our candid self-assessment of where we’re at, we can get cracking on starting the journey. There are a few key bits to get in motion right away.
1. Document Every Lead
Every time devrel hands off a lead to sales, get it documented in some format we can quickly reference. Our instinct here as a technologist is likely to go whole hog with Salesforce or some other sort of CRM – that is not necessary at the beginning of our journey. Just a BCC on the handoff and jotting date/time, prospect name and deal value is enough for this early stage.
Lead quantity and quality will be the two vectors by which our sales organization will at some point evaluate the efficacy of our marketing organization. You’ll want to establish a good handle on developer evangelism’s overall contribution here.
Quality will be the important bit at first – the thing we want to hear out of every sales rep’s mouth is “devrel gives the best leads.”
This will be enough for a while.
2. Get Access To Your Web Analytics And Segment Your /docs Users
This is where some degree of business realism should be socialized. Getting exact numbers on the number of developers in a demographic segment your company has connected with is going to be expensive. Coca-Cola, Red Bull, Anheuser-Busch, Monster Beverage deploy hundreds of thousands of field marketing professionals physically handing product to prospective consumers. Measuring their work is a more significant annual investment than the sugar water (or beer) they are handing out.
Measuring multi-channel marketing return is a solved business problem, but the solution so expensive and time consuming it is not manifest in any but the largest software companies.
Good numbers may not be perfect numbers, but they are better than no numbers. Certainty isn’t cheap, but is is also unnecessary at an early stage. Demonstrating percentage growth in a target consumer behavior can serve as social proof to get the more expensive measurement tools and integrations you’ll need down the line.
Hopefully, there is already a Google Analytics or Mixpanel or Heap tag on the site everywhere. As the leader of our devrel squad, you – not some other resource on some other team but you – need a login to that tool.
Next step you will create an audience segment using whatever mechanism the tool allows to show all the users that touched your documentation in some way. Non-technical users are going to be steering clear of this section of your site – this is the fastest, cheapest proxy available for the size of our developer audience.
Once you have this segment, we’re going to be looking for year-over-year and month-over-month growth of this audience. Provided our product is working and solves a developer problem, more developer to the documentation should be generating more revenue for the company.
This will also be the segment that shows spikes that should correlate to your activity. Examples would be seeing a spike in documentation pageviews after a hackathon or a spike in signups after a piece of content trends on Reddit. All these are useful anecdotes to evidence short-term success.
Long-term we want to see our devrel effort doubling or tripling action indicators like signups or credit card swipes. Each of these will require some degree of investment from whomever controls the marketing website. The more we can lend the technical expertise to get this tracking right, the easier that lift will be.
What we’re looking for is 20 / 100. If we have a new developer evangelist in a new city, we want to see our monthly growth in that metro get up to 20-30% so we can see a 100% to 200% jump the following year. If we have a new product launch, we want to see 20% month-over-month growth in docs traffic so we’ll see a hefty starting block of customers by that product’s first anniversary.
3. Get The /docs Growth In Front Of Everyone
Now that we’ve cobbled at least one number we can observe going up and down, our next move is to close the feedback loop. A green or red number sitting in a Quarterly Business Review deck shown to management only isn’t worth the slide on which it sits. More important is equipping the crew with quantitative results that can help them course correct in their daily decision making.
On this score, I think I’ve made every possible mistake – here’s a quick run through the two worst ones.
Bury Meaning In Absolute Numbers
Hopefully, you have some weekly function for the team to do some critical reflection of the week’s work and provide visibility into parts of the organization they might not see regularly. For an embarrassingly long stretch, I would open the meeting with the absolute numbers of some of the initial segments we were measuring. “Hey gang! We reached 1,079 users on the /docs and they spent three minutes on average with another 2,039 showing up on our blog posts.”
After a few weeks of this, it was clear there wasn’t a lot of retention, so I made sure to include it in bullet points for the meeting recap email. “There!” I declared confidently, dusting off my hands like the action hero walking away from the boss fight, “It’s in front of everyone weekly! Alignment is assured!”
Of course, it didn’t work. An absolute number without relative growth is a star without a constellation. Without some structure to anchor that value to, its just one more data point in the ocean the individual evangelist looks up to at the end of the day. Without relative growth, there is no way of knowing if 1,000 is good or 10,000 is good or 100,000 is good.
When growing from zero, directional guidance is more important than absolute certainty. “1,079 uniques” is less informative than “20% growth in uniques.”
Train Everyone On [Web Analytics Tool]
Like most misguided management, it began with the best of intentions. “Right now we have three people who can do this,” I pitched enthusiastically. “What would happen if we had thirty people who could do this?”
Get more eyes in the data and they’ll bring more brains into the data, I reasoned. This is going to be awesome.
Of course, it also did not work. There were three main causes of failure:
- Web analytics interfaces are horrible. It takes half a day just to get to a common definition of terms.
- Web analytics are old. Measuring web performance has been around for three decades and the discipline has accumulated a bajillion metrics, subjecting the novice user to a paralysis of choice. There is no way to tell what is important.
- Most web analytics metrics don’t map to business performance in a B2D context. The big spenders for these tools are consumer brands – the features just aren’t built for a tiny, hyperspecific demographic like the software developer.
So what do you do?
Build Your Own Dashboard
Frustrated with the lack of progress on socialization of measurement inside the team, one of the folks on the crew spent a Saturday kicking out a crappy single page dashboard behind a Google Oauth login. That crappy dashboard would end up being the thing we all looked at when we arrived in the morning and left in the evening.
Some time later I was in a conference call with Eric Ries talking about application of his now-canonical volume The Lean Startup. Expecting some vendor endorsement that would finally get us off our crappy dashboard, I asked him, “What tool are most of your consulting clients using to inform their iteration?”
“That’s a great question, Rob, but you’re not going to be thrilled with the answer. The ones who are successful always end up building their own.”
Put a pot of coffee on, dust off your Heroku credentials and just create a single dashboard your team can use with the 3-4 things that matter right now. All these analytics platforms have APIs for external visualization – dump a few of the things you care about into chart.js and marvel at the progress.
Crawl, Walk, Run
The most difficult step you’ll have in your journey measuring your developer outreach program. The quality of that first effort is less important than getting it behind you. Iterating on the metrics, measurement and course correction that will follow will end up producing more value than whatever imperfect manifestation this first try gets.
This is among the top questions every devrel professional asks – share your spot on the pyramid and what you learned on your journey below.
Serving Developers 