The Founder Product Analytics Bible

How to truly understand how users use your product, and how to take action on it.

There are many surface-level guides online about product analytics, but over the 3 years of building products people love there are a few things I’ve observed:

These resources don’t give you anything actionable about approaching analytics (ok I got info, now what do I do?)

These resources are fluff, they don’t tell you how to approach the analytics that REALLY matter

These resources are typically written by people that have no skin in their analytics - e.g. their identity is not based on whether that line moves up or down

As an early stage founder, chances are you entire identity is wrapped up in the success of your startup. This means that those simple lines, whether you calculate your retention and NDR (net dollar retention) correctly, is the difference between losing your identity and building a life-changing product.

Employees simply don’t have this skin in the game, if they don’t have massive stake in the company, taking (next to) no salary, and don’t get overwhelming anxiety every time your charts dip then they simply will not care as much to understand the product.

It’s analagous to parents vs. baby-sitters: Yes, both of them don’t want to let the child die. Chances are the parents care particularly more about that then the baby-sitter, as the baby-sitter can just go find another child to care for (assuming it was not their fault).

Somewhat macabre analogy aside, this document serves to aggregate all my learnings from building multiple products that hundreds to hundreds of thousands of people used every day!

Let’s jump in!

Preparing your product analytics

Before you even take on customers, I believe proper instrumentation and collection is a pillar of early stage product development.

Put into your engineering culture to collect events everywhere, and make sure all contributors understand why this information is useful. Otherwise you will get people just slamming JSON into events that become useless later, copy-paste typos, and more.

Tools

At a minimum, you should have a simple product analytics tool like Mixpanel, and a data warehouse like BigQuery.

I’m partial to using more arcanely optimized processes such as IceDB and python notebooks, but for now the data stack we will discuss will be Mixpanel, BigQuery, and Colab (python notebooks).

For collection I typically suggest segment. This is nice because it has a standard api that I can shim in place of segment for use with things like IceDB, as well as proxying and manipulating settings responses to allow for sending through your domain (without their help).

Instrumenting

Ensure that events are useful too. They shouldn’t be Settings page changed with a JSON payload of the entire settings page state. They should be more specific such as Setting toggle changed with the properties of setting (what setting changed) and state (did they just turn it on or off). A Settings page saved could include the total state, but you really should be able to derive that from the preceeding events.

However, keep in mind that doing this wrong can do more harm than not doing it at all. Working with incorrect data (e.g. misnamed events due to copy-paste) can permanently plague data, and greatly mislead analysis!

📋

Summary: Build instrumentation into your engineering culture, and use the proper tools at the start. This will save you hours of engineering time and headaches down the road. Improper instrumentation is damaging as bad data = bad analysis = bad decisions.

Segmenting Users Properly

It’s important because of aggregation bias

Superhuman does a good job of explaining this

Segmenting should be done by customer type, but also customer behavior

Do you notice non-native speakers have a particularly hard time? Work on visualizations
Do you notice people that don’t use a feature have lower retention? See if you can get them to use that feature (without forcing them and poisoning your metrics)

Example of segmenting (show examples of data from colab to prove point)

By ACV range (type)
By streaming language (type)
By whether they create custom interactions (behavior)
By whether they turned auto-add off or on (behavior)

📋

Summary: It’s cricual that we look at product users in terms of properly defined segments, both in terms of user type (things they don’t necessarily control) and activity (things they do). If we don’t, then we fall prey to aggregation bias.

Useful Retention Charts - Do users stick around?

Step 1: Properly defining retention

Retention is not people logging in.

Retention is not any event from a user.

Retention is entirely based on the thing that you want your users to do.

For the Stripe example, they will have many different retention charts for each of their products. For payments, they will look at payment processed events. E.g. a user is retained when they process payments

The simple ones you get in tools like Mixpanel are just fluff, they’re not helpful.

You NEED to be breaking down your retention by all kinds of channels to find what user segments are having the worst experience. Define what retention

Net Dollar Retention - How much are you making?

Platform-level vs aggregating on users (mean vs median - maybe do ☝️ callout on this)

Relative vs absolute (based on the first interval or the previous interval)

Share some sql for this

Taking Action

The best time to encourage your users to do something is within the first few days of getting started.

Whether that be preventing churn, nudging them towards features that correlate with higher retention, portraying functionality that’s often asked in support, you want to get on top of it EARLY.

After the first 3-5 uses or so, they start to become dogmatic about thier usage and have either decided “I don’t want to use this” or “this is how I will use it”, and any help from you will be seen as marketing tactics responded with a “fuck off” of varying respectfulness.

Automated Emails

Automated emails are a peculiar opportunity.

On one hand, they can look like total useless dogshit and dehumanize the user experience. On the other hand, if used properly, they can move mountains in both retention and customer relations.

How to identify when people need to be sent an email, and automate the hell out of it (don’t bug them though)

Know when to send a platform email, and when to look like a founder

Professional Persistence: If sending from you, bug them respectfully a few times (you will get some shade)

Myth: People who respond are representative of the whole. Fact: People who respond are typically either extremely angry at your product, love your product, or are email addicts. The responses will be biased so only use them as indicators, not representative signals

Example: Detecting churn

Biased ex: churn detection email
Good ex: kick detection

It’s not useless, as you may find users where something doesn’t work reliably, confusing settings, and more. But don’t take the feedback of one as representative of all churned users.

Nudging users in-product

Another spectacular place to encourage users to explore your product is nudging them when they use it.

Whether this be a blocking modal for a required action, non-blocking tooltips highlights navgiation buttons and features, or toasts that notify them of semi-important events like releases or payouts.

This is never an excuse not to talk to users

Whether you’re inspired by the low-level product person from Duolingo writing to 37 people on Medium about UX optimization, or doing analytics makes you feel like a real startup business founder person, the best thing you can do is just talk to users.

Humanity is lost through staring at events, and the most valuable way to collect data hands-down is to just talk to users.

But be careful, don’t bias your discussions! That dragon of a topic is outside the scope of this piece, so I’ll leave you to your curiousity for that one 😄