Day 116: Calculating Time Saved

product

Say you’re me. You’re working on Intention. You also may or may not have Lyme disease, but that’s not important right now. Back to Intention. You want to implement a feature that allows users to see how much time they’ve saved with Intention. How would you do it?

Let’s start by defining “time saved”. We can say that Intention saves someone time when they would’ve spent more time browsing a distracted site but didn’t because of Intention. This could happen in various ways (e.g. having a timer that counts down their remaining time might make someone more selective about what they browse; setting an intention to browse for a short period of time might curb their desire), but many of them are difficult to detect. Thankfully, one of Intention’s central mechanisms can be detected. It’s when Intention pauses someone’s browsing with the lock popup:

This stops mindless browsing in two common scenarios:

  1. Habitual navigation

    Someone reflexively types in “t”<enter>, ends up on Twitter, and then realizes (thanks to the pause created by Intention’s lock) that they actually don’t want to scroll through their feed right now.

  2. Mindless browsing

    Someone unlocks for 1 minute, starts scrolling, and then has the lock activates at the end of their session. Without Intention, they would’ve continued scrolling for much longer, but thanks to Intention’s intervention, they go back to work instead.

Since my goal isn’t to get things exactly right but rather to offer a reasonable ballpark estimate, I’m not going to try to figure out exactly how much time they would have spent if they had gone on browsing. That would require basically building a predictive model using signals like their past behavior, the site, the type of content on the site, etc. to estimate whether they would have spent 3 minutes or 3 hours. For simplicity’s sake, we’ll estimate that 5 minutes is saved whenever one of the above scenarios happen.

Great, so how do we detect when these scenarios occurred? In other words, what’s the condition that would tell us that Intention successfully intervened?

Let’s try this one:

A successful intervention occurs when the lock activates and the user closes the tab.

While this condition is simple and clear, it fails a common scenario: Instead of closing the tab, someone might switch tabs to get back to work. In that case, Intention successfully intervened but it wouldn’t be counted.

Let’s expand our condition from “closing the tab” to something a bit broader to capture more cases where Intention helped someone stop browsing:

A successful intervention occurs when the lock activates and the user doesn’t unlock the site.

Instead of trying to define the success case like the previous condition, we can define the failure case and take the inverse of that. If someone unlocks the site, the lock clearly didn’t stop them from browsing; but if the lock activates and they don’t unlock the site, we can reasonably assume that the lock helped them stop.

This is a nice, clean condition, but of course, there are additional details to consider. For example, consider the following scenario:

  1. Someone unlocks Reddit for 5 minutes
  2. They open three threads in new tabs
  3. At the end of the 5 minutes, Intention’s lock activates

If they don’t continue browsing, should that count as one successful intervention or four (one for each tab)? An argument for the latter is that since each tab would’ve been a source of distraction, each one should contribute to time saved. But intuitively, that doesn’t seem right. If you have 10 tabs with distracting sites open, it seems ridiculous to say that you save 50 minutes every time the lock activates. It makes more sense to consider time saved by looking at sessions rather than tabs.

So finally, we’ve arrived at a way to measure time saved:

  • A successful intervention occurs when the lock activates and the user doesn’t unlock the site

    • If the lock activates on multiple tabs, only one counts

  • For each successful intervention, we’ll estimate that the user saved 5 minutes

I don’t know who in their right mind would find the above interesting, but if you were curious about how Intention calculates “time saved”, this is how I thought through it!

Update (5/6/2020): I’ve since changed how time saved is calculated due to some undesired effects of the way I was doing it above.