Day 2: Open vs. Private Beta

product

1/7/2020

I’ve been working on Hide Feed for a bit over a month now (first commit was on 11/27/2019), and I’m extremely proud of how far it’s come. Out of all the random projects I’ve built over the years, I’m most proud of this one.

Last night, I went to sleep with a grand plan in mind for what I’d do today: I’d finally open Hide Feed to the world by posting on Facebook and Twitter and start collecting feedback outside of the few friends I’ve shared it with.

But when I woke up today, a different thought popped into mind. What if I went with a private beta instead of an open one? This was probably triggered by looking through Mailbrew last night (incredible design btw) and seeing that they were using a private beta. Doing that hadn’t really crossed my mind before that.

Here’s how I’m distinguishing between open and private:

  • Open beta: Anyone can install it using a public link
  • Private beta: People have to request access and I’d share the link with them privately

I want to take a step back and start from first principles to try to figure out whether a private or public beta would make more sense right now. What exactly am I trying to accomplish by going into beta in the first place?

Let’s start with why I wouldn’t just launch with what I have. By launch, I mean trying to get as much attention as possible on it by posting on Hacker News, Product Hunt, etc. I want to go through several cycles of catching bugs and making improvements before doing that, because I’m sure that there are weird edge cases I missed, people might have different configurations for which the extension doesn’t work, etc. So launching is a no-go.

In addition to fixing any bugs, I want to collect a lot more feedback and suggestions for ways that the extension can be improved. For example, I didn’t even think to add Amazon as a supported site until a friend mentioned that he’d like to see it there because he gets distracted when shopping. After I added it, a different friend messaged me to tell me that he loved that feature.

So the most important thing for Hide Feed to progress right now is to catch bugs and collect feedback. Would an open beta or a private beta be more helpful for that?

I think it’d be a private beta. Right now, I don’t ask for any information when people install Hide Feed, which is by design because I personally find it annoying when I need to create an account in order to try a new product. I typically bounce right away unless I’m really motivated to check it out. The downside of not asking for any contact information is that I don’t have a way to communicate with the people who are using Hide Feed. One way is to show messages in the feed area that I’m replacing with the image, but I want to use that sparingly because I’m not trying to bombard people with random things in that space.

The advantage of a private beta is that I can proactively ask people for feedback after they’ve had some time to try it since I’ll have their email addresses. With an open beta, I wouldn’t have that opportunity since I have no way to get in touch with them. Time to make a pros and cons list!

Private Beta Pros

  • Have a way to contact users

    • Can ask for feedback directly
    • Can ask people to add reviews on the Chrome Web Store or in a future Product Hunt post if they want to

  • Makes people feel more invested in the product because they applied to try rather than being persuaded to try

Private Beta Cons

  • Fewer people trying it out compared to open beta
  • People are less likely to share the product because they haven’t tried it themselves

Since my goal is to collect as much feedback as possible during this stage, I think it’s pretty clear that I should go with a private beta. Even though there will be fewer people who would be using the product compared to an open beta, I think I’d still be able to get more feedback. That is, I’m assuming that (# of private beta testers) * (% of people responding when asked for feedback) is greater than (# of public beta testers) * (% of people who reach out on their own with feedback).

Anyway, I can always go from private beta to open beta but not really for the other way around so this seems like the way to go.

I decided to also find out how much time I’ve spent on Hide Feed to get it to where it is right now, and the result is….

  • November 27 - 30: 15h 13m
  • December 1 - 31: 109h 29m
  • January 1 - 7: 32h 13m

Total time spent so far on Hide Feed: 156h 55m

Geez, that seems like a lot of time. That averages out to 157h / 42 days =~ 3h 45m of active work a day (with two caveats: (1) I have a tendency to undercount time spent working b/c I sometimes forget to use my timer (2) weekends, holidays, and other days I didn’t work are being included in the number of days so the actual hours worked per day is higher).