The Minimum Viable Beta Program, and how to build it.

photo courtesy of
photo courtesy of

Let’s say you’re starting a new company, feature, or product. You have an idea that you want to test with some beta customers. So what would you do today to “get out of the building” (in best Steve Blank Lean Startup style) once you’ve done some initial customer development to determine your Minimum Viable Product and have some ideas and data about the kind of customer who might use or provide feedback on your idea? One way to do this to set up a beta program, where you combine your ideas, your preliminary feedback (and some actual people) to see what will happen. (The feedback will likely be both sweet, and a bit tart.)

What are the goals of a beta program?

At face value, the goals of a beta program seem simple: find some potential customers, ask them to use the product, identify bugs, and get feedback on what’s working and what’s not working in a defined period of time. “Customers” probably look the same as the initial persona you identified during your customer development phase, and since you’re looking for a directional indication at this point, you don’t have to get them completely right (the beta program is an extension of your customer development efforts.) “Using the product” and “defining bugs” will work best if you define a few tight scripts at first to help people understand your vision of what they should be doing, and not just what it looks like in your prototype. And “getting feedback” means finding out the most important thing to improve or fix at that point of your development process.

The Perils of Talking to Customers You Know

Finding potential customers is the first step, and also not the easiest. When you start, you’re likely to ask people you know – which is great because they will be more receptive and accommodating of problems, and not so great because they’re biased to give you good feedback – so you need a mix of people you know and don’t know. One way to solve this problem is to ask the people you know to recommend 2-5 people that they know who will provide practical feedback and who don’t know you all that well. Once your group reaches 30 people, you’ll be able to know more confidently that you have at least directional statistical information.

Practical tip: manage the list of customers in a Google Spreadsheet, identifying their name, email, Twitter handle, external ID in your system, “customer type” (this could be ‘experienced, noob, mainstream’ or another taxonomy), the date they joined the beta or the identifier for their cohort group, a comment field, and a “last contacted” date. This will allow you to model the list of beta customers by cohort, give you a way to communicate with them, and provide you with a data structure you can use to pivot their feedback by customer type, externalID, beta cohort, and date range.

Using the Product is Not Following The Instructions

It’s tempting to think that customers (any customers) will “read the manual” and follow your instructions to the “T”. Well, when was the last time you read the manual? It’s important when considering your beta group to include both those people who are likely to follow your instructions, those who are not very imaginative and who just want to “hire your product” to do a job for them, and those who want to break your product or will think of unusual ways to interact. And if you want really great bug descriptions, you need to make it really easy for customers to provide inline feedback and to give them a prompt every time to identify the key items you need to understand what’s going on.

Practical tip: provide instructions for your scenario using multiple learning styles, including listing items in an ordered list (“Step 1, Step 2, Step 3”), asking an open-ended question (“What’s the 1 thing you’d like us to improve”), and communicating in other media, e.g. asking them to record a Skype conversation, a screencast, or setting up a group Google+ Hangout to discuss the “how do you do it” aspect of your product.

If you’d like to use a survey, here’s an example of a few qualitative questions you might ask.

You have feedback, and now what?

The “90-9-1” rule for participation inequality suggests that some of your beta participants are going to give you a LOT of information, and the feedback they provide will be overweighted toward the one or five or ten people who feel really passionate in your first group of 100 or so beta participants. So how can you take what they’re telling you and make it more meaningful? First, you should identify the functional bugs they point out, and fix those: these are potential blockers for any new customers. If you’re not going to fix an issue, log it and let the reporter know you’re placing it in a lower priority queue (and then once every month or so, prune the queue aggressively to remove the “not-urgent not-important” items.) Second, you can use a Kanban or other scrum technique to organize the volume and priority of the work (Asana, Trello, Do, Jira, Pivotal, and others are all good for this task). And third, keep asking your beta testers short surveys frequently (1-3 minutes) to see whether what you think you’re doing is actually working for them.

Practical tip: use some bug tracking software to manage this process. Don’t reinvent the wheel: if you don’t like Jira, Bugzilla, or another solution you can always find a bug tracking template in another tool. And make sure you identify the type of customer for which you’re solving the problem; the severity of the issue (does it stop them from using the product; is it a major deficiency; or is it just nice to have) and the priority (get it done now, get it done soon, log it to see if it becomes more important later).

What’s realistic to expect from your results?

It’s reasonable to think you’ll get feedback, and that some of your customers will like (or even love) what you’re working on for them. And it’s also likely that some will dislike, or even hate, the thing you think is cool. To get the most mileage out of this feedback, use a dedicated email alias or distribution list (e.g. beta-feedback@) to share this information with as wide a group as possible within your company. And view the feedback in the context of the type of customer who’s providing that feedback. When you get positive feedback clustered among multiple people in the same persona in the same area of your product, that’s a good sign you’re heading in the right direction. And then use that feedback as a data input to make decisions: what kind of company do you want to be? Which customers are you serving, and will you serve them better by improving this feature or fixing this bug? If the answer is yes, it’s time to invest time and money in making that change.

Join the discussion on Quora.

You don’t need a business plan yet: 7 ideas for Customer Development

You’re thinking of starting a business? Hooray! Cool idea, and good luck. There are a few things that you can do to get started, and the first ought to be something counter-intuitive: don’t build your business plan. Yet.

The first thing you’ll need to do before you decide to really go for it is to figure out who your customers are and to learn more about what they need.

Then, try out your ideas on them.

Prepare a Pitch

What would someone like to know about your business in a very short piece of information? When someone says “What is ….” if you can respond with [my widget or service] is … and make it short, compelling, and interesting then you’ve gained their interest in having a larger conversation. Here are a few tips.

Go Practice

If you think your pitch is good right now and you’ve never delivered it before, you’re probably wrong. A great way to find the holes in your customer pitch is to tell about 40 people about your idea. You can use friends, family, or a coffee shop to get going. A particularly good place to do this is at an industry event or “speed networking” night.

Make a Survey Based on What You Learn

Now that you’ve tested your idea, make a survey (a short one will work better – people hate long surveys) and try to get answers to the 5-10 questions that will help you to move forward. There are great tools to help you to do this, including Google Docs, Wufoo and Survey Monkey. Get at least 30 people to answer your survey and you’ll be on your way to getting some actionable data.

Build an “Up and Stumbling” Prototype

When you think you have a better idea of what to build, go ahead and build a prototype – it doesn’t need to work but it does need to share the essence of your idea quickly (if you’re a business person, code in index cards or code in Powerpoint. If you’re a dev, build in whatever language you like that’s fast.) Balsamiq is a cool tool for this purpose, giving you enough information to show what you want to do, but not limiting you by creating an enormous prototyping framework.

Talk to Customers and Get Their Best 1 Piece of Feedback

People love to talk about your idea when you get a chance to ask them what they think. Because that feedback doesn’t always cost them anything, they might not focus on the one thing that matters to them about your product or service. So ask customers for their best 1 piece of feedback, not every piece of feedback they have – this will challenge them to refine their advice and you’ll have a better shot and finding out what really engages or bothers them about your idea.

Now, think about your Business Plan

Once you define your customer, pitch your idea, learn and build a prototype, and get some feedback, you’ll be a lot further along in the information you’ll need to build your business plan. Steve Blank lists some great tools for startups that will help you in this effort.

If you don’t launch the product, the kitten gets it!

Office kitten

Originally uploaded by gregmeyer

Sometimes, the world of the lean startup feels like the world as experienced by a kitten — everything’s exciting, we make big movements and try to catch every shiny thing that comes our way. And other days, we just have to take a nap.

Having just been through a couple of hectic sprints, I had the chance today to take a breather and think about the last couple of weeks: some big launches, exciting days, long nights, and great teamwork. The key factor in all of this? The ability to maintain the excitement that our one-day office kitten showed when he sat on my keyboard (all of my emails that day looked like erhi;wr;ehpi4t43tp) and the discipline to make sure that the product still got done and shipped.

So, the idea for the day? Keep the kittenish wonder; pounce on some stuff even if you’re not sure you can make the leap, and make sure to catch the mouse.

Create a free website or blog at

Up ↑

%d bloggers like this: