Developer Fail
As you can see from the date of this post and the date of the previous post (about 5.5 months between them), my goal of getting AuditShark to alpha status at the end of July was, needless to say, a bit late. In fact, I wasn’t particularly thrilled by the sound of that self-imposed deadline whooshing by. Looking back at it now, I realize that it was inherently unrealistic to even attempt the end of July to get to Alpha.
The thing is, I’ve only ever been able to dedicate about 5-10 hours/week to the software. I’ve taken a couple of weeks off here and there which really helps with productivity, but it takes a lot longer when you’re only squeezing 5 hours of coding each week into a product. But extrapolating that back to July, that means that I likely did around 110 hours of coding. That would be 5.5 months * 5 hours/week =110 hours. There were two weeks I think where I spent about 40 hours because I took time off from consulting, so it was closer to 190-200 hours total. Had I been working on it full time, then I would have met my goals rather handily.
But I wasn’t. And I knew I wasn’t working on it full time. Perhaps I felt I would get more time into it, but with all of the other things I’ve been doing, such as the Micropreneur Academy, working on MicroConf 2012, my podcast, other products I’ve been putting maintenance time into, various work and family obligations it just wasn’t going to happen. Again, I should have known better, but I was trying to push the envelope of time and that was just unrealistic… or dumb, depending on your point of view. I’ve discussed this to a large extent on my podcast, Startups for the Rest of Us and have provided some sporadic updates via Twitter so it’s not a big secret or anything. I haven’t blogged about it because blogging takes time. It’s a lot more time consuming to write a blog article than it is to just talk about it on a podcast for 45 minutes. There’s one problem with this.
However you look at it, the result is the same. Developer Fail.
But there is a bright side of things.
As of last week, I finally reached my goal. AuditShark is far enough along that it has reached an Alpha stage, or MVP (Minimum Viable Product) as I referenced via my Twitter feed. In fact, I’ve got it running in my lab environment full time and piping data back out into the cloud so that I can do reporting. I also have a volunteer who has offered to put it in his lab environment as just as soon as he gets back from vacation, which I expect to be January 2, 2012.
No income yet, nor do I intend to try charging people until after I get through some sort of alpha testing. In fact, I will be relying on manual billing for the time being until I get enough customers where automating it is necessary. The current problem I’m wrestling with is that I haven’t reached a solid conclusion about what I consider to be the success criteria for a successful Alpha testing phase. Probably just that nothing really major breaks. I need to get some feedback from my Alpha tester to establish what they view as necessary for them to make full use of the product. But I think this is a problem that every software entrepreneur must address.
The question becomes: when should you convert from alpha to beta, and from beta to paying customers?
I have some ideas in my head, but if you have any thoughts, please leave them in the comments.
Your last question is the million dollar question we are going through right now.
If you check out Lincoln Murphy’s site, http://sixteenventures.com/, he seems to talk a lot about free trials and SaaS pricing though there is a little on beta testing.
I would love to see someone right about a more rigorous process for Alpha -> Beta -> Live product process. So if you discover something a better than just shooting from the hip make sure to write about it.
I’ve seen his site before, but you’re right. It’s a lot more about trials and pricing. Those issues are a lot easier to figure out when you have customers coming through the door because you can review the customer analytics to help figure out what to do and when to do it. You can tweak things and do tests.
The issue of going from Alpha to Beta, to production is more challenging because there’s a lot less to measure. It’s difficult to know whether something is failing or not. Not all of the log files are local for me and even if I could pipe them back, does having no log files mean that I’m good, or are they not getting there at all because of an outright crash on the endpoints?
Yeah the only thing I know to do is be super hands on with early customers. Try and email them every few weeks to check in on problems, educate them on how to use the product, and try and keep analytics on product usage as best as you can.
You probably won’t get any strong quantitative results from the analytics, but you’ll be able to track users individually and since you have an email relationship you should be able to anticipate problems.
As the big problems start to go away you’re probably close to a launch.
This is kind of the process I’m using. Not sure it’s the best, but it has seemed to work alright so far.
I think a pragmatic approach is a good one. Move from alpha to beta when the product is feature complete for this release. This helps customers know what this release of the product really do. Move from beta to paying customers when this release’s high and medium bugs / issues have been fixed. I think the question is how many customers can you manage in your beta effort? For a complex product such as Audit Shark, I would consider a roll out approach to the beta. This would help you to learn from real customer experiences and not become overwhelmed with support requests.
Mike, first off, congratulations. One question: is just one alpha tester really enough? Before I released my product, I had about 8 people pounding on it. Perhaps I was being a bit too worried though.
Thanks Joel,
I do not think that one alpha customer is enough. I’d prefer to have 5-10 customers, with an average node count of about 50. That would mean I should have 250-500 computers reporting results every single day. I’m “mostly” confident the system can handle it, but I do intend to add more alpha customers as my own confidence in the system builds. I have yet to do any major stress testing on the system, so I have some hesitation on adding too much too quickly. I anticipate that there will be some configuration changes I will need to make here and there to assist with scalability issues that arise.
When to move from one stage to another? Does there really need to be “named” stage?
When we started, we had a few customers testing our product. Great to have a few of those. We quickly put it together, got it into customers hands, when they stopped asking questions and submitting requests/etc we made it public. Then started charging, improving features, adding more functionality. Every app is different not only because of what it does, but its customers. If customers are fine with the current features/functionality, that’s when you should move on. At least that’s what we did.
As to manual billing… Look at something like Freshbooks. Use their API and their recurring billing to get things off the ground.
I don’t think it needs to be a named stage unless it’s closed to the general public. Once you open it up, then basically it moves from one stage to the next, regardless of the label you’re mentally using.
I already own Quickbooks, but it might be easier to use Freshbooks to keep track of it. I’ll make that decision when I get there.