What to Build and How to Build It
Mark Ramm: Answering questions with Python
Previous SourceForge technical lead TurboGears 2 technical lead Current Juju technical lead
Don’t waste your life
9 out of 10 projects fail
9 out of 10 startups fail
Sturgeon’s Law!
Building products - failing for technical reasons or poor execution - failing because it wasn’t actually useful
Try to build things people want!
Market Research vs Total Bullshit
Not all product managers bother to do research
Even when they do, results can be questionable
Management culled at SourceForge
XKCD: Stand back, I’m going to try: SCIENCE!
Test Driven Management
In dev, tests provide measurable progress
Can you apply the same idea to management?
3 screenshots from SF download page - updated with large download button - redesigned (ended up slightly worse) - current design is actually measured
First two redesigns had no metrics. Final redesign had metrics to track how often people made it through to get the files they wanted.
Scientific Management
Document assumptions Work with the team to learn what works Don’t just make shit up!
Scientific method - ask a question - do background research - construct a hypothesis - test hypothesis through experiment (or, observation if necessary) - analyse results and potentially reformulate hypothesis - repeat!
Science requires that your hypothesis is falsifiable. It isn’t about proving we’re right, it’s about proving we’re “Not Wrong Yet”
(Popper quote) The role of evidence is to:
Arguing based on opinions can easily go nowhere. Arguing based on objective data can be
Metrics for Pirates
Dave McClure
(these are for a public facing website)
Acquisition: users reaching your site Activation: getting visitors to go further Retention: getting vistitors to come back Referral: getting visitors to recommend the site Revenue: getting visitors to pay you!
Tools - how do you gather data? Tests - what data do you measure?
Examples
Zappo’s: selling shoes on the internet! Took photos of shoes in stores Set up the online store Bought at retail and shipped them Low overhead way to test if people would buy shoes online Return policy dealt with “What if the shoes don’t fit?” Figured all this out before expending a lot of capital
Pets.com: spent the money first, it didn’t work!
Building a great product
Traditional: - Great idea - build it - market it - profit!
Not so much: this often fails
The Structure of Scientific Revolutions
Normal progress (refining the existing models) vs revolution (rebuilding the models on a new foundation)
Optimisation (doing an existing thing better) vs Discovery (building something new)
Customer development
Searching for a busines - discover customers - validate customers (will/can they pay?)
Growing a business - create customers - scale company
Desirable, feasible, viable
is it worth doing?
can it be done?
is there a business model?
Figure out which of these is the biggest risk for any idea
A “Pivot” is a change in direction during the customer discovery and validation phase
Closing
More Popper: great scientists have bold ideas, but are highly critical of their own ideas. They try to find whether their ideas are right by trying first to see whether the are, perhaps, not wrong.
Q & A
Data vs knowledge - e.g. visitor -> download conversion may go down if the search page is bad - can refine an experiment (e.g. only use referrers that are almost certainly trying to download) - balancing trade-offs (e.g. ad revenue at source forge)
Low volume testing? - how do you test when you don’t have traffic yet? - initial focus needs to be on the acquisition step - need a lot of data to distinguish between 2% and 3% conversion rate - but 0% vs 1% is easier to figure out (and is where you can start)
How to explain the need for this? - jackass answer: get the hell out! - explaining the scientific method to someone that doesn’t get it is hard - in the discovery phase, “it doesn’t matter” is a common answer
My Thoughts
Wondering how you could apply this in an intranet context…