What to Build and How to Build It ================================= Mark Ramm: Answering questions with Python Previous SourceForge technical lead TurboGears 2 technical lead Current Juju technical lead Don't waste your life --------------------- - 9 out of 10 projects fail - 9 out of 10 startups fail - Sturgeon's Law! Building products - failing for technical reasons or poor execution - failing because it wasn't actually useful Try to build things people want! Market Research vs Total Bullshit --------------------------------- Not all product managers bother to do research Even when they do, results can be questionable Management culled at SourceForge XKCD: Stand back, I'm going to try: SCIENCE! Test Driven Management ---------------------- In dev, tests provide measurable progress Can you apply the same idea to management? 3 screenshots from SF download page - updated with large download button - redesigned (ended up slightly worse) - current design is actually measured First two redesigns had no metrics. Final redesign had metrics to track how often people made it through to get the files they wanted. Scientific Management --------------------- Document assumptions Work with the team to learn what works Don't just make shit up! Scientific method - ask a question - do background research - construct a hypothesis - test hypothesis through experiment (or, observation if necessary) - analyse results and potentially reformulate hypothesis - repeat! Science requires that your hypothesis is falsifiable. It isn't about proving we're right, it's about proving we're "Not Wrong Yet" (Popper quote) The role of evidence is to: Arguing based on opinions can easily go nowhere. Arguing based on objective data can be Metrics for Pirates ------------------- Dave McClure (these are for a public facing website) Acquisition: users reaching your site Activation: getting visitors to go further Retention: getting vistitors to come back Referral: getting visitors to recommend the site Revenue: getting visitors to pay you! Tools - how do you gather data? Tests - *what* data do you measure? Examples -------- Zappo's: selling shoes on the internet! Took photos of shoes in stores Set up the online store Bought at retail and shipped them Low overhead way to test if people would buy shoes online Return policy dealt with "What if the shoes don't fit?" Figured all this out *before* expending a lot of capital Pets.com: spent the money first, it didn't work! Building a great product ------------------------ Traditional: - Great idea - build it - market it - profit! Not so much: this often fails The Structure of Scientific Revolutions Normal progress (refining the existing models) vs revolution (rebuilding the models on a new foundation) Optimisation (doing an existing thing better) vs Discovery (building something new) Customer development -------------------- Searching for a busines - discover customers - validate customers (will/can they pay?) Growing a business - create customers - scale company Desirable, feasible, viable --------------------------- - is it worth doing? - can it be done? - is there a business model? Figure out which of these is the biggest risk for any idea A "Pivot" is a change in direction during the customer discovery and validation phase Closing ------- More Popper: great scientists have bold ideas, but are highly critical of their own ideas. They try to find whether their ideas are right by trying first to see whether the are, perhaps, not wrong. Q & A ----- Data vs knowledge - e.g. visitor -> download conversion may go down if the search page is bad - can refine an experiment (e.g. only use referrers that are almost certainly trying to download) - balancing trade-offs (e.g. ad revenue at source forge) Low volume testing? - how do you test when you don't have traffic yet? - initial focus needs to be on the acquisition step - need a lot of data to distinguish between 2% and 3% conversion rate - but 0% vs 1% is easier to figure out (and is where you can start) How to explain the need for this? - jackass answer: get the hell out! - explaining the scientific method to someone that doesn't get it is hard - in the discovery phase, "it doesn't matter" is a common answer My Thoughts ----------- Wondering how you could apply this in an intranet context...