The Lazy Dev's Guide to Testing Your Web API ============================================ Ryan Kelly Love/Hate Testing? ------------------ Quality assurance = diminishing returns Early testing efforts rewarded with great improvements Tests get harder, improvements get more obscure Process improvement! Better testing tools to do more with less effort Invest in laziness! Be pro-actively lazy! (And I clearly need to re-read Moving Pictures!) What does Ryan know? -------------------- - Mozilla web services (Firefox Sync Server) - Tools (WebTest, WSGIProxy, FunkLoad) - enthusiast, not yet expert WSGI ---- - webtest = WSGI for humans - drive a WSGI application in a temporary in-process environment Demo ---- Using webtest to ensure the API operates as expected Many helpers to perform useful function tests WSGIProxy --------- In-memory WSGI "app" that maps WSGI requests to HTTP requests Can easily run webtest based functional tests against an out-of-process web server Repurpose functional tests as integration tests and deployment tests Can pick up anything that goes wrong during deployment Can use as a basis for simple load testing FunkLoad -------- Wants to be a monolithic "do everything" test framework Standard mode of use requires that tests be implemented specifcally for FunkLoad (this annoys Ryan, just demoing it to illustrate the point) Can gather benchmarking data Use benchmarking tools, as they deal with problems you may not have thought of yet (e.g. only recording data while under full load, ignoring ramp up and ramp down periods) Nice report generation tools FunkLoad + WebTest ------------------ Write an adapter between webtest API and FunkLoad API Can then reuse *any* of your webtest based tests as a FunkLoad test Can use to get differential results to see how your response times are going Distributed Mode ---------------- Spin up clients on multiple hosts Aggregate benchmarks from multiple clients Other Approaches ---------------- Write FunkLoad tests, use WSGI-Intercept to redirect to in-process WSGI application Q & A ----- Just some specific questions about FunkLoad and making in-process WSGI calls on a live application My Thoughts ----------- Not a lot to add! Some of the ideas here seem similar to those in django-sanetesting, but it in a way that doesn't drive the inheritance model of the test suite. I may look at migrating the PulpDist tests over to webtest (depending on how complex it is to get the in-process Django environment up and running)