Experience Report in XP2010: Automated Acceptance Testing of High Capacity Network Gateway

I'm going to be in XP2010 and be one of the presenters of our Experience Report. First time for me to publish something. If you are around, please come to say hello :). Here is our abstract:

Abstract.
In this paper we will explore how agile acceptance testing is applied in testing a high capacity network gateway. We will demonstrate how the organisation managed to grow agile acceptance testing testing from two co-located teams to 20+ multi-site team setup and how acceptance test driven development is applied to complex network protocol testing. We will also cover how the initial ideas that we had of agile acceptance testing evolved during product development. At the end of paper we give recommendations to future projects using agile acceptance testing based on feedback that we have collected from our first customer trials.

What Key Performance Indicators (KPI) should Agile Team(s) use?

It would be nice to measure how we are doing and also check, has our newly implemented improvements hit the spot. Metrics are always nice, but (un)fortunately only few of them are usable. Here is a list of those KPIs what I think should be used.

Velocity
How many Story Points(SP) has team consumed during previous iterations and use this information to forecast. Keep your Story Points uncorrupted and here you have good and handy tool. If you are wondering what is SP, you should read Mike Cohn's excellent book Agile Estimating And Planning.

Cycle Time
Cycle Time (or Lead Time), means a time from when a customer request comes into a process to a time when it has actually delivered to a customer. Good way to measure the whole process, not only R&D efficiency. With this, you can check how good your flow is.

Boomerangs
Boomerangs are things, which are coming back to the process after they have declared as done and delivered. Because those things are defects like bugs and misunderstandings in customer requirements, this is a measurement for quality and communication. You can read more about this from Gojko Adzic's blog.

Customer Satisfaction
It's good to check, time to time, what people who are paying our salary think about us :)

Agile Testing vs. Waterfall Testing

We had this small conversation with my colleague, what Agile Testing really is and is there a conceptional difference between these two?

One thing should be common. Both should verify that the customer expectations are satisfied. But when in waterfall way of testing, the testing phase is long and in the beginning of testing condition of software is uncertain, in Agile the emphasis is always on working software, situation where we are each moment should be clear. This is done by using Continuous Integration, with Automated Regression tests.

Finding bugs is one of the main reasons of testing in waterfall model. In Agile, on the other hand, and especially when using Engineering practices like Test Driven Development (TDD) and Acceptance Test Driven Development (ATDD), tests are telling when things are done and bugs are avoided in the first place.

Also iterative way of doing makes planning different. In waterfall, testers have a long period of test planning, but in Agile test planning is made in the same way as everything else, in the last moment (of course not too late :) ) and also it's short term.