The Business Analyst and I had spent months working with ‘business’ people crafting these perfect (giant?) Use Cases. The Use Cases were finally handed over to 2 groups. The Developers and the Testers. The Developers went off to build The World’s Greatest Engineering Feat from The World’s Greatest Requirements, while the Testers were creating The World’s Greatest Test Cases.

This was bad behavior, and here’s why.

• Expensive. After we spent all this time writing requirements, we were now paying to have someone else spend even more time translating the requirements into a different format (traditional Test Cases).
• The developers are working off the requirements document, not the the Test Cases. So how are they supposed to know about all the conditions that need to be satisfied? We tried distributing/reviewing the Test Cases when they were completed, but this was way too late in the development cycle. We were seeing mismatches between the requirements and test cases, and the developers were realizing something they thought was done is going to be tested under conditions they weren’t aware of. Confusion and rework is costly.
• The ‘business’ people never saw the Test Cases because, a) they’re boring as all get-out, and b) they’re typically created after the “sign-off” of the requirements. It’s hard enough to pull them away from, you know, running the business. Business people can’t stand this stuff. “We’re going to do another round of read-the-stuff-written-down-by-the-tech-people? Ug.” If the people responsible for “accepting” your completed work aren’t aware of how you’re testing it, can they, or should they, really trust the final product? If it’s wrong, you’ll have to fix it. Again, more confusion and rework – not cheap.
• Using nothing but the requirements you’re given, it’s awfully hard to get to the point where you’re giving the Testers a build of your application with no bugs in it. And we certainly weren’t able to pull it off. I’ve read, as I’m sure you have, that finding and fixing bugs costs a gazillon-bazillon dollars. Seriously. It’s expensive to find a bug – you’re literally paying someone to write down a problem (after they’ve double-checked that it’s actually a problem), give it to someone else to fix, that person has to make sure there’s a problem (“recreate the bug”), figure out where in the code the problem is, fix it, make sure they fixed it, perform another build & deploy of your application, and have that original person verify that it’s fixed now. That’s kind of expensive.

I refer to the above-described model as the pop-quiz approach to testing, and as you can tell, it’s rather expensive.

I understand the reaction some of you have when you hear criticism levied against the classic pop-quiz approach. “Of course!”, they say, “That’s the point of QA!” To which I reply, I want the testers to find all the bugs, but everyone should know what’s expected of the final product.

Imagine if a Toyota Camry rolled off an assembly line and it took the last person reviewing the car to discover that you can’t flip the a/c to cold while the dash vent is on, otherwise the car explodes. Congratulations. You’ve found a bug. You’re also dead. Why the hell didn’t anyone talk about testing the cold switch like that a little sooner in the process?

People with “Quality Assurance” in their title should help assure quality, not just find all the problems. Pull the Quality Assurance people earlier in the process. Let them help the team and product never have a bug to begin with.

I’ve led teams that successfully delivered multi-million dollar projects, at impressive levels of quality, by doing just that. I put the Quality Assurance people at the beginning of any new feature development. They then work with business people/analysts to create User Stories with Acceptance Criteria for the developers to work from. User stories are comprised of two parts (not including the optional title). A narrative, which is the story sounding part, and acceptance criteria, which replace both traditional requirements and traditional test cases.

The narrative takes on a pretty standard form as seen below. This conveys the actor involved, the statement of the feature and why on Earth you’re building the feature.

As a [type of user]
I want [some functionality]
So I [can get some benefit]

A newer alternative to this format, which tries to emphasize the business value first takes on this shape.

In order to [achieve some value]
as a [type of user]
I want [some functionality]

Acceptance criteria can take almost any form, my suggestion is to go with the most terse and concise form possible. Acceptance criteria can be a sentence, a scenario (my personal favorite), a wireframe (if existing UI standards aren’t enough) – really anything. The only real rule I’m aware of is that each piece of acceptance criteria must be testable/verifiable.

Oh, and I’m going to make another rule right here on the spot. It must be as terse and concise as possible.

Example story:

Title: Forgot password?

Narrative:
As a member of the web site,
I want to gain access to my account even when I forget my password
So I can still use my account, even though I suffer from short term memory loss

Acceptance Criteria:
* Scenario 1: Send new password
Given I have an active account
And I’m not logged in
When I indiciate I have forgotten my password
And I provide my email address
Then the web site sends me a new password to log in with

* Scenario 2: Bad email address format
Given I am trying to provide my email address
When I submit an improperly formed email address
Then I see an error message explaining the format is incorrect

* Scenario 3: Email address not found
Given I am trying to provide my email address
When I submit an email address that the web site does not have on file
Then I see an error message explaining they couldn’t find that email address

Here is a more terse version of the above acceptance criteria.

Acceptance Criteria:
* Take an email address, and send a new password to the email address given
* Show an error message if the email address format is bad or not found

Here is a more formal version of Scenario 2, that works with a certain tool for automated testing.

* Scenario 2: Bad email address format
Given I am trying to provide my email address
When I submit an improperly formed
Then I see an error message explaining the format is incorrect

Examples:
| email address |
| example @example.com |
| .example@example.com |
| example@example.com. |
| .example @example.com. |

I’ve used this format on multiple e-commerce projects inside a Fortune 100 company (on web sites that generate over a billion dollars a year in sales) and a “rich internet application” for a mid-sized technology company with great success. The User Story with Acceptance Criteria format (specifically the scenario format) is even spreading in the tech company to large-scale C++ infrastructure software, not something people typically associate with “User Stories”.

This approach & format corrects all of the bad behavior I’ve seen in the past.

• It’s cheaper. You don’t pay someone to write down requirements, and another person to rewrite it all over again. You can still involve both the Business Analyst and the Quality Assurance Analyst in the creation of the Acceptance Criteria (you can even have the Product Manager write them). But now they’re working toward completing the same document instead writing their own version.
• The developers are working off of the same document as the testers, analysts, and product/business people. Everyone is on the same page – massive reduction in confusion and “misses”, which lowers cost.
• Business people get to see the complete picture, once. And it’s in a compact format that’s easy to digest and written in plain English.
• I’ve seen dozens and dozens of features delivered to the testing environment with zero bugs. That reduces your time to market and drives down your cost of development.

Please help spread the word. Stop giving pop-quizes!

So you’ve gotten a promotion or a lateral move to a new comany. You’ve gone from being a developer, only worrying about the code. Or from managing/directing a department that’s been running well for years, to a brand new team or company, and you need to get your bearings.

There’s a laundry list of things to setup and understand when you’re taking over a development department. This probably isn’t a complete list, but off the top of my head, here’s what I need to know my first week on job.

The List

Which source control tool do we use? (SVN, Perforce, CVS, Visual Source Safe)

What is the branching/merging strategy? How do we handle ongoing development for planned releases? How do we handle an unplanned release? (hotfix, “live site”, emergency bug fix, etc…)

Do we have a CI/CD server?

What is the approach to leveraging CI/CD?

What is the on-call policy/rotation?

What is the SLA for responding to a production issue?

What issue/ticket/bug tracking system do we use? How do we use it? (Issue types, workflows)

How do we estimate our work?

Who has traditionally prioritized our work?

Do we have release plans?

Do we have a standardized release notes format?

Do we include rollback instructions in our release notes?

What requirement and test case format do we use? (Use Cases, User Stories w/Acceptance Criteria, Traditional Test Cases)

How many environments do we have (Dev, Test, Load Test, CI, etc…), and do we have swim lanes for different activities? (Long-running project work vs. any ongoing maintenance)

Can we develop 100% on our local machine? If no, why not?

How frequently do we deliver builds to our testing environment? (And why isn’t it 1 or more a day?)

How much automated testing to do we have? Unit only? UI/Functional?