| HN Mirror

The kinds of "test" and "verification" automation I've done in the past are vastly different from the kinds you'll be doing. Though I've never personally faced the really extreme needs (read: "requirements") of diving physical test equipment with a Real-Time OS to get sub-millisecond resolution, or need to resort doing the testing (read: "verification") through simulation for even faster resolutions, the general concepts of "Quality Assurance" will still apply to your situation.

1.) Always show up with solutions, not problems.

With good reason, Rule #1 is the first thing to always remember. You never want to be the person who impedes development or release. You always want to be the person who improves development and release. By the nature of you job, you will be the person who often needs to deliver the unwanted bad news, but if you gain a reputation for delivering the bad news with some good news, like ways around a problem, then people will seldom dread talking to you. Take responsibility for improving the product, offer fixes, and still offer fixes when it's not your job to fix the bugs.

Never utter the words "show stopper." Even if you are right, it may not be your decision, and you can easily make enemies of the people you need to work with every day. Instead educate others on the potential harm to customers, potential loss of customers, and if that doesn't work, the potential harm to the viability of the company as an on-going venture. When you need to do this, and you will need to do it, be prepared with multiple paths for working around or fixing the issue along with a cost estimate for fixing the issue. In other words, lead others into uttering the fateful words "show stopper." Surprisingly, you job is to make the risks and consequences known, as well as provide alternatives to mitigate the risks and avoid the consequences.

Pointing out mistakes is always a touchy situation. Many people react poorly to being told that something they did is wrong, so try to memorize the secret formula, "We can improve X by doing Y to avoid Z." The "we" is important and you can even toss a "probably" in there somewhere for added effect.

If you can reliably remember the secret formula even when your scalp is sore from pulling out all of your hair, then please tell me how. ;)

2.) You will always have constraints, so know and memorize them.

When an executive spends $200K on a piece of test equipment, and the other engineers want data at some super fast resolution beyond the capacity of the equipment, you are the person responsible for knowing the constraints of the test equipment. This has actually happened to me, and it's a whole lot of no-fun. You get stuck between a rock (the other engineers) and a bad place (the exec who doesn't want to look bad for buying the wrong/cheap equipment). Sure, it may seem anecdotal, but you'll be surprised how often you are asked to do the impossible.

When you know the constraints of your test system, then you can show up saying, "We can do X with what we have currently, or you can push upstream for more investment in test infrastructure. We try doing X to see if it will suffice for your needs?"

In your situation with web apps, particularly mobile-ready web apps, testing will require a big investment in test infrastructure. If your web app is using any of the newer direct-to-hardware (WebGL, AudioAPI, ...) features, just using emulation (system/browser images with VMware or similar) may not suffice to give proper test coverage. Emulated hardware is never perfect, so you'll often need access to real hardware. Of course, you can often do tons with just emulation, but knowing where emulation will fail in strange and unexpected ways means knowing the constraints of emulation.

Knowing your constraints is knowing what you can actually test. When others have unrealistic expectations, knowing your constraints puts you in charge of the negotiations... --And it's always a negotiation. If need be, keep a constraints cheat-sheet around with the details. You'll be surprised how often it comes in handy.

3.) Within your now known constraints, define the specific capacities you want to verify, and the means to verify them.

Once you know the desired capacities and the means to test them, the automation of verification becomes a whole lot easier. Your existing bug tracking database should be a good place to start for building up your regression tests, but preventing the reappearance of old bugs is only a small part of the problem. The majority of the problem is finding the never ending stream of new bugs.

Since browsers and operating systems are constantly changing, you never have a solid foundation. This means your test environment requires extremely strong versioning to manage the vast multitude of browser, OS, hardware and version combinations. It's important to realize how automatic updates are your sworn enemy. They will hose your test environment since if you don't know what you're running, then you don't know what you're testing. Of course you'll test the latest and greatest versions of everything, but each new update will be a new environment version that needs to be preserved.

Environment versioning is a real pain. The number of environments you'll need to manage and test will grow exponentially, so with the always limiting resource of time, you'll not only need to automate as much as possible, but you'll also need to do triage. Testing every possible combination is intractable.

Many sites will try to avoid the pain of environment versioning by only supporting the most current version combinations, but in doing so, they only avoid the expense of supporting "legacy" combinations. There will still be plenty of combinations of "newest" so their test infrastructure will undoubtedly fail to give proper coverage on some of them.

Virtual machines, emulation, simulation, system images (snapshots), and similar along with your favorite version control system (git, cvs, svn) will really help to manage the env versioning issues. It can be a ton of work to get things all set up the way you want them, but the benefit of being able to give good and repeatable coverage makes it worthwhile.

Personally, I'd go with open source as much as possible since it will use a normal programming language. It will mean recreating the wheel in some situations, but for me, it seems better than the alternatives. The generally known open source web app testing frameworks are Selenium [1], Watir [2], and RobotFramework [3]. If you hate real text editors and prefer to suffer from using an IDE, some IDE's have plug-ins like CubicTest [4] for Eclipse which can drive Selenium/Watir.

[1] http://www.seleniumhq.org/

[2] http://watir.com/

[3] http://robotframework.org/

[4] http://www.cubictest.org/

Vendor lock-in from proprietary test suites and proprietary languages can be immensely aggravating and expensive. You might have the budget for expensive proprietary test suites, or you might not, but if you ever need to do something out-of-scope or need to change vendors, then you'll be proverbially stuffed. Sadly, I don't know the proprietary tools at all, and worse, I'm against them without adequate information. Yes, I suffer from the usual "I can do it" bias and personality fault, namely valuing my time less than valuing my money. Your situation may be different.

There are a quite a few different "Software Test" and/or "Quality Assurance" groups and conferences around. As you might expect, they often favor proprietary test solutions since it's usually the vendors of said solutions footing the bill for the conferences. Even with the deck stacked in favor of the proprietary solutions, the groups/conferences can be a good place to learn. EuroStar is one common conference but there are many others if you look around. When looking up EuroStar, I found a writeup from last year that you might want to look over:

http://www.eurostarconferences.com/blog/2013/7/25/comparison...

If you've ever seen the musical called "The Wiz" or even the movie of the same name, try to memorize the lyrics of the song, "No Bad News". If you can learn to sing a reasonable A Cappella rendition of it, even better. It will definitely come in handy, and more often than you'll want to admit. ;)

http://www.youtube.com/watch?v=pQT-QFy5Nig

Good Luck!