Community Posts

When Will You Make an End?

By Dave Whalen

Well?

If I only had a dollar for all the times a project manager has asked me that!  In the classic movie "The Agony and the Ecstasy" Pope Julius II (played by Rex Harrison) stops by the Sistine Chapel on a daily basis to ask Michelangelo (Charleton Heston) "When will you make an end?"  Michelangelo responds as most test managers do when asked this same question:  "When I am finished!"  Four years later, the ceiling of the Sistine Chapel was finished.

How many times have you responded the same way?  A primary responsibility of any test manager is to communicate test status to anyone who will listen.  OK I exaggerate a little.  How about those who need to know - or even better - want to know? 

We never have enough time to do all the testing we would like to do.   As a test manager it’s important to know when to stop testing.   Even more important is letting others know when it’s time to stop. 

Unfortunately, we will also never find or fix all the bugs, nor is it economically feasible to do so.   There will always be bugs, but is having bugs enough to prevent delivery?  Nope.  Even if we could find and fix everything, the final product would be so expensive that no one could afford it.  So how do we strike that delicate balance between cost and quality?  One way would be to "set the bar" early in the development process.  That bar is the "Quality Bar".

What is the Quality Bar?  The Quality Bar or "Q-Bar" is a metric that I typically use on all test projects.  It’s primarily used to communicate when testing is complete from a defect perspective.  A good test plan will typically define the "exit criteria" for a given test phase or cycle.  This exit criteria defines the conditions under which testing is considered complete.  The Quality Bar is essentially one measurement of exit criteria.  You may need others

The Quality Bar is a measurement of defects.  It is best combined with additional metrics such as Test Progress.   I like to call them "Readiness For Release" metrics. 

In many cases, exit criteria are defined in terms of defect rates and test case completion.  For example: "100% of all Test Cases must be executed with no open Critical or Severe defects".  Or you may define your open defects in terms of Priority.  The Q-Bar measures the defect side of the equation.

Presentation of the Q-Bar may vary but I typically report it as a trend over time.  I plot the open defects, by severity over time for a given test cycle.  I use the days or dates of the current Test Cycle on the X-axis, and the number of defects on the y-axis.  Then I include a trend line for each severity level.  If only Critical and Severe defects determine the Q-Bar, only show those - the other are irrelevant.  Personally I like to use Microsoft Excel to build the metrics.

I update the metric daily and publish it on the QA page of the company Wiki, on a bulletin board or tape it to a wall (the developer’s area is a great location).  I also e-mail it to key stake holders and the team as well.  I also bring copies to defect review meeting if I begin to see any adverse trends (like the number of defect going up instead of down). Now, everyone knows where we are and can get some idea of when we’ll be finished.  The best part is that we can catch adverse trends early and react to them before it is too late.

When will we make an end?  Read your email and find out! 

 

Tags: metrics

posted by Dave on Wednesday, August 06 2008 permalink | comments (0)

Boulder New Tech Meetup

At least 300 people attended the Boulder New Tech Meetup. Five Techstar companies presented their companies including eventvue, gyminee, ignighter.com, people's software, flux capacity, and foodzie. See techstars.org/community. The crowd was very high energy. Great place to network. They have time at the begininng for announcements by people looking for employees and those lookng for employment.

 see http://newtech.meetup.com/27/

I think there are a number of meetups in Denver that might be worth checking out. 

Tags: employment, job hunting

posted by Patti on Wednesday, August 06 2008 permalink | comments (0)

TestBench for Oracle Released

TestBench for Oracle has been released by Original Software. The new release includes an intelligent data scrambling module, protecting companies from data exposure and satisfying audit and compliance requirements, as well as safeguarding the individuals or companies referenced in the data.

Common sense and good test practice say that exposing real production data during testing is risky. Despite legal penalties for protecting privacy, testing on live data still tends to be a common practice. One survey showed 62 percent of companies were using live customer data to test applications and 49 percent shared this data with outsourced testers, with no way of knowing if it was ever compromised*.

This is no accident. Production data as a source of background data is valuable because it allows the test environment to represent the live system as closely as possible. A recent report from analyst group Freeform Dynamics - ‘Data Governance in the Software Lifecycle’ – highlights the need for improvements in automation in areas such as test data management, and live data sanitation in the testing process.

Using the new data scrambling module within Original's TestBench for Oracle, real data can be used that accurately reflects the live environment, but has been made anonymous, neutralizing the risk of testing on live data by no longer exposing actual customer details.

TestBench transforms data in two ways:

By using existing data – Real records are used but the data fields are randomly scrambled between rows. Developers may choose to keep some related fields together – for example lines 1, 2, and 3 of the address, for address validation purposes – and just mix up the other details.

By generating new data – recommended for when data is so sensitive (for example, credit card numbers and bank details) that just mixing the field with a different customer name is not sufficient. Developers have three options – to randomly generate, to generate numbers with a sequential start value, or they have the option to write their own program to generate data and plug it straight in. To maintain database integrity, the module will intelligently change these details everywhere they appear – for example in order files, within the customer records database and in all invoices.

“The scrambling module completes our Oracle data management capabilities," Colin Armitage, CEO of Original Software. "Customers can already extract data from live systems to a test environment and the testing functionality enables them to view the effects of the application under test on the database. They can then build rules to validate database effects for all future tests. Our data protection module allows users to roll back the changes and reset the environment if it becomes corrupted by any testing activity, and now we can also ensure that the data being used, is not only risk-free but is still representative of the live environment.”

All of the TestBench for Oracle functionality including data scrambling is available on 9i, 10g and 11i versions.

* The Insecurity of Test Data: The Unseen Crisis, - a Compuware / Ponemon Institute study

Tags: test bench, original, data, scramble, colin armitage

posted by Administrator on Tuesday, August 05 2008 permalink | comments (0)

We Don't Need No Stinkin' Test Plan!

By Dave Whalen

My apologies to Mel Brooks.

When I was in the military I was told the only difference between a "war story" and a "fairy tale" was that a fairy tale always begins with "once upon a time..." but a war story always begins with "No sh*&, this really happened!"

No sh*&, this really happened!

My wife's boss bought a small 2-bedroom house in the foothills just west of Denver.  He wanted to add an adddition to the existing house and recruited volunteer labor from his employees, their spouses, and kids.  A "weekend project" I was told.  So one gorgeous,cool, crisp, Saturday morning in July, we drove up to his house in the foothills to help build the "addition."  Since I had some experience with construction in a prior life, I was soon made project forman. 

It was a bit more than an addition.  He had added a 4-car Garage to the existing house and wanted to build a new house on top of the new garages. So I asked the "boss" where his plans or blueprints were.  He pulled out a spiral notebook and a window catalog and began to draw his "vision" of the addition. "I really like these windows and I think they should go here, here, and here."

Soon a parade of trucks arrived and droped off loads of construction materials and lumber.  How he knew what to order and how much is beyond me.  We began building.  We built and raised walls with rough openings for the new windows (which had not been ordered yet).  As we broke for lunch another truck arrived carrying pre-made roof trusses.  A few minutes later, a crane arrived.  We finished raising the walls and had the crane operator hoist the trusses to the top of the structure.  As we were nailing the first truss in place, the boss looked out at the magnificnet view of Denver below us.  He immediately declared - "We need another floor!"  Construction came to a halt, and we packed up with plans to return the next weekend to build the 3rd floor.

The next weekend arrived, and the scenario replayed itself.  We built the third floor using basically the same process as before, with only a drawing on a piece of notebook paper to guide us.  We put on the roof this time.

Needless to say the project was a nightmare.  Most of us had no idea what we were doing, We ran short of supplies, and had to stop to wait for them to be delivered or go get them ourselves.  We had too much of some items, which ended up being wasted.  Some things were built incorrectly only to be torn down and rebuilt.  When the code inspector arrived, we failed miserably (did I mention we didn't have a building permit).  The biggest problem - the newly added floor pushed the roof to beyond the building code height limit.  We had to tear off the roof, order new trusses, and completely rebuild it. It had to have cost a fortune, even with volunteer labor!

I told you that to tell you this - Always Have a Plan!!!  Or, As Dr' Covey mentions in his "7 Habits" - Begin With The End In Mind!  The same thing applies to software testing.  Take the time to write a Test Plan,  Identify the resouces you need, the time you have, what tools you need, etc.  It can be an exhausting mental process, but it's a valuable one.  It ultimately saves you time and money!

Saddly, most Project Managers will take a wrong approach to test planning - they throw a bunch of people at it, typically in the last couple of weeks before release (which they then cut to a week because "development is a little behind schedule"), then figue it will all work out.  Never has, never will!

Take the time to think through the test process and write a comprehensive Test Plan.  I'm alway asked:  "Should I have it approved and signed?"  I personally like to,  I view the Test Plan as my contract to the project team as to what we will test, how we will test it, how long it will take, how many people I will need, tools, etc.  Will they read it?  Probably not!  But, when things don't go according to plan, its nice to pull it out and point out the signatures.  I usually keep a copy handy.  It should be a working document, continually updated as the project changes.

Take the time to write a good test plan!  A word on templates - I like templates, but templates are just guidelines.  Don't just cut-and-paste your project name into an existing template.  Use it as a guide to write your own project-specific plan.  Personally, I'm a great collector of Test Plans.  I save them and refer to them often, but always write one from scratch for each project.  I may steal a little from this one, or a paragraph form that one, but at the end of the day, I have a unique, project-specific plan.  A roadmap to the future!  My vision of the end.

Tags: test plan, test management

posted by Dave on Tuesday, August 05 2008 permalink | comments (1)

Where to find good job postings?

Hi,

I have been posting job opportunities on Test Common. I am currently just posting jobs for the Denver area. I mostly take jobs listed on SQUAD, Craig's List, Dice, and I find a few interesting ones on www.linkedin.com. I just happened to see one listed for Rivet Software when I was researching the company. Where are other people finding good job listings?

Tags: employment

posted by Patti on Tuesday, August 05 2008 permalink | comments (0)

Net Neutrality Testing Tool Released

Switzerland, a tool for testing the neutrality of Internet service providers (ISPs) has been released by the Electronic Frontier Foundation (EFF). The EFF is campaigning network providers to stop interfering with the traffic that travels over them by throttling, blocking or modifying packets.

Switzerland is designed to check the integrity of communications over networks, ISPs and firewalls by spotting IP packets which are forged or modified by some element in the connection and alert the user with copies of the modified packets. The EFF says that Switzerland should be able to detect advertising injection systems like Phorm, anti-P2P tools from Sandvine and AudibleMagic and censorship systems like the Great Firewall of China, but is not limited to these detections as it spots any packet modifications.

Using a semi-P2P architecture, Switzerland clients send packets to each other and to Switzerland servers and analyzes the packets as received between the different locations. The applications design automates the analysis operations which previously had to be done by hand when using a previous EFF tool pcapdiff.

"The sad truth is that the FCC is ill-equipped to detect ISPs interfering with your Internet connection," said Fred Lochmann, EFF Senior Intellectual Property Attorney. "It's up to concerned Internet users to investigate possible network neutrality violations, and EFF's Switzerland software is designed to help with that effort."

Switzerland is a GPL licensed command line application available as source code hosted on Sourceforge and is part of the EFF's "Test Your ISP" project.

Tags: switzerland, eff, p2p, isp, neutrality, pcapdiff

posted by Administrator on Tuesday, August 05 2008 permalink | comments (0)

Risk Based Testing

Goldilocks and the Two Bears!

Lets face the truth, we are testers, we test, that’s what we do. We love it. A good day for us is when everything development delivers to us is sent back with a host of Sev 1’s. Oh yes, we pretend we are concerned about ensuring quality, protecting the organisation and adding value, but really we just love to find fault. The trouble is, those mean tight fisted bean counters won’t give us unlimited budget, the pin striped suites in Corporate keep reminding us we have contractual dates to meet and those frilly young things in marketing have gone and told the world that ‘WonderWidget’ will be demonstrated at ‘such and such geek show’. 

You know the story of Goldilocks and the Three bears, well guess what? In testing we never get to the third bowl, its never just right. Like it or not, we are going to have to accept that we only have limited budget and limited time to test this application.

Given the limitations above, a risk-based approach to testing can help ensure that we get the biggest bang for our buck. Risk-based testing is not new or difficult, its something we do all the time when writing tests, but often don’t realise. It’s a case of asking, “Where is the software most likely to fail, and what will be the consequence if it does?”

Time is a requirement too.

One of the big challenges for testers who adopt a risk-based approach is to accept that ‘Time To Market’ is a requirement too. In fact it may be the most important requirement for a particular project. If we have to meet a deadline, its no good saying we still have 100 tests to run, we have to meet a deadline. We have to be able to say, “We have 100 tests left to run, however the risk associated with those 100 test is XXXXX, therefore if you release now the possible consequences are, YYYYYY”. We have to have to be able to say, at any given stage, “These are the areas we identified as High, Medium, Low risk, these are the tests we have run, these are the results, and based on these it suggests that, blah blah blah.” 

Risk Factor

The key to running a successful risk-based test cycle is to correctly identify, %%@@# and address the risks associated with the software.  The process follows five stages;

1 Risk Identification

Understand the risks by meeting with the relevant ‘experts’ from both the technical and the business communities and prepare a register of risks. Workshops, document reviews, past projects data can all be used to feed into this activity.

2 Risk Analysis

Not all risks are equal, to ensure that testing addresses the high exposure risks first, relevant experts and stakeholders need to meet to discuss and analyse the risks. Each risk is assigned a probability and an impact, this activity involves all parties, as sometimes, for example, IT staff may not realise that a LOW technical risk, might represent a HIGH business risk. For each risk identified ask the question, “How likely is this to happen?”, and assign the risk the appropriate score based in the agreed answer, i.e.

·          Almost certain to happen, highly likely = 5

·          This is probably going to happen, we think its likely = 4

·          Not sure this will happen, we think this is 50/50 = 3

·          This probably wont happen,  unlikely = 2

·          The chances of this happening are a million to one, this is very unlikely = 1

Again, for each risk ask the question, “If this happens, what will be the impact on the business?” and assign the risk the appropriate score, i.e.

·          It would be critical, the business objective could not be achieved, we would suffer immense loss = 5

·          It would be severe, the business objective would be undermined, we would suffer significant loss = 4

·          It would be moderate, the business objective would be affected, we would suffer some loss = 3

·          It would be low, the business objective could still be achieved, we would suffer minor loss = 2

·          It would be negligible, there would be no real impact on the business objective, we would suffer no loss = 1

Having given each risk a Probability and an Impact score out of five, the exposure that each risk represents can be clearly seen and agreed.

 Probability + Impact = Exposure, i.e.

 Risk                                                            Probability       Impact      Exposure

---

System won’t cope with number of users                    3            +        4       =        7
Converting Name field may cause data loss              1            +        5        =       6
Transmitting confidential data between 9 systems    5            +        5        =     10

---

3 Risk Response

Agree the appropriate response to each risk. It may be that not all risks require a test written, there might be another mitigating action that is more appropriate.

Formulate test criteria for each risk with clear objectives and pass/fail conditions. To help with this ask the following questions;

·          What would cause this risk to materialise, what data, action, circumstance or event would need to be in play?

·          In which area(s) of the system is this most likely to occur?

·          Who would be the most likely to experience the consequence of this risk, and when and why?

·          What types of testing, techniques, or tools would be best to expose this risk?

Document the requirements for each test, such as data, environments, time scales, business resource.

4 Test Scope Definition

Agree the scope for the testing, which risks are in and which are out of scope, what the test schedule will be, who is responsible for what, and what the minimum success criteria is for each set of testing.

It is very helpful to have agreed both, generic quality or test objectives for the system; (e.g. Demonstrate that contractual requirements have been met.) as well as, specific risk based quality or test objectives, (e.g. Demonstrate secure data transfer between system A and system B.)

Produce a Test Coverage matrix, based not on functional requirements, but on identified risks.

Agree what reports will be produced, and what information should be included. Agree who these reports should go to and how often. The main purpose of risk based test reporting is to be able to show which of the identified risks have been tested and which have not yet been addressed. It should provide the reader with the answer to the following question;

“If I release now, what is the danger of any of the identified risks becoming an actual issue?”

5 Test Phase

Testing will now follow a traditional test cycle, (preparation, execution, reporting) however the focus is now not on testing each individual function, but on running the tests that exercise the areas of risk identified, of course concentrating first on the high exposure risks.

Give it a go

I believe that risk based testing can offer significant return on investment for many organisations, directing the test effort to where the pain is most likely to be felt. I believe it can empower the business decision makers, giving them the information they really need. It tells them not,  how many tests have been run, and how many bugs have been found,  but what are the likely consequences to the  business if the code is released in its present state?

I would encourage anyone that finds themselves either with too little budget or two little time to do all the testing they want to do to consider piloting a risk-based test strategy to discover if it can benefit their organisation.

 

Tony Simms is the Principal Consultant at Roque Consulting (www.roque.co.uk) and is available to run training on risk based testing or to facilitate risk identification and response workshops. He can be contacted via email at tony.simms@roque.co.uk 

 

 
 
 

 

Tags: testing, risk based testing, risk, roque, roque consultiung, testing based on risk

posted by Tony on Friday, August 01 2008 permalink | comments (1)

HP, Intel and Yahoo! Create Global Cloud Computing Research Test Bed

HP, Intel Corporation and Yahoo! Inc. (Nasdaq: YHOO) have announced the creation of a global, multi-data center, open source test bed for the advancement of cloud computing research and education. The goal of the initiative is to promote open collaboration among industry, academia and governments by removing the financial and logistical barriers to research in data-intensive, Internet-scale computing.

The HP, Intel and Yahoo! Cloud Computing Test Bed will provide a globally distributed, Internet-scale testing environment designed to encourage research on the software, data center management and hardware issues associated with cloud computing at a larger scale than ever before. The initiative will also support research of cloud applications and services.

HP, Intel and Yahoo! have partnered with the Infocomm Development Authority of Singapore (IDA), the University of Illinois at Urbana-Champaign, and the Karlsruhe Institute of Technology (KIT) in Germany to form the research initiative. The partnership with Illinois also includes the National Science Foundation.

The test bed will initially consist of six “centers of excellence” at IDA facilities, the University of Illinois at Urbana-Champaign, the Steinbuch Centre for Computing of the Karlsruhe Institute of Technology, HP Labs, Intel Research and Yahoo!. Each location will host a cloud computing infrastructure, largely based on HP hardware and Intel processors, and will have 1,000 to 4,000 processor cores capable of supporting the data-intensive research associated with cloud computing. The test bed locations are expected to be fully operational and made accessible to researchers worldwide through a selection process later this year.

The test bed will leverage Yahoo!’s technical leadership in open source projects by running Apache Hadoop – an open source, distributed computing project of the Apache Software Foundation – and other open source, distributed computing software such as Pig, the parallel programming language developed by Yahoo! Research.

“The HP, Intel and Yahoo! Cloud Computing Test Bed furthers our commitment to the global, collaborative research community that is advancing the new sciences of the Internet,” said Prabhakar Raghavan, head of Yahoo! Research. “With this test bed, not only can researchers test applications at Internet scale, they will also have access to the underlying computing systems to advance understanding of how systems software and hardware function in a cloud environment.”

Researchers at HP Labs, the central research arm of HP, will use the test bed to conduct advanced research in the areas of intelligent infrastructure and dynamic cloud services. HP Labs recently sharpened its focus to help HP and its customers capitalize on the industry’s shift toward cloud computing, a driving force behind HP’s vision of Everything as a Service. With Everything as a Service, devices and services will interact seamlessly through the cloud, and businesses and individuals will use services that anticipate their needs based on location, preferences, calendar and communities.

“To realize the full potential of cloud computing, the technology industry must think about the cloud as a platform for creating new services and experiences. This requires an entirely new approach to the way we design, deploy and manage cloud infrastructure and services,” said Prith Banerjee, senior vice president of Research at HP and director of HP Labs. “The HP, Intel and Yahoo! Cloud Computing Test Bed lets us tap the brightest minds in the industry, academia and government to drive innovation in this area.”

Intel is a leading provider of platform technologies, including processors, chipsets, networking and SSD (solid state drives), for cloud computing data centers. Current platform features such as Data Center Management Interface (DCMI), Node Manager (NM) and virtualization have been designed to improve the manageability and energy efficiency of data centers. This open, collaborative research effort will give researchers full access to the system’s hardware for further innovation of existing and future platform features.

“We are pleased to engage with the academic research community – open collaboration with the academia is in our DNA at Intel Research,” said Andrew A. Chien, vice president and director of Intel Research. “Creating large-scale test beds is important because they lower barriers to innovation and provide the opportunity to experiment and learn at scale. Intel’s support of Tashi, an open source cluster management system for cloud computing, and this HP, Intel, Yahoo! Cloud Computing Test Bed are a natural extension of our ongoing, mutually beneficial partnerships with the research community, such as the Universal Parallel Computing Research Centers.”

IDA will facilitate research in the test bed by providing its users with the computing resources required to develop cloud computing software and applications. IDA will also leverage the test bed and its industry partnerships to train local students and professionals on the technologies and programs associated with cloud computing.

“With the ready and available Internet-scale resources in Singapore to support cloud computer research and development work, we can collaborate with like-minded partners to advance the field,” said Khoong Hock Yun, assistant chief executive of the Infrastructure Development Group at the Infocomm Development Authority of Singapore. “Cloud computing is considered by many to be the next paradigm shift in computer technology, and this may be the next ‘platform’ for innovative ecosystems. Partnerships like this will allow Singapore to leverage this new paradigm for greater economic and social growth.”

Partnership builds on initiatives in cloud computing

The Cloud Computing Test Bed is the next step in expanding each company’s ongoing initiatives in cloud computing. In November 2007, Yahoo! announced the deployment of a supercomputing-class data center, called M45, for cloud computing research; Carnegie Mellon University was the first institution to take advantage of this supercomputer. Yahoo! also announced this year an agreement with Computational Research Laboratories (CRL) to jointly support cloud computing research and make one of the 10 fastest supercomputers in the world available to academic institutions in India. Earlier this year, Yahoo! hosted the first-ever Hadoop Summit and Data-Intensive Computing Symposium. Co-sponsored with the Computing Community Consortium (CCC), these meetings brought together leading experts from industry, academia and government to discuss the future directions of Hadoop and data-intensive computing.

In 2008, HP announced the formation of its Scalable Computing & Infrastructure Organization (SCI), which includes a dedicated set of resources that provide expertise and spearhead development efforts to build scalable solutions designed for high-performance and cloud computing customers. The company introduced scalable computing offerings including the Intel Xeon-based HP ProLiant BL2x220c G5, the world’s first server blade to combine two independent servers in a single blade, and the HP StorageWorks 9100 Extreme Data Storage System (ExDS9100), a highly scalable storage system designed to simplify the management of multiple petabytes.⁽¹⁾ HP also introduced the HP POD (Performance-Optimized Datacenter), an open architecture, compact, shipped-to-order alternative for deploying IT resources.

Tags: hp, cloud computing, intel, yahoo, testing center

posted by Administrator on Thursday, July 31 2008 permalink | comments (1)

Give us feedback - What would you like to see more or less of on the site?

What is important to you as a tester or QA professional?

 Jobs?

Content?

Community?

Training?

Career advancement?

 What would make you more excited about Test Common?

Tags: jobs, testing, community

posted by Michelle on Thursday, July 31 2008 permalink | comments (1)

What do YOU think makes a good tester?

Ability to break things?

Ability to understand the perspective of the customer?

The number of bugs found?

Their smile?

 Focus on quality?

 

 

Tags: agile, jobs, testing, qa

posted by Michelle on Tuesday, July 29 2008 permalink | comments (5)