QA's topics - tribe.net

How can new Software QA processes be introduced in an existing organization?

2007-10-23T08:02:13Z

How can new Software QA processes be introduced in an existing organization?

* A lot depends on the size of the organization and the risks involved. For large organizations with high-risk (in terms of lives or property) projects, serious management buy-in is required and a formalized QA process is necessary.
* Where the risk is lower, management and organizational buy-in and QA implementation may be a slower, step-at-a-time process. QA processes should be balanced with productivity so as to keep bureaucracy from getting out of hand.
* For small groups or projects, a more ad-hoc process may be appropriate, depending on the type of customers and projects. A lot will depend on team leads or managers, feedback to developers, and ensuring adequate communications among customers, managers, developers, and testers.
* The most value for effort will often be in (a) requirements management processes, with a goal of clear, complete, testable requirement specifications embodied in requirements or design documentation, or in 'agile'-type environments extensive continuous coordination with end-users, (b) design inspections and code inspections, and (c) post-mortems/retrospectives.
* Other possibilities include incremental self-managed team approaches such as 'Kaizen' methods of continuous process improvement, the Deming-Shewhart Plan-Do-Check-Act cycle, and others.

posted in QA - 1 reply

Why does software have bugs?

2007-10-23T07:59:43Z

* miscommunication or no communication - as to specifics of what an application should or shouldn't do (the application's requirements).
* software complexity - the complexity of current software applications can be difficult to comprehend for anyone without experience in modern-day software development. Multi-tiered applications, client-server and distributed applications, data communications, enormous relational databases, and sheer size of applications have all contributed to the exponential growth in software/system complexity.
* programming errors - programmers, like anyone else, can make mistakes.
* changing requirements (whether documented or undocumented) - the end-user may not understand the effects of changes, or may understand and request them anyway - redesign, rescheduling of engineers, effects on other projects, work already completed that may have to be redone or thrown out, hardware requirements that may be affected, etc. If there are many minor changes or any major changes, known and unknown dependencies among parts of the project are likely to interact and cause problems, and the complexity of coordinating changes may result in errors. Enthusiasm of engineering staff may be affected. In some fast-changing business environments, continuously modified requirements may be a fact of life. In this case, management must understand the resulting risks, and QA and test engineers must adapt and plan for continuous extensive testing to keep the inevitable bugs from running out of control - see 'What can be done if requirements are changing continuously?' in the LFAQ. Also see information about 'agile' approaches such as XP, in Part 2 of the FAQ.
* time pressures - scheduling of software projects is difficult at best, often requiring a lot of guesswork. When deadlines loom and the crunch comes, mistakes will be made.
* egos - people prefer to say things like:

'no problem'
'piece of cake'
'I can whip that out in a few hours'
'it should be easy to update that old code'

instead of:
'that adds a lot of complexity and we could end up
making a lot of mistakes'
'we have no idea if we can do that; we'll wing it'
'I can't estimate how long it will take, until I
take a close look at it'
'we can't figure out what that old spaghetti code
did in the first place'

If there are too many unrealistic 'no problem's', the
result is bugs.

* poorly documented code - it's tough to maintain and modify code that is badly written or poorly documented; the result is bugs. In many organizations management provides no incentive for programmers to document their code or write clear, understandable, maintainable code. In fact, it's usually the opposite: they get points mostly for quickly turning out code, and there's job security if nobody else can understand it ('if it was hard to write, it should be hard to read').
* software development tools - visual tools, class libraries, compilers, scripting tools, etc. often introduce their own bugs or are poorly documented, resulting in added bugs.

Return to top of this page's FAQ list

How can new Software QA processes be introduced in an existing organization?

* A lot depends on the size of the organization and the risks involved. For large organizations with high-risk (in terms of lives or property) projects, serious management buy-in is required and a formalized QA process is necessary.
* Where the risk is lower, management and organizational buy-in and QA implementation may be a slower, step-at-a-time process. QA processes should be balanced with productivity so as to keep bureaucracy from getting out of hand.
* For small groups or projects, a more ad-hoc process may be appropriate, depending on the type of customers and projects. A lot will depend on team leads or managers, feedback to developers, and ensuring adequate communications among customers, managers, developers, and testers.
* The most value for effort will often be in (a) requirements management processes, with a goal of clear, complete, testable requirement specifications embodied in requirements or design documentation, or in 'agile'-type environments extensive continuous coordination with end-users, (b) design inspections and code inspections, and (c) post-mortems/retrospectives.
* Other possibilities include incremental self-managed team approaches such as 'Kaizen' methods of continuous process improvement, the Deming-Shewhart Plan-Do-Check-Act cycle, and others.

posted in QA - 1 reply

What are some recent major computer system failures caused by software bugs?

2007-10-23T07:52:21Z

* In June of 2007 news reports claimed that software flaws in a popular online stock-picking contest could be used to gain an unfair advantage in pursuit of the game's large cash prizes. Outside investigators were called in and in July the contest winner was announced. Reportedly the winner had previously been in 6th place, indicating that the top 5 contestants may have been disqualified.
* A software problem contributed to a rail car fire in a major underground metro system in April of 2007 according to newspaper accounts. The software reportedly failed to perform as expected in detecting and preventing excess power usage in equipment on a new passenger rail car, resulting in overheating and fire in the rail car, and evacuation and shutdown of part of the system.
* Tens of thousands of medical devices were recalled in March of 2007 to correct a software bug. According to news reports, the software would not reliably indicate when available power to the device was too low.
* A September 2006 news report indicated problems with software utilized in a state government's primary election, resulting in periodic unexpected rebooting of voter checkin machines, which were separate from the electronic voting machines, and resulted in confusion and delays at voting sites. The problem was reportedly due to insufficient testing.
* In August of 2006 a U.S. government student loan service erroneously made public the personal data of as many as 21,000 borrowers on it's web site, due to a software error. The bug was fixed and the government department subsequently offered to arrange for free credit monitoring services for those affected.
* A software error reportedly resulted in overbilling of up to several thousand dollars to each of 11,000 customers of a major telecommunications company in June of 2006. It was reported that the software bug was fixed within days, but that correcting the billing errors would take much longer.
* News reports in May of 2006 described a multi-million dollar lawsuit settlement paid by a healthcare software vendor to one of its customers. It was reported that the customer claimed there were problems with the software they had contracted for, including poor integration of software modules, and problems that resulted in missing or incorrect data used by medical personnel.
* In early 2006 problems in a government's financial monitoring software resulted in incorrect election candidate financial reports being made available to the public. The government's election finance reporting web site had to be shut down until the software was repaired.
* Trading on a major Asian stock exchange was brought to a halt in November of 2005, reportedly due to an error in a system software upgrade. The problem was rectified and trading resumed later the same day.
* A May 2005 newspaper article reported that a major hybrid car manufacturer had to install a software fix on 20,000 vehicles due to problems with invalid engine warning lights and occasional stalling. In the article, an automotive software specialist indicated that the automobile industry spends $2 billion to $3 billion per year fixing software problems.
* Media reports in January of 2005 detailed severe problems with a $170 million high-profile U.S. government IT systems project. Software testing was one of the five major problem areas according to a report of the commission reviewing the project. In March of 2005 it was decided to scrap the entire project.
* In July 2004 newspapers reported that a new government welfare management system in Canada costing several hundred million dollars was unable to handle a simple benefits rate increase after being put into live operation. Reportedly the original contract allowed for only 6 weeks of acceptance testing and the system was never tested for its ability to handle a rate increase.
* Millions of bank accounts were impacted by errors due to installation of inadequately tested software code in the transaction processing system of a major North American bank, according to mid-2004 news reports. Articles about the incident stated that it took two weeks to fix all the resulting errors, that additional problems resulted when the incident drew a large number of e-mail phishing attacks against the bank's customers, and that the total cost of the incident could exceed $100 million.
* A bug in site management software utilized by companies with a significant percentage of worldwide web traffic was reported in May of 2004. The bug resulted in performance problems for many of the sites simultaneously and required disabling of the software until the bug was fixed.
* According to news reports in April of 2004, a software bug was determined to be a major contributor to the 2003 Northeast blackout, the worst power system failure in North American history. The failure involved loss of electrical power to 50 million customers, forced shutdown of 100 power plants, and economic losses estimated at $6 billion. The bug was reportedly in one utility company's vendor-supplied power monitoring and management system, which was unable to correctly handle and report on an unusual confluence of initially localized events. The error was found and corrected after examining millions of lines of code.
* In early 2004, news reports revealed the intentional use of a software bug as a counter-espionage tool. According to the report, in the early 1980's one nation surreptitiously allowed a hostile nation's espionage service to steal a version of sophisticated industrial software that had intentionally-added flaws. This eventually resulted in major industrial disruption in the country that used the stolen flawed software.
* A major U.S. retailer was reportedly hit with a large government fine in October of 2003 due to web site errors that enabled customers to view one anothers' online orders.
* News stories in the fall of 2003 stated that a manufacturing company recalled all their transportation products in order to fix a software problem causing instability in certain circumstances. The company found and reported the bug itself and initiated the recall procedure in which a software upgrade fixed the problems.
* In August of 2003 a U.S. court ruled that a lawsuit against a large online brokerage company could proceed; the lawsuit reportedly involved claims that the company was not fixing system problems that sometimes resulted in failed stock trades, based on the experiences of 4 plaintiffs during an 8-month period. A previous lower court's ruling that "...six miscues out of more than 400 trades does not indicate negligence." was invalidated.
* In April of 2003 it was announced that a large student loan company in the U.S. made a software error in calculating the monthly payments on 800,000 loans. Although borrowers were to be notified of an increase in their required payments, the company will still reportedly lose $8 million in interest. The error was uncovered when borrowers began reporting inconsistencies in their bills.
* News reports in February of 2003 revealed that the U.S. Treasury Department mailed 50,000 Social Security checks without any beneficiary names. A spokesperson indicated that the missing names were due to an error in a software change. Replacement checks were subsequently mailed out with the problem corrected, and recipients were then able to cash their Social Security checks.
* In March of 2002 it was reported that software bugs in Britain's national tax system resulted in more than 100,000 erroneous tax overcharges. The problem was partly attributed to the difficulty of testing the integration of multiple systems.
* A newspaper columnist reported in July 2001 that a serious flaw was found in off-the-shelf software that had long been used in systems for tracking certain U.S. nuclear materials. The same software had been recently donated to another country to be used in tracking their own nuclear materials, and it was not until scientists in that country discovered the problem, and shared the information, that U.S. officials became aware of the problems.
* According to newspaper stories in mid-2001, a major systems development contractor was fired and sued over problems with a large retirement plan management system. According to the reports, the client claimed that system deliveries were late, the software had excessive defects, and it caused other systems to crash.
* In January of 2001 newspapers reported that a major European railroad was hit by the aftereffects of the Y2K bug. The company found that many of their newer trains would not run due to their inability to recognize the date '31/12/2000'; the trains were started by altering the control system's date settings.
* News reports in September of 2000 told of a software vendor settling a lawsuit with a large mortgage lender; the vendor had reportedly delivered an online mortgage processing system that did not meet specifications, was delivered late, and didn't work.
* In early 2000, major problems were reported with a new computer system in a large suburban U.S. public school district with 100,000+ students; problems included 10,000 erroneous report cards and students left stranded by failed class registration systems; the district's CIO was fired. The school district decided to reinstate it's original 25-year old system for at least a year until the bugs were worked out of the new system by the software vendors.
* A review board concluded that the NASA Mars Polar Lander failed in December 1999 due to software problems that caused improper functioning of retro rockets utilized by the Lander as it entered the Martian atmosphere.
* In October of 1999 the $125 million NASA Mars Climate Orbiter spacecraft was believed to be lost in space due to a simple data conversion error. It was determined that spacecraft software used certain data in English units that should have been in metric units. Among other tasks, the orbiter was to serve as a communications relay for the Mars Polar Lander mission, which failed for unknown reasons in December 1999. Several investigating panels were convened to determine the process failures that allowed the error to go undetected.
* Bugs in software supporting a large commercial high-speed data network affected 70,000 business customers over a period of 8 days in August of 1999. Among those affected was the electronic trading system of the largest U.S. futures exchange, which was shut down for most of a week as a result of the outages.
* In April of 1999 a software bug caused the failure of a $1.2 billion U.S. military satellite launch, the costliest unmanned accident in the history of Cape Canaveral launches. The failure was the latest in a string of launch failures, triggering a complete military and industry review of U.S. space launch programs, including software integration and testing processes. Congressional oversight hearings were requested.
* A small town in Illinois in the U.S. received an unusually large monthly electric bill of $7 million in March of 1999. This was about 700 times larger than its normal bill. It turned out to be due to bugs in new software that had been purchased by the local power company to deal with Y2K software issues.
* In early 1999 a major computer game company recalled all copies of a popular new product due to software problems. The company made a public apology for releasing a product before it was ready.
* The computer system of a major online U.S. stock trading service failed during trading hours several times over a period of days in February of 1999 according to nationwide news reports. The problem was reportedly due to bugs in a software upgrade intended to speed online trade confirmations.
* In April of 1998 a major U.S. data communications network failed for 24 hours, crippling a large part of some U.S. credit card transaction authorization systems as well as other large U.S. bank, retail, and government data systems. The cause was eventually traced to a software bug.
* January 1998 news reports told of software problems at a major U.S. telecommunications company that resulted in no charges for long distance calls for a month for 400,000 customers. The problem went undetected until customers called up with questions about their bills.
* In November of 1997 the stock of a major health industry company dropped 60% due to reports of failures in computer billing systems, problems with a large database conversion, and inadequate software testing. It was reported that more than $100,000,000 in receivables had to be written off and that multi-million dollar fines were levied on the company by government agencies.
* A retail store chain filed suit in August of 1997 against a transaction processing system vendor (not a credit card company) due to the software's inability to handle credit cards with year 2000 expiration dates.
* In August of 1997 one of the leading consumer credit reporting companies reportedly shut down their new public web site after less than two days of operation due to software problems. The new site allowed web site visitors instant access, for a small fee, to their personal credit reports. However, a number of initial users ended up viewing each others' reports instead of their own, resulting in irate customers and nationwide publicity. The problem was attributed to "...unexpectedly high demand from consumers and faulty software that routed the files to the wrong computers."
* In November of 1996, newspapers reported that software bugs caused the 411 telephone information system of one of the U.S. RBOC's to fail for most of a day. Most of the 2000 operators had to search through phone books instead of using their 13,000,000-listing database. The bugs were introduced by new software modifications and the problem software had been installed on both the production and backup systems. A spokesman for the software vendor reportedly stated that 'It had nothing to do with the integrity of the software. It was human error.'
* On June 4 1996 the first flight of the European Space Agency's new Ariane 5 rocket failed shortly after launching, resulting in an estimated uninsured loss of a half billion dollars. It was reportedly due to the lack of exception handling of a floating-point error in a conversion from a 64-bit integer to a 16-bit signed integer.
* Software bugs caused the bank accounts of 823 customers of a major U.S. bank to be credited with $924,844,208.32 each in May of 1996, according to newspaper reports. The American Bankers Association claimed it was the largest such error in banking history. A bank spokesman said the programming errors were corrected and all funds were recovered.
* Software bugs in a Soviet early-warning monitoring system nearly brought on nuclear war in 1983, according to news reports in early 1999. The software was supposed to filter out false missile detections caused by Soviet satellites picking up sunlight reflections off cloud-tops, but failed to do so. Disaster was averted when a Soviet commander, based on what he said was a '...funny feeling in my gut', decided the apparent missile attack was a false alarm. The filtering software code was rewritten.

posted in QA - 1 reply

What is Software Testing?

2007-10-23T07:44:46Z

Testing involves operation of a system or application under controlled conditions and evaluating the results (eg, 'if the user is in interface A of the application while using hardware B, and does C, then D should happen'). The controlled conditions should include both normal and abnormal conditions. Testing should intentionally attempt to make things go wrong to determine if things happen when they shouldn't or things don't happen when they should. It is oriented to 'detection'.

Organizations vary considerably in how they assign responsibility for QA and testing. Sometimes they're the combined responsibility of one group or individual. Also common are project teams that include a mix of testers and developers who work closely together, with overall QA processes monitored by project managers. It will depend on what best fits an organization's size and business structure.

posted in QA - 1 reply

Does every software project need testers?

2007-10-23T06:38:18Z

While all projects will benefit from testing, some projects may not require independent test staff to succeed.

Which projects may not need independent test staff? The answer depends on the size and context of the project, the risks, the development methodology, the skill and experience of the developers, and other factors. For instance, if the project is a short-term, small, low risk project, with highly experienced programmers utilizing thorough unit testing or test-first development, then test engineers may not be required for the project to succeed.

In some cases an IT organization may be too small or new to have a testing staff even if the situation calls for it. In these circumstances it may be appropriate to instead use contractors or outsourcing, or adjust the project management and development approach (by switching to more senior developers and agile test-first development, for example). Inexperienced managers sometimes gamble on the success of a project by skipping thorough testing or having programmers do post-development functional testing of their own work, a decidedly high risk gamble.

For non-trivial-size projects or projects with non-trivial risks, a testing staff is usually necessary. As in any business, the use of personnel with specialized skills enhances an organization's ability to be successful in large, complex, or difficult tasks. It allows for both a) deeper and stronger skills and b) the contribution of differing perspectives. For example, programmers typically have the perspective of 'what are the technical issues in making this functionality work?'. A test engineer typically has the perspective of 'what might go wrong with this functionality, and how can we ensure it meets expectations?'. Technical people who can be highly effective in approaching tasks from both of those perspectives are rare, which is why, sooner or later, organizations bring in test specialists.

posted in QA - 0 replies

What is SQA?

2007-10-23T06:35:07Z

Software QA involves the entire software development PROCESS - monitoring and improving the process, making sure that any agreed-upon standards and procedures are followed, and ensuring that problems are found and dealt with. It is oriented to 'prevention'.

posted in QA - 0 replies

What is a Software Life Cycle?

2007-10-23T06:34:11Z

What is the 'software life cycle'?

The life cycle begins when an application is first conceived and ends when it is no longer in use. It includes aspects such as initial concept, requirements analysis, functional design, internal design, documentation planning, test planning, coding, document preparation, integration, testing, maintenance, updates, retesting, phase-out, and other aspects.

posted in QA - 0 replies

How do you define Software Test Strategy?

2005-12-19T19:45:19Z

A few thoughts and suggestions:

A solid testing strategy provides the framework necessary to implement your testing methodology. A separate strategy should be developed for each system being developed taking into account the development methodology being used and the specific application architecture.

The heart of any testing strategy is the master testing strategy document. It aggregates all the information from the requirements, system design and acceptance criteria into a detailed plan for testing. A detailed master strategy should cover the following:

Project Scope

Restate the business objective of the application and define the scope of the testing. The statement should be a list of activities that will be in scope or out of scope. A sample list would include:

* List of software to be tested
* Software configurations to be tested
* Documentation to be validated
* Hardware to be tested

Test Objectives

The system under test should be measured by its compliance to the requirements and the user acceptance criteria. Each requirement and acceptance criteria must be mapped to specific test plans that validate and measure the expected results for each test being performed. The objectives should be listed in order of importance and weighted by Risk.

Features and Functions to be Tested

Every feature and function must be listed for test inclusion or exclusion, along with a description of the exceptions. Some features may not be testable due to a lack of hardware or lack of control etc. The list should be grouped by functional area to add clarity. The following is a basic list of functional areas:

* Backup and recovery
* Workflow
* Interface design
* Installation
* Procedures (users, operational, installation)
* Requirements and design
* Messaging
* Notifications
* Error handling
* System exceptions and third-party application faults

Testing Approach

The approach provides the detail necessary to describe the levels and types of testing. The basic V-Model shows what types of testing are needed to validate the system.

Figure 1

More specific test types include functionality, performance testing, backup and recovery, security testing, environmental testing, conversion testing, usability testing, installation and regression testing. The specific testing methodology should be described and the entry/exit criteria for each phase noted in a matrix by phase. A project plan that list the resources and schedule for each testing cycle should also be created that maps the specific testing task to the overall development project plan.

Testing Process and Procedures

The order of test execution and the steps necessary to perform each type of test should be described in sufficient detail to provide clear input into the creation of test plans and test cases. Procedures should include how test data is created, managed and loaded. Test cycles should be planned and scheduled based on system availability and deliverable dates from development. All application and environmental dependencies should be identified along with the procedures necessary to gain access to all the dependent systems.

Test Compliance

Every level of testing must have a defined set of entry/exit criteria which is used to validate that all prerequisites for a valid test have been met. All mainstream software testing methodologies provide an extensive list of entry/exit criteria and checklist. In addition to the standard list, additional items should be added based on specific testing needs. Some common additions are, environmental availability, data availability, and validated code which is ready to be tested.

Each level of testing should define specific pass/fail acceptance criteria, to ensure to ensure that all quality gates have been validated and that the test plan focuses on developing test that validate the specific criteria defined by the user acceptance plan.

Testing Tools

All testing tools should be identified and their use, ownership and dependencies defined. The tools category includes manual tools, such as templates in spreadsheets and documents as well as automated tools for test management, defect tracking, regression testing and performance/load testing. Any specific skill sets should be identified and compared against the existing skills identified for the project to highlight any training needs.

Defect Resolution

A plan to address the resolution of failed tests needs to be created that lists the escalation procedures to seek correction and retest of the failed tests along with a risk mitigation plan for high-risk test. Defect tracking should include basic metrics for compliance based on number and type of defect found.

Roles and Responsibilities

A matrix listing the roles and responsibilities of everyone involved in the testing activities, along with the anticipated amount of their time allocated to the project, must be prepared.

Process Improvement

The entire testing process should be focused on process improvement. The strategy should list ways to monitor progress and provide constant feedback. This feedback can serve to enhance the process, deliverables and metrics used in the testing. Root cause analysis should be performed on all reported defects to help isolate the true nature of the problem and prevent unnecessary repeat offenses.

Deliverables

All deliverables should be defined and their location specified. Common deliverables are test plans, test cases, test scripts, test matrix and a defect log.

Schedule

All testing activities should be combined into one master testing schedule. The schedule should include an estimate of time for each task and the dependences for each. Testing resources should be assigned to each task and quality gates should be listed to insure oversight of the entire process.

Environmental Needs

All the requirements of the testing environment need to be listed. Common ones include a description of the environment's use, management, hardware and software, specific tools needed, data loading and security requirements.

Resource Management

The skills of all personnel involved in the testing effort need to be assessed and the gaps noted so that a comprehensive training program can be designed. Specialty skills that will not be filled with in-house staff will require job descriptions and budgeting.

Risk and Contingencies

Planning for risk in advance and ways to mitigate it are essential for a robust strategy. A risk assessment that is prioritized by severity of risk and covers technology, resource, schedule and environmental issues should feed a detailed plan to mitigate each red flag.

Approvals and Workflow

All items on the critical path must go through an approval cycle. The procedures for approval and escalation must be well defined and assigned to resources prior to the start of the testing.

The above covers the main sections of a well-drafted and documented testing strategy. The more detail that you include in the strategy document, the less ambiguity and chance for deviation there will be throughout the project

The completion of the strategy signals the beginning of the test planning phase. For each type of testing identified in the master test strategy there should be a test plan identifying the components to be tested, the location of the test data, the test environment needs, the test procedures, resources required, and the tests schedule. For each plan a series of test conditions should be identified so that test cases with expected results can be generated for later execution.

In summary the strategy and planning documents are the most critical documents to any successful testing. A good source of detail on testing documents is IEEE Std 829-1998. It provides the specific form and content for a basic set of software testing documents. A set of basic software test documents is described. This standard specifies the form and content of individual test documents.

Source: http://66.102.7.104/search?q=cache:XoLD57G2jA4J:www.dmreview.com/article_sub.cfm%3FarticleId%3D1027888+How+do+you+define+Software+Test+Strategy%3F&hl=en

posted in QA - 0 replies

How do you define Software Test Policy?

2005-12-19T19:38:59Z

Here are a few suggestions:

1. Ensure that the software engineering tasks are defined, integrated, and consistently performed to produce the software.

2. Keep software work products consistent with each other.

3. Provide adequate resources and funding for performing the software engineering tasks.

4. Require training for the members of the software engineering technical staff in the performance of their technical assignments.

5. Orient members of the software engineering technical staff in related software engineering disciplines.

6. Orient the project manager and all software managers in the technical aspects of their software project.

7. Integrate the appropriate software engineering methods and tools into each project's defined software process.

8. Ensure that the software requirements are developed, maintained, documented, and verified by systematically analyzing the allocated requirements according to each project's defined software process.

9. Ensure that the software design is developed, maintained, documented, and verified, according to each project's defined software process, to accommodate the software requirements and to form the framework for coding.

10. Ensure that the software code is developed, maintained, documented, and verified, according to each project's defined software process, to implement the software requirements and software design.

11. Perform software testing according to each project's defined software process.

12. Plan and perform integration testing of the software according to each project's defined software process.

13. Plan and perform system and acceptance testing of the software to demonstrate that the software satisfies its requirements.

14. Develop and maintain the documentation that will be used to operate and maintain the software according to each project's defined software process.

15. Collect and analyze data on defects identified in peer reviews and testing according to each project's defined software process.

16. Maintain consistency across software work products, including the software plans, process descriptions, allocated requirements, software requirements, software design, code, test plans, and test procedures.

17. Make and use measurements to determine the functionality and quality of the software products.

18. Make and use measurements to determine the status of the software product engineering activities.

19. Review the activities for software product engineering with senior management on a periodic basis.

20. Review the activities for software product engineering with the project manager on both a periodic and event-driven basis.

21. Have the software quality assurance group conduct reviews or audits of the activities and work products for software product engineering and report the results.

posted in QA - 0 replies

Software Testing by Statistical Methods

2005-12-19T19:37:01Z

Software Testing by Statistical Methods
Preliminary Success Estimates for Approaches based on Binomial Models, Coverage Designs, Mutation Testing, and Usage Models [PDF]
David Banks, William Dashiell, Leonard Gallagher, Charles Hagwood, Raghu Kacker and Lynne Rosenthal
Exhaustive conformance testing of software is not practical because variable input values and variable sequencing of inputs result in too many possible combinations to test. This paper addresses alternatives for exhaustive testing based on statistical methods, including experimental designs of tests, statistical selection of representative test cases, quantification of test results, and provision of statistical levels of confidence or probability that a program implements its functional specification correctly. The goal of this work is to ensure software quality and to develop methods for software conformance testing based on known statistical techniques, including multivariable analysis, design of experiments, coverage designs, usage models, and optimization techniques, as well as to provide quantitative measures of quality, reliability, and conformance to specifications, including statistical measures and confidence levels.

See: http://www.itl.nist.gov/div897/ctg/stat/mar98ir.pdf

See:

posted in QA - 2 replies

NEW: UML Tribe

2005-12-19T19:35:48Z

The Unified Modeling Language Tribe is open to discuss UML and modeling in general.

http://tribes.tribe.net/uml

posted in QA - 1 reply

Software Testing Techniques

2005-10-17T03:48:51Z

Resources:
http://www.rspa.com/reflib/TestingTactics.html

See also:

http://www.testingcraft.com/techniques.html

posted in QA - 0 replies

On Fault Coverage of Tests for Finite State Specifications

2005-10-17T03:45:46Z

On Fault Coverage of Tests for Finite State Specifications [PDF]
A. Petrenko and G. v. Bochmann
Testing is a trade-off between increased confidence in the correctness of the implementation under test and constraints on the amount of time and effort that can be spent in testing. This paper analyzes basic ideas underlying the techniques for fault coverage analysis and assurance mainly developed in the context of protocol conformance testing based on finite state models. Special attention is paid to parameters which determine the testability of a given specification and influence the length of a test suite which guarantees complete fault coverage.

See: http://citeseer.nj.nec.com/cache/papers/cs/8780/ftp:zSzzSzbeethoven.site.uottawa.cazSzPublicationszSzPetr96b.pdf/petrenko96fault.pdf

posted in QA - 0 replies

UML - Based Integration Testing for Component

2005-10-17T03:42:51Z

UML - Based Integration Testing for Component - Based Software [PDF]
Ye Wu, Mei-Hwa Chen and Jeff Offutt
The Unified Modeling Language (UML) has been widely adopted in component-based software development processes. Many of its useful tools, such as interaction diagrams, statechart diagrams, and component diagrams, characterize the behavior of components in various aspects, and thus can be used to help test component-based systems. This paper first analyzes different test elements that are critical to test component-based software, then a group of UML-based test adequacy criteria is proposed that can be used to test component-based software.

See: http://www.isse.gmu.edu/faculty/ofut/rsrch/papers/umlcomp03.pdf

posted in QA - 0 replies

New Models for Test Development

2005-10-17T03:41:39Z

New Models for Test Development [PDF]
Brian Marick
In this paper the author discusses why the "V model" doesn't work. The author then describes a model he thinks is better. Important requirements for test models are then presented.

See: http://www.testing.com/writings/new-models.pdf

posted in QA - 0 replies

A Framework for Good Enough Testing

2005-10-17T03:40:40Z

A Framework for Good Enough Testing [PDF]
James Bach
This article discusses a Good Enough quality model. The Good Enough model is first defined, then the parts of the definition are assessed. The author asks how good the testing is and is it worth improving

See: http://www.satisfice.com/articles/good_enough_testing.pdf

posted in QA - 0 replies

Agile Testing

2005-10-17T03:39:59Z

Agile Testing
What is it? Can it work? [PDF]
Bret Pettichord
The topics for this slide presentation are: what agile testing is, agile development methodologies, the role of testing, test-first programming, refactoring: improve the design of existing code, acceptance testing, a practice for agile testing (conversation test creation, coaching tests, providing test interfaces, exploratory learning).

See: http://www.io.com/~wazmo/papers/agile_testing_20021015.pdf

posted in QA - 0 replies

Heuristic Risk-Based Testing

2005-10-17T03:39:11Z

Heuristic Risk-Based Testing [PDF]
James Bach
This article explains what risk-based testing is and then explains why you would want to do it and how to do it well. Heuristic analysis and two approaches to analysis (inside-out and outside-in) are then presented. Three ways to organize risk - based testing and making it all work are then discussed.

See: http://www.satisfice.com/articles/hrbt.pdf

posted in QA - 0 replies

Request for QA / Test Engineer Opportunities

2005-10-12T20:37:20Z

Request for QA Opportunities in Open Source Projects, Employment, Full-Time, Contract, Consulting

I'm seeking interesting Full Time, Contract, Consulting/Freelance Opportunities *and* Open Source Projects that are looking for a QA engineer with a solid background and diverse technical skillset. I've been in Software / Hardware Quality Assurance for 12+ years and am seeking projects to contribute towards the quality of the Gnu Public license and/or any Open Source model. I also am interested in Full-time employment, contract/consulting roles, as well as part-time/freelance work on any projects that seek a stellar QA engineer. If you know of someone who needs help or is looking for a tester with a technical edge, please feel free to refer them to me.

Roles: Full-time employment, contract, freelance, W2 opportunities in the QA field.

Objective: Senior QA Engineer, Lead Quality Assurance Engineer or QA Manager role
Summary & Qualifications: 12 years experience in Hardware/Software QA Engineering
- 6 years experience managing Quality Assurance organizations, leading multiple QA engineers and projects.
- 8 years experience in a Product Support and Technical Support Engineering role.
- 12 years experience in a Senior QA Engineer and Lead QA Engineer role.
- 12 years experience establishing various QA processes and procedures including: Product Development and Software LifeCycle, the QA Lifecycle process, Defect Submittal, Reporting, Tracking and Escalation Lifecycle, Product Validation, Certification and Release Lifecycle, QA Project Planning and QA Test Methodologies and Processes.
- 12 Years planning and executing Test Plans as part of the design, development and QA cycle of various projects.
- 12 years experience authoring and conducting Acceptance, Ad-Hoc, Black Box (Behavioral, Functional, Closed Box, Opaque box), GUI, Functional, Integration, Installation, Manual, Navigation, Module and Regression Tests.
- 11 Years performing release certification and validation, Unit and Integration Checks, Source Code Management.
- 10 years conducting White Box tests (Structural, Glassbox, Clearbox, System Internals, Branch/Path Coverage Tests).
- 12 years authoring and executing Performance, Database, Security, Smokescreen, Stress, Boundary, Load, Validation, SW & HW Compatibility Tests using advanced and innovative Test Methodologies in Macintosh, Windows, Unix, WAP, WML, Console game systems and Palm OS environments.
- 12 years authoring, documenting and defining standards for QA (Test Plans, Test Cases, Test Scripts and Test Procedures).
- 7 years experience administrating QA Lab Operations (Network/Test/Lab Environment), QA Release Cycle and QA Intranet website. Functions of this capacity included: QA Website Webmaster, QA Lab Administrator, QA Production Lab Administrator, QA Release certification and Defect Tracking Systems Administrator.

Resume and extensive references furnished upon request

posted in QA - 0 replies

Top 5 List of QA Resources

2005-07-27T16:20:26Z

Top 5 List

comp.software.testing usenet news group - Via Google Groups web site (formerly the Deja News site), can be used to search through past postings; postings go back to 1995.

Stickyminds.com - Comprehensive software testing resource site associated with 'Better Software' Magazine, with articles, news, information on software testing and quality engineering, books, tools, conferences, message boards.

Cem Kaner's software testing site - Cem Kaner's site contains a large selection of his articles about software testing, legal issues, test management, and more (see the 'Publications' section of the site). Also see his 'badsoftware.com' website , a consumer and legal-issues orientation to software quality issues.

STORM - Software Testing Online Resources/MTSU - a well-organized site with listings of many links to software QA and testing-related web sites.

The comp.software.testing FAQ - The comp.software.testing FAQ; maintained by Raymond Rivest; excellent resource for testing-related conferences, mailing lists, books, periodicals, organizations, and links to other sites. Not maintained recently but still useful.

See Source for URLS:
http://www.softwareqatest.com/qatlnks1.html#TOP_5

posted in QA - 0 replies

Web Security Testing Resources

2005-07-27T16:18:59Z

Computer Audit FAQ - Good introductory information from IsecT Ltd. on 'Computer Audit', which refers to the analysis of computer systems and networks by examining the effectiveness of their technical and procedural controls (information security control systems) to minimise risks. Also has links to other resources, and some articles such as 'Strategic Approach to Information Security Management'

SANS Top 20 List - List and descriptions of top Windows and UNIX internet security vulnerabilities, along with links to other resources.

CVE - Searchable, downloadable, and on-the-web 'Common Vulnerabilities and Exposures' list hosted by Mitre Corp. CVE goal is to standardize the names for all publicly known vulnerabilities and security exposures, so that security information can be efficiently shared and handled. Many security test tools are utilizing or planning on utilizing this standardized naming/numbering system.

IBM DeveloperWorks Security Zone - Software Security-related articles, resources, and tutorials from IBM's developerWorks web site.

W3 Security Resources - Large collection of information and resources on web security, including an FAQ, hosted by the W3C Consortium (the folks who set web standards/protocols, etc.)

Microsoft Security Advisor - Microsoft's web site for discussion of security issues for MS products, including their web server products.

NIAP website - The National Information Assurance Partnership web site - partners are US govt. agencies NIST and NSA. Includes sections for 'Security Testing', 'Tools and Techniques', 'Automated Testing', info re international IT security standard ISO/IEC 15408, the 'Common Criteria for Information Technology Security Evaluation' and the associated 'Common Evaluation Methodology'.

SANS website - Web site of SANS (System Administration, Networking, and Security Institute), a cooperative research and education organization through which more than 96,000 sysadmins, security professionals, and network administrators share lessons learned and solutions.

Security Focus.Com - Site for news, forums, resources, vulnerability info, conference info, tools, etc. related to computer security including web and internet security issues. Search vulnerability database by keywords, date, vendor, version, etc.

COAST Security Archive - Purdue University's computer security site; includes extensive collection of links organized by subject to security tools, info resources, etc. Tools list of more than 100 security tools includes many test tools such as CRACK, COPS, IPSend, Tiger, Secure Sun, etc.; all tools listed are available for download from the COAST site.

Computer Emergency Response Team site - CERT's internet security web site; includes web server security information; hosted by the Software Engineering Institute at Carnegie Mellon University.

See Source for URLS mentioned above:
http://www.softwareqatest.com/qatlnks1.html#WEB_SECURITY

posted in QA - 0 replies

Web Usability Resources for QA

2005-07-27T16:15:37Z

Useit.com - Jakob Nielsen's web usability website with such articles as 'How Users Read on the Web", 'Costs of User Testing', and 'Differences between Print Design and Web Design'.

Usability Testing of Advanced Web Concepts - Article on usability testing at Sun web site.

User Interface Engineering - Site of UIE, a UI training and consulting company; info and links re usability testing and a chapter from the book "Web Site Usability" by Jarod Spool.

Microsoft Usability Home Page - Microsoft's collection of info about their usability labs, related issues, listings of resources; includes info from the 'Human Factors and the Web' conference series.

See: http://www.softwareqatest.com/qatlnks1.html#WEB_USABILITY For URLS of each

posted in QA - 0 replies

General Software QA and Testing Resources

2005-07-27T16:14:20Z

Agile Methodologies - Martin Fowler's online discussion of 'agile' methodologies (XP, Scrum, Crystal, FDD, DSDM, etc.) includes summaries of various approaches as well as reference information, and factors to consider in choosing these approaches.

XP Resources - Large collection of resources from Ron Jeffries about 'Extreme Programming' including a discussion of how QA fits into the XP approach, XP Magazine archives with articles such as 'Test-First Design', 'Incremental Requirements', 'Extreme Programming and the CMM', and more. Also see 'The Rules and Practices of Extreme Programming ' at the www.extremeprogramming.org web site.

Scrum - Web site of Advanced Development Methods, Inc and their Scrum methodology, a team-based agile approach to iteratively, incrementally develop software with rapidly changing requirements.

Crystal - Web site about Allistair Cockburn's collection of 'shrink-to-fit, human-powered software development methodologies' based on the premise that each project needs an approach appropriate to its particular member talents and people issues, and that efficiency is best maximized via frequent deliveries and improved communications. Methodology suggestions are grouped by team size (no larger than 50 as of 2004) and recommended for teams at a single geographic location working on non-life-critical projects.

Practical Software and Systems Measurement - Web site with extensive information on software development metrics, sponsored by U.S. government. Site contains articles, reports, examples, and a free PC-based software tool to assiste in project-specific metrics development.

Testing Education Articles - Collection of articles on software testing and the teaching of software testing from the Florida Institute of Technology funded by the U.S. National Science Foundation.

ITIL - 'IT Infrastructure Library' - a set of best-practices guides on the management and provision of operational IT Services. From itSMF, a UK-based organization comprised of 1000 companies and government organizations worldwide that is developing and promoting best practice IT Service Management standards and qualifications. There is a related British Standards Institution BS15000 IT Service Management standard and certification scheme. An online organizational ITIL Service Management Self Assessment is available.

Big Ball of Mud - Outstanding essay on the 'de-facto standard software architecture', by Briane Foote and Joseph Yoder of the U. of Illinois at Urbana-Champaign. The 'Big Ball of Mud' architecture is defined as 'a casually, even haphazardly, structured system. Its organization, if one can call it that, is dictated more by expediency than design....The overall structure of the system may never have been well defined. If it was, it may have eroded beyond recognition.' They discuss why this architecture is so popular, advantages and disadvantages, and what can be done to improve such systems.

The Frameworks Quagmire - Article from the Software Productivity Consortium summarizing and exploring the conflicts and complexities among various software process standards existing in the late 1990's - CMMi, SW-CMM, SDCE, Trillium, IEEE, FAA-iCMM, EIA-632, NATO-AQAP, MIL-STD-498, ISO/IEC-12207, ISO-9000, etc.

Satisfice.com Web Site - James Bach's Satisfice.com Web Site with a great collection of his articles on various aspects of software testing.

Bret Pettichord's Software Testing Hotlist - Web site with links to various test and QA-related info; good list of test automation articles and other useful web sites.

BetaSoft Web Site - QA and Testing discussion forums, jobs and resumes, other resources.

SQATester.com - QA and Testing information, discussion forums, other resources.

Seven Steps to Test Automation Success - Good introductory article on how to approach automated testing; by Bret Pettichord.

When Should a Test Be Automated? - Another useful introductory article on automated testing, by Brian Marick.

Test Tools for Free Short article by Danny Faught discussing some free test tools, along with the basics of 'freeware', and information on a testing freeware newsletter. (Note: Many free web testing tools are also included among the web testing tools listed in the Softwareqatest.com 'Web Test Tools' page .)

Loadtester.com - Resource for performance testing, performance monitoring, and capacity planning as it applies to software application development. Although focus is web applications, site has information about other applications. Includes articles, forums, other resources, and news.

Evaluating and Choosing the Right Tool - Elisabeth Hendrickson describes a five-step process for comparing, evaluating, and choosing the right test tool; from the Stickyminds.com web site.

Totally Data-Driven Automated Testing - Introductory article by Keith Zambelich on 'data-driven' and 'keyword-driven' automated testing. Article is from 1998 but information in the article is still valid and applicable.

JUnit.org - Site for test/development engineers using JUnit or one of the other XUnit testing frameworks. Has many useful articles and resources on automated Java regression testing and 'agile' testing processes in general.

Java GUI Testing - Short discussion of automated Java GUI testing issues, includes interesting discussion of methods of identifying a component in a GUI hierarchy for use in developing automated test scripts.

Function Point FAQ - Basics of the Function Point approach to estimating software development effort, based on an end-user/functional view of a software application's 'size'.

Illustrative Risks to the Public in the Use of Computer Systems - Enormous list of software, system, and related problems compiled by Peter Neumann/SRI International. Organized by categories such as space, defense, medical, stock market, elections, insurance, cryptography, etc. Includes related book list, other information. (Also see 'Risks Digest' listed below.)

Process Improvement Case Study Featuring Reviews and Inspections - Article titled 'Process Improvement: Case Study of an Improvement Program Featuring Reviews and Inspections' in Software Quality Professional magazine.

Will Bugs Eat Up the U.S. Lead in Software? - Business Week (International Edition) article comparing present state of U.S. software industry to U.S. automobile industry in the 1970's, when Japan took away huge market share with better and cheaper products by adopting Deming's and Juran's quality approaches. Discusses a possible repeat with the current US software industry losing software development market share to countries such as India. Entire issue and cover story reports on software problems - discussion of past problems, problems with software engineering, and possible fixes. Also see similar Infoweek article - 'The Big Picture: Killer Apps And Dead Bodies'.

ARIANE 5 Flight 501 Failure Report by the Inquiry Board - A rare and instructive detailed public analysis of a major software failure - the 1996 launch failure of the new Ariane 5 rocket. This is the official report of the inquiry board appointed by the French National Center for Space Studies and the European Space Agency. Also see the article 'Design by Contract: The Lessons of Ariane' which includes a discussion of the code reuse issues brought to light by the Ariane 5 failure.

Eiffel FAQ - FAQ site for a programming approach, based on the ideas of Bertrand Meyer, with the goal of improving software component reusability, extendibility and reliability using assertions, preconditions, and postconditions.

Internationalization Testing - Article at Sun's web site; includes guidelines and good check list of questions for testing an internationalized software product.

German-Language QA/Testing Web Site - German-language QA and testing web site, with articles, downloadable software and documentation, discussion forum, book reviews.

Risks Digest - Digest of the 'Forum on Risks to the Public in Computers and Related Systems'. Includes latest issue and archives covering software and system problems, vulnerabilities, disasters; based on the comp.risks newsgroup.

SEI Capability Maturity Models - SEI's CMM/CMMI web site, with info including the SW-CMM (Software Capability Maturity Model), P-CMM (People CMM) SA-CMM (Software Acquisition CMM), SE-CMM (Systems Engineering CMM), and IPD-CMM (Integrated Product Development CMM).

Construx Software Resources - Site with many useful resources such as CxOne, a lightweight, tailorable, modular, and scalable software engineering framework, estimation info and resources, various checklists, and Steve McConnell's 'Software Survival Guide' website.

CM Crossroads - Online community and resource center for configuration management, with library of resource links, interactive discussion forums, monthly newsletter, Career Search and Jobs Board. Also see the CmWiki , an open collaboration site for information about CM that includes information about CM, Build Management, Change Management, and concepts, templates, tools, and processes. Also links to an ITILWiki.

CM FAQ - Configuration Management FAQ edited by David Eaton; includes 'What is CM?', 'How should a CM system relate to process enforcement?', CM books and other resources, etc.

SR/Institute's Software Quality Hot List - Extensive collection of links to many QA and testing-related articles, resources, etc.

Uniform Computer Information Transactions Act (UCITA) - Text of controversial new laws (formerly Uniform Commercial Code Article 2B) concerning software quality. This essentially implements new laws in all 50 states in the U.S. Additional info on UCITA controversies at Ralph Nader's CPT (Consumer Project on Technology) web site and Cem Kaner's BadSoftware.com web site. .

How To Build Reliable Code - Old but still-relevant article from a past issue of Byte Magazine.

NASA SW Engineering Lab site - NASA's Software Engineering Lab web site; includes publications such as "Recommended Approach to Software Development". Though this publication is several years old, it is a good online guide to the SW development process with extenisve sections on System Testing, Acceptance Testing, and other QA and testing issues. Other publications freely available at the site include "Cost and Schedule Estimation Study Report", "C Style Guide", "Software Measurement Guidebook", and "Improving the Software Testing Process...".

FDA Medical Device Software Validation Guidelines - FDA's 1997 draft guidelines for medical device software validation.

Negotiating Testing Resources - Excellent article by Cem Kaner about testing project planning and budgeting; from a 1996 software quality conference.

Software Engineering Resources - Large collection of useful information and links to many other sites and resources, all related to the SW engineering process including project planning and management, metrics, risk analysis, programming methods, OO SW engineering, testing, QA, CM. From R.S. Pressman, author of the book 'Software Engineering, A Practitioner's Approach'.

Software Technology Review - Software Engineering Institute's technology descriptions listing - summaries of many software terms and technologies such as COM, OO Design, Function Point Analysis, etc

Software Test Coverage Analysis article Article containing a good discussion of test coverage analysis from Bullseye Testing Technology, maker of "C-Cover Test Coverage Analyzer" tool.

Software-engineer.org - Comprehesive collection of resources on software engineering including QA and testing. Includes articles, tools listings, and extensive collection of other web resources in a wide variety of useful categories.

comp.object FAQ - Extensive FAQ for object oriented subjects; includes some info about object-oriented testing.

CMMI (Capability Maturity Model Integration) - A suite of process improvement models for product and service development and maintenance. The suite includes the CMMI-SW model, and there is a 'staged' and 'continuous' version. Each of the CMMI models can be coordinated with other CMMI models to enable enterprise-wide process improvement. CMMI-SW builds on the previous SW-CMM model which is being 'sunsetted'.

Return to top of Resources Listing
Web QA and Testing Resources

Load Testing Of Web Sites - A collection of useful information on various aspects of performance testing, from Scott Barber's web site. Topics include: "Pinpointing and Exploiting Specific Performance Bottlenecks", "Common Performance Testing Challenges", "How Fast is Fast Enough", and "Introduction to Performance Testing". Although some of the information is not specifically oriented to web performance testing, it is still highly applicable.

Load Testing Of Web Sites - Article from IEEE Internet Computing about web load testing; useful overview from 2002.

Evaluating Web Sites for Accessibility - Article on the World Wide Web Consortium web site's 'Web Accessibility Initiative' section on how to assess and test web sites for accessibility issues.

Keynote Systems Resources Page - Useful collection of articles and information on web site performance testing. Also see the related site performance indices which lists a variety of business, consumer, government, and other web sites along with their 'performance index'.

CMSWatch - Web Content Management web site from CMSWorks Inc. with information, news, opinions, analysis, products, best-practices.

Handling and Avoiding Web Page Errors - Three part series from Microsoft site; covers sources of common Web page errors, how to handle run-time script errors, and techniques for avoiding preventable errors.

Testing Database-Driven Web Sites - Article on web testing from 'DBMS Magazine' (now 'Internet Systems' Magazine).

Return to top of Resources Listing

See: SOURCE: For URLS of each above:

http://www.softwareqatest.com/qatlnks1.html#GENERAL

posted in QA - 0 replies