The International Software Testing Qualifications Board (ISTQB) , one of the world’s leading software certification bodies defines software quality as “The totality of functionality and features of a software product that bear on its ability to satisfy stated or implied needs”, whereas the IEEE defines software quality as “The degree to which a system, component, or process meets customer or user needs or expectations.”
To get a better idea of what these definitions mean in the context of setting SMART goals for software quality, it is necessary to understand how the lack of Quality has impacted some major organizations in the past, through the following Case Studies.
The Mariner 1 Spacecraft, 1962. NASA – In 1962, NASA launched an unmanned space mission for data-gathering to fly past Venus. It barely made its way out of Cape Canaveral, before veering off course. With a dangerous crash-landing on their hands, NASA activated the self-destruct button. The subsequent investigation revealed that a very simple software error, a missing hyphen in a line of code, led to an estimated loss of 18 Million USD (as of 1962), let alone the loss of face.
EDS Child Support System, 2004, Government of the United Kingdom – In 2004, the UK government introduced a new system to manage the operations of the Child Support Agency (CSA). A Texas-based company called Electronic Data Systems (EDS) was awarded the contract and called it CS2. A plethora of problems plagued it from the initial go-live with over 1,000 reported problems, of which 400 had no known workaround , resulting in around 3,000 IT incidents a week . An internal memo leaked later described the system as badly designed-tested-implemented . Not only did it lead to the UK government spending 768 million GBP instead of the estimated 450 million GBP , but the contractor also announced a fiscal loss of USD 153 million in the subsequent year.
Bitcoin Hack, Mt. Gox, 2011 – Bacon in the 2010’s Mt Gox was the largest bitcoin exchange in the world until a software glitch led to the creation of unredeemable transactions costing them USD 1.5 Million in lost bitcoins . This severe loss, in an already fluctuating market, led to the company declaring bankruptcy a few years later.
Considering the above three cases, a clear inference is that the software failed in its ability to satisfy stated or implied needs (as defined by ISTQB) or meet customer or user needs or expectations (as defined by IEEE). Thus comprehensively failing the spirit of both the accepted definitions of Software Quality.
Even though companies spend amounts of money, human resources, and time on QA processes still not being able to achieve the goals is clearly due to a lack of Software Quality.
Even the 12th edition of the World Quality Report 2020-21 lists “ QA is a key business priority for organizations to achieve digital transformation”, the Consortium for Information & Software Quality (CISQ) records in its 2020 report that poor quality software costs US companies over USD 2 trillion annually .
Diligent QA Leaders would always seek to ensure optimal software quality for their organizations and set quality goals in the software testing process to avoid such fatal consequences, and this article aims to aid them in their endeavor of Setting SMART goals for Software Quality.
Table of ContentsThe major objectives of any software testing process are outlined in the diagram above. Any attempt to set quality goals for software development will need to take these objectives into account.
These software quality objectives provide a useful yardstick for the QA team to evaluate and modify any software testing goals based on their progress. These also enable us to understand the activities which lead to the fulfillment of these objectives to meet our software quality goals.
1. Understand what’s most important, get buy-in from your stakeholders
Before we set any goals for software quality, it is important to understand what are the customer’s primary objectives from the software test process. Depending on what the user wants, it is important to ask a variety of questions –
The primary goal for software quality management in software engineering should be to measure software performance across the entire user journey. That means considering the entire user journey from login to checkout. The users should be guaranteed the desired experience at every stage of the user journey.
This stage of the process helps to check the boxes for Building Confidence and Sharing Information with stakeholders from the objectives mentioned above.
2. Define a SMART Goal
Once the QA Leader has a roadmap for setting the software quality goals, they should set SMART goals to obtain clarity and motivation to complete actionable tasks. By having defined objectives and fixed deadlines, they are more likely to achieve success in what matters rather than just pluck low-hanging fruit.
The SMART acronym stands for –
Pro Tip: Continuous Testing helps the companies develop software that is customer-centric and simultaneously in line with healthy development practices. Bear in mind that the resultant code’s accuracy can only be guaranteed when it is tested on a real device cloud . BrowserStack’s cloud Selenium grid equips QAs with 3000+ real browsers and devices for automated testing so that every feature can be monitored in real-world circumstances.
Some examples of such quality objectives are:
An example of a bad quality objective could be something vague like “Reduce the number of bugs in every sprint”, as it is not really achievable or time-bound!
Defining SMART goals helps in Evaluating Work Products, Building Confidence, and Sharing Information with stakeholders from the objectives mentioned above.
3. Elucidate the activities required to reach the goal
The activities required to achieve the set goals for software quality can be defined in 3 ways:
Preventive Actions
Appraisal Activities
An appraisal is an objective evaluation of an artifact against a defined set of standards. It consists of –
These are periodic checks of the test process, the progress of test execution, and even the documents created during the software development lifecycle to ensure that the activities being performed on a daily basis are in sync with the user goals and customer requirements.
This is the backbone of any software quality management process. Tests must be performed often and as early as possible. Test automation and parallel testing are two ways to speed up the test process and ensure faster test coverage in minimal time. The entire QA process hinges on the use of a real device cloud . Without real device testing, it is not possible to identify every possible bug a user may encounter. Naturally, undetected bugs cannot be tracked, monitored, or resolved. Moreover, without procuring accurate information on bugs, QA metrics cannot be used to set baselines and measure success. This is true for manual testing and automation testing .
Audits are usually done by an independent auditor outside of the QA team. They evaluate the work against existing industry standards like ISO standards and so on. Audits need to be planned carefully, defined using a standard model like CMMI, and formalized into a report to share with the stakeholders. Any areas of discrepancy must be converted into a set of SMART goals to ensure full compliance with industry standards.
NOTE: BrowserStack has achieved SOC2 Type 2 compliance, uses HTTPS by default, stores credentials securely, and does much more to ensure the security of our users. Please read our security documentation for detailed descriptions of our policies and controls.
Fix/Repair Activities
These include the efforts to fix the bugs found during testing. A plethora of associated tasks like changing the team structure, testing tool, or test methodology can also be part of this activity.
This step helps to Prevent Defects, Validate test objects, Reduce Risk, Share Information with stakeholders, and Find Failures and Defects from the objectives defined initially.
4. Determine the resources available to see if they are sufficient to match your goals
There are 3 “actors” in the team who coexist and influence the team dynamic. These are the people (including their skills, availability, and knowledge), the technology (that supports the QA process), and the work processes (waterfall, agile, DevOps, and so on). There must be a balance between these three to ensure the optimal creation and fulfillment of software quality goals. However, there is a “cost” to this balance, and that can be summed up using the following formula –
A successful quality assurance plan will aim to keep the CoQ within an acceptable range in the overall project’s budget.
5. Time your activities to suit your goals and objectives
Some of the activities described above are once-off activities like Audits, or defining SMART goals. However, others are more continuous. Any processes related to review, verification, and validation will be repetitive across sprints for an agile process for instance.
6. Monitor Quality Objectives continuously
Once the goals have been set, it is important to monitor them continuously in order to ensure that they are relevant to the project and that they are optimal in satisfying the overarching need to adhere to agreed quality standards and user expectations.
Conclusion
In the course of this article, we have defined what software quality means, why is it important for software engineering teams, and how to set about defining goals to meet it. Creating SMART goals for software quality helps keep the process within an agreed time frame and budget, and keeps the team focused on what is most critical for the success of a software project.
When done correctly, the returns from setting Software Quality Goals in terms of team and customer satisfaction, defect reduction, and product quality are undeniable. Add to that the advantage of being seen as acting towards a goal, rather than reacting to what is happening in the SDLC process, which builds customer confidence in the product and leads to a real quality product and not one that barely does what it is set out to do.
