A typical test cycle often involves testing the specific features that form part of a release, raising bugs and then retesting the bug fixes to verify they are fixed. Following that, there is usually some regression testing to be done, based on an assessment of risk and confidence, to make sure there are no new bugs and the release is ready to go.
This is in pretty simple terms but once that release is deployed to production and the post launch checks are carried out, the tester's job is done until the next release is ready for testing, right?
Not so fast, outside of any specific releases or new features to test there is still much testing that could and should take place.
Here's why - away from any deployments of new features, updates, bug fixes, hotfixes or any changes to the code, there are still a bunch of things that can and often do change.
These other things that can change can sometimes have a big detrimental effect and sometimes small (but still detrimental) impact on how the user experiences your website or application, such as browsing, searching, finding, purchasing, enquiring, downloading, logging in, applying, pretty much doing anything = experiencing.
And I'm not talking about outages, such as when AWS goes down about half the Internet becomes unresponsive. I'm not talking about monitoring and alerting when there are outages. Although, it is useful to know what happens to your website or SaaS product when a particular service that you rely on has an outage - but that's a story for another post.
Here, I'm interested in what things can and should be tested even when there are no upcoming code deployments or releases taking all our attention.
Let's take a look at what those are.
Thinking about the major desktop browsers, new major versions of Chrome are released roughly every 6 weeks with minor updates and security fixes every 2 or 3 weeks. Other browsers, such as Firefox, Edge and Safari also release updated versions on a frequent basis, although Edge is Chromium-based, same as Chrome.
These new versions introduce new features but they may also change or deprecate features or support for certain aspects that your website uses or even relies on.
For example, there has been a gradual move to using HTTPS instead of HTTP over the past decade. In Chrome version 68, released in July 2018, any web pages that were not using HTTPS were marked as "not secure". And then in Chrome 94, released in September 2021, Google Chrome rolled out HTTPS-First mode and began forcing sites to open in HTTPS where possible for users who enable this capability.
This example is a little extreme to prove my point, but if your site did not have HTTPS configured and did not have an SSL certificate then you would have seen your traffic steadily drop away as more people upgraded to newer versions of Chrome that preferred HTTPS over HTTP. In fact, it became more difficult to even access a website that used HTTP and did not have a valid SSL certificate.
There are many more less severe examples of this where parts of your website might not display or work as originally intended or stop working altogether because newer browsers no longer support it or have moved onto a different approach.
Occasionally, Chrome and other browsers introduce bugs into their own browsing experience. And so a function that worked perfectly fine in your site or app may stop working in newer browser versions because Chrome (or Safari, Firefox, etc.) introduced a bug. I know this happens because it has just happened on one of the SaaS platforms that I am involved with testing.
Mostly, this is to demonstrate that new browser versions can and do break existing website functions, sometimes in a minor inconvenient way and sometimes in a major way.
There are a number of strategies for handing new browser versions - you can make sure you are testing on the latest version or latest 2 versions of each browser, you can have a bunch of automated tests that might test on the latest or latest 2 versions of each browser, or you can do some testing using canary or beta builds to check for potential problems before the full version of that browser is released.
My preference is not to leave everything to automated tests but do some exploratory testing in new browser versions on a regular basis. It will depend on your project as to which browsers are priorities and how much time is available for exploration in each new browser version.
Onto the major operating systems and new versions of those.
For Apple devices, there is a new major version of iOS and iPadOS every year and minor version updates throughout the year. Each year there are usually older devices that can no longer be upgraded to the latest OS and so get stuck on a particular iOS version. For example, I have an iPhone 8 now stuck for ever on iOS 16.
For Android devices, there are major versions released every year but the situation is much more complex as not all devices automatically get the new versions. Each manufacturer will roll out the new OS versions to specific handset models over a period of time. Some manufacturers will roll out their flavour of the new Android version to specific handset models.
This means that potentially there are a wide variety of different devices with different OS versions (and different browser versions installed) that might be accessing your website or mobile app.
As with desktop browsers, some changes to operating systems can have an impact on how your website is experienced.
One example is that with iOS 15 released in September 2021, the address bar for Safari was moved from the top of the screen to the bottom by default. It was easy to move it back to the top if you wanted to by changing the settings but it did cause some problems with some of our customer websites, specifically where modals or popups had a button near the bottom of the screen that all of a sudden was getting covered by an address bar. If that button was important, such as to move to the next step in a sequence or close the popup then the user couldn't see it and they would get stuck. Purchase journeys and villa booking journeys, for example, could not be completed.
For mobile apps, with most of the mobile apps we test, we'll install the beta versions of each major OS update to carry out testing before they are released to the public. However, minor version updates can incorporate changes to watch out for too.
As with browser versions, there are strategies for handling new OS versions and incorporating them into testing. We maintain a library of devices where we are looking to cover the most recent 2 major OS versions across phones and tablets, plus a range of Android handset manufacturers.
We have found, certainly with mobile apps, that having some dedicated testing of the app on each new major OS version is very helpful. This could be a run through of the regression tests (not automated) or some in depth exploratory testing for each new major OS version. Some of the larger 'minor' OS version releases may benefit from this approach too.
If your website uses external integrations or APIs then these can change and introduce problems into your site or application. The integration may even break entirely. Usually, the third party should communicate any upcoming changes to APIs, such as deprecating particular endpoints or making changes to response structures but sometimes changes are introduced without any warning.
As I mentioned above, I don't mean outages from external APIs but actual changes made to external APIs that your website or application relies on.
For example, for a SaaS I was involved in testing several years ago, there was an integration with an external service that several customers of the SaaS used frequently and depended on. The external integration deprecated several endpoints but didn't communicate those changes - or at least those emails went into spam and did not end up in someones inbox. The Engineering team were unprepared for the changes and they essentially broke the service that the customers were using. If we had been regularly testing those external APIs we could have picked up the issue at an earlier point and have got workarounds or remediation worked out for affected customers. There were reasons why we weren't actively testing the application, one main reason was that there was no active development taking place any longer. However, this severe issue in particular is one of the inspirations behind this post - even when there is no active development taking place there should still be regular testing if you have customers paying for and depending on that application.
Authentication of external APIs can also be a problem, if a token or certificate is relied on and expires or does not renew then the integration will stop working.
Similar to third party integrations, a payment system may roll out changes and whilst they should be communicated they still result in problems. Obviously if a change breaks your checkout then it becomes very urgent to investigate and fix the problem.
Sometimes issues with payment systems are restricted to particular types of cards or different payment methods, such as the use of 3D Secure or the use of certain buy now pay later platforms.
Users are hopefully actively using your application, product or website. They will always encounter less than ideal situations or problems due to a variety of factors - the task they were trying to accomplish, their tech setup (browser, device, internet connection, etc.), the time and headspace they have available - they're in a rush, they're clicking quickly to finish the task, they're not reading or understanding what is presented to them and then they get into a muddle.
Sometimes users may run into problems because the UI allowed them to do something that it should not, such as uploading a file and allowing the user to click to the next step before the file finished uploading. This can cause a state where the user starts to run into errors because the action from the previous step did not fully finish.
Smaller issues that users encounter lead to friction using your product and a buildup of minor issues can lead to a prospective customer clicking away to a competitor, or a user to stop using your product. Sometimes the user or customer may contact the support team to complain but often they won't, they will quit silently.
A tester is in a great position to do something about this and actually use the product or website to find and report these issues. Or by taking customer complains or reports of problems and trying to reproduce them and investigate what led to the problem.
This type of testing, whether it is reproducing issues reported by users or actually using the product should be carried out frequently and with an open mind. Highlighting real problems that customers are getting into, investigating, reproducing and pushing the bug reports through to getting fixed and deployed provides measurable improvements to the product, happier customers and reduces pressure on your customer support team.
Some other aspects to the website or application can change frequently and sometimes can cause issues.
Content can become out of date or may no longer be relevant. Products or services may be discontinued, help or supporting content may need revising or removing. Removing the content may cause internal links that reference that content to become broken. URLs may still be accessible, and indexed by search engines, but should redirect to other relevant content.
Special offers that have an expiry date may still be promoted, or products may be discontinued or out of stock but still promoted in key positions, such as the home page. Events may have finished but can still be signed up for. Locations are being promoted or offered in search results but there are no longer any hotels, or villas, or cottages, at that location.
As features change sometimes help content needs to be revised or updated to align with the recent changes.
There are many content related things that can be tested on an ongoing basis.
If you have a separate content team, product marketing team or eCommerce team that deals with these content related items, then assistance can be offered to help tidy up all the places affected when content is removed or products are out of stock, etc. Some websites grow to thousands of pages and so content can be difficult to keep track of and it is not as simple as carrying out a site-wide scan to check for broken links (although that does help). Sometimes it is processes that can be documented and improved.
There is always something to test, not for the sake of testing it or as busywork but to continue to find real problems that are affecting users and website visitors. Reporting and fixing these issues, shining a light on areas that could be better and improving processes will make a difference and lead to a better product overall.
Photo by Chris Lawton on Unsplash
