Cloud computing is causing disruptive shifts and revolutions in the information technology and business intelligence sectors. It involves delivering computing services including intelligence, storage, networking and analytics over the internet to facilitate innovation, flexible access to computing resources and reduction of the upfront costs of owning and maintaining IT infrastructure. Cloud computing underpins several IT services like online video streaming, Gmail services and basic user activities such as backing photos on a smartphone. It provides a broad access network by allowing users to access or upload data to the cloud from any part of the world using any device, provided there is internet access. Notably, there are several cloud providers with distinct layers of cloud services, such as software as a service (SaaS), platform as a service (PaaS), and infrastructure as a service (IaaS) (Riungu et al. 7). Computing services such as storage, intelligence and networking services are provided through the IaaS model. Additionally, the IaaS model allows users to control and run independent software testing. Software testing over the cloud or software testing as a service (STaaS) enables organizations to perform software maintenance, operations and testing support over web-based testing servers and frameworks. According to Riungu et al., software testing in the cloud has three facets. The first facet is the application or system that is tested (4). The application or software must be accessible online to enable cloud testing. The second facet is the testing infrastructure or platforms available in different deployment models like the private or community clouds (Riungu et al. 6). The third facet is testing the cloud environments to assess their security, performance, availability and scalability. However, software testing in a cloud environment has significant challenges that can compromise an organization’s privacy, operation and performance. This paper aims at identifying the key challenges and emergent problems of software testing in the context of cloud computing and related works toward solving the software testing challenges in cloud environments.
Cloud-Based Software Testing and Testing Methods for Cloud Computing
Testing software in a cloud environment means assessing a software application’s performance, reliability, scalability and security using testing resources found in the cloud. Quality assurance teams rely on cloud resources to test software security, functionality and usability before launching developed applications to the market. Notably, there are various types of cloud-based testing, such as functional, cloud-specific, and non-functional testing. Functional cloud-based testing includes data migration testing, compatibility testing, and exploratory and automation testing (Ali et al. 6). The primary objective of cloud-based functional testing is to ensure that the software application meets all the user expectations and functions seamlessly. Contrarily, non-functional cloud-based testing includes security testing and performance testing. This type of testing is done to validate the non-functional aspects of an application to ensure performance reliability, capacity to handle the high-user load and end-to-end security. Finally, cloud-specific testing includes disaster recovery testing, compatibility testing, and multi-tenancy evaluation.
Other testing methods for cloud computing include stress testing to evaluate the software’s ability to work at peak workloads, system testing to validate the functionality of the system features, and load testing to assess the software’s behavior under high user traffic loads. In addition, cloud-based software testing supports availability testing to ensure the software has minimal downtimes and disaster recovery testing to assess the software’s ability to minimize damages during emergency scenarios. The tests are done to assess the software’s ability to be available in minimum time during disaster recovery, its compatibility with other cloud resources such as servers and networks and its ability to handle high user load to ensure that all users share resources efficiently (Ali et al. 6). Notably, cloud testing is different from on-premise testing in that in on-premise software testing, the software is tested using equipment or resources owned by the organization, unlike cloud testing, which allows the quality assurance team to utilize third-party resources. However, despite its notable advantages, cloud-based software testing has significant problems due to its operation in a third-party environment.
Emergent Problems of Cloud-Based Software Testing
Security and Privacy Concerns
Cloud computing introduces several security challenges because the cloud services are provided by third-party vendors, and resources are accessed by several users, leading to a broader attack surface. The cloud concept is premised on the idea of a resource pool for utilization and sharing. Notably, this strategy exacerbates security risks for organizations relying on cloud resources for software testing because it increases the chances of leakage of private, valuable or sensitive information (Ali et al. 8). For instance, sharing testing resources encourages malware propagation thus increasing organizations’ vulnerability to cybersecurity threats. Additionally, cloud-based software testing increases exposure to cybersecurity risks due to the lack of standard cloud-testing methods. Cloud security testing lacks standard measures since the existing methods depend on client needs and vendor offerings. Although many third-party vendors providing cloud-based software testing services offer the option of data backup or disaster recovery, the security standards do not provide adequate protection against data theft or unauthorized access to intellectual property (Ali et al. 9). This is contrary to on-premise software testing, which is entirely secure since all the testing tools and resources are within the organization’s premises. However, the main advantage of conducting software testing over the cloud is that it enables testers and quality assurance teams to detect all security bugs and avoid costly human errors. Thus, security and privacy concerns are among the major emergent problems in software testing in cloud environments.
Challenge of Testing Diverse Components
Software testing requires the preparation of multiple test cases like compatibility, performance, and security to allow the tester to validate the functional and non-functional capabilities of the software. However, cloud-based testing has infrastructure issues with limited testing configurations that prevent testers from performing exhaustive tests to ensure diverse components are tested. For example, testers may have significant challenges in testing diverse components if the provider does not support the required test configurations (Ali et al. 7). Notably, this is a major challenge with public cloud resource providers developing their own testing architectures and standards. In addition, testers may experience additional challenges in transferring data and test configurations when changing vendors. Finally, cloud-based testing exacerbates difficulties in creating test environments that accommodate all the necessary settings when testing diverse components since prioritizing test cases is time-consuming and can lead to delivery delays (Ali et al. 9). Thus, cloud-based testing limits testers’ ability to perform exhaustive tests on diverse components due to limited test configurations provided by the vendors. However, testers and quality assurance teams can overcome these bottlenecks by defining their requirements (like test configurations) to vendors and assessing vendor services before signing agreements.
Complexities of Creating a Test Environment
Creating a test environment that contains all the necessary application test data and configurations in a cloud context is a complex and time-consuming process. This complexity poses a significant challenge to the testers as they have to maintain the pace of the short testing cycle in software development to ensure timely delivery. Besides, the complex test environment can drain resources and lead to other risks due to the wrong configuration of the test environments. For example, moving a SaaS application with multiple microservices that require each segment of users to own their databases for testing on the cloud would require a complex test environment that requires a lot of time to create, leading to halts in delivery. Moreover, the incorrect usage of cloud-based test environments can inflate software development costs thus leading to cost creep and delivery challenges (Ali et al. 13). Additionally, managing test data in cloud contexts is a complex issue for complex test environments since some testing tasks may require customer data or private information that cannot be provided legally to third-party vendors due to legal restrictions. This implies that testers and quality assurance teams must scrutinize all the test data before performing cloud-based tests to avoid legal complications about sharing client data. Thus, cloud-based software testing creates emergent problems such as time and resource-consuming test environments.
Integration Testing Issues
Integration testing in cloud-based software validation is performed on multiple components like databases, systems, networks and servers. However, since testers lack control over cloud-based testing resources, the testing exercise may be crippled by breakdown issues like database crashes, vendor server downtimes and other breakdowns on the testing resources. In addition, reliance on cloud-based software testing resources increases an organization’s vulnerability to cloud service latency, which is the delay between the service request time and the vendor’s delivery time (Ali et al. 14). Notably, the integration testing challenges and latency issues can be further exacerbated by evasive cloud providers or service vendors who neglect to inform their clients about any breakdowns or scheduled downtimes. A prime example of this scenario occurred in 2021 when Microsoft Azure experienced breakdown issues, blocking access to testers and developers who rely on the vendor’s cloud testing resources (Abdulsalam et al. 14). Thus, cloud-based software testing exposes testers and quality assurance teams to risks such as breakdowns in vendor networks, database crashes or server downtimes. This is a significant challenge because it leads to delivery delays and can increase development costs since testers lack control over the cloud-based resources.
Performance Issues
Cloud-based services such as software testing are provided in a shared environment where a single server and several resources such as CPUs, SAN storage, and random access memories (RAM) are divided among several clients. Notably, the shared resources, such as the server, can be overloaded when many clients are using the services intensively, leading to slow performance and delays (Ali and Hongqi 17). Additionally, the performance of cloud-based testing services is inconsistent as it varies from time to time, depending on the number of users utilizing the resources. The performance challenges can be worsened by bandwidth issues when the provider lacks adequate bandwidth to sustain testing. Besides limited bandwidth, connectivity issues and intensive usage of cloud resources, performance issues in cloud-based software testing can be triggered by network latency which worsens as the distance between cloud servers and clients’ request devices increases. Finally, performance problems in cloud-based software testing can be triggered by data loss, like when data is overwritten during software updates. Additionally, key data in cloud resources can be lost during an attack in case the vendor neglects frequent backups. Additional data-related performance issues in cloud-based software testing can arise from incompatibility issues since test data may not be transferable over network architectures. Therefore, cloud-based software testing exposes testers and quality assurance teams to performance-related delays from limited bandwidth, intensive use of the cloud resources, incompatibility issues and data loss.
Legal and Financial Issues
Managing test data in cloud-based software testing is a challenging issue since customers or employees may be prohibited from providing sensitive customer or production information to third parties. For example, testers can face legal limitations preventing customers from providing end-user data to overseas vendors to test new software applications (Ali and Hongqi 15). This limited test data sharing increases software testing problems in the cloud environment. However, the legal challenges regarding test data sharing can be mitigated by generating identical data for testing purposes. Notably, financial issues in software testing in cloud environments arise from hidden costs such as additional costs from improper use of test environments, additional costs for data encryption and unused or underutilized cloud resources. In addition, adopting cloud services may require employing the services of professional IT experts when dealing with multiple vendors, leading to additional costs. For example, organizations relying on cloud software testing can be forced to hire IT consultants to manage the transition leading to an increase in expenses.
Related Work Towards Solving the Emergent Problems of Software Testing in the Context of Cloud Computing
Notably, cloud-based testing has several challenges that limit its preference and adoption. The primary emergent challenges related to software testing in a cloud environment include security and privacy concerns, the complexity of test environment issues, legal issues and integration challenges. However, compared to on-premise software testing, cloud-based testing offers notable advantages as it overcomes a lot of responsibilities, expenses and human errors that are prevalent in on-premise testing. Contrary to the traditional on-premise software testing methods, cloud testing enables testers to reduce the number of manual tests through automation using preconfigured test labs and environments. As such, developers and IT experts have developed effective measures to help organizations avoid the challenges of cloud-based testing to leverage cloud technologies and resources to enhance scalability, cost-efficiency and automation. Research studies about software testing in cloud computing use new optimized load and performance monitoring tools to accelerate the testing process and identify the solutions to the identified bottlenecks. For instance, Chawla et al.’s study used a high-level architecture to assess cloud-based software testing data generation service (CSTS) and how the system can be used to solve security challenges in cloud-based testing (1309). Similarly, Abdulsalam et al. proposed a Data Protection Impact Assessment Tool (DPIAT) to enhance user and vendor accountability toward data protection and security management in cloud-based testing (6).
Solving Security and Privacy Concerns in Cloud-Based Software Testing
One of the significant steps toward mitigating security and privacy risks in cloud-based testing environments is the incorporation of a symmetric cryptosystem for authentication (Chawla et al. 1309). Symmetric key cryptography enhances security in cloud testing by enabling the testers and QA teams to encrypt and decrypt data files using the same key to protect valuable information from third-party vendors and other users utilizing cloud testing resources (Chawla et al. 1309). Another step toward mitigating security and privacy concerns in cloud-based testing is to use role-based access control for authorization. Role-based access control for authorization is a unified model that accommodates diverse control requirements for multiple authorizations without compromising security (Chawla et al. 1309). In this framework, the authenticated users are only granted access to cloud resources after successful authorization associated with user identification (Chawla et al. 1309). Additionally, security issues in cloud-based computing are addressed through Zero-Knowledge Proofs, an architect that allows users to propagate encrypted test data without disclosing data contents. The main advantage of Zero Knowledge Proof in cloud computing is validating the integrity of the vendor or servers involved in the cloud-based testing (Abdulsalam et al. 11). This is particularly important in the healthcare sector, where sensitive and valuable data is stored in the cloud. Additionally, organizations using cloud testing resources can overcome data exfiltration concerns by encrypting data at rest and in transit. Finally, testers and QA teams can limit security issues by leveraging cloud-based security testing resources like Nmap. Thus, although performing software testing in a cloud environment increases vulnerability to cyber and information threats, there is significant progress to ensure that testers and QA teams can utilize cloud-based testing resources without increasing the risk of exposure.
Solutions to Complex Test Environments and Diverse Component Testing Challenges
Notably, testers and developers experience significant challenges when testing applications with many features or when adding new functionalities to the existing software because cloud testing resources may not be equipped with test configurations for the new features, leading to incomplete test coverage. However, this challenge can be mitigated by ensuring that testers and QA teams partner with third-party vendors and communicate their test requirements to ensure that the selected vendor can deliver the required services(Abdulsalam et al. 14). The QA team must understand all the testing requirements for the developed application before migrating software testing activities to the cloud. These measures ensure that the QA team chooses the right vendor with all the testing tools for the software to eliminate the challenge of diverse component testing. Similarly, the challenge of a complex test environment can be mitigated by leveraging cloud-based test environments management resources like Plutora and Testim. These tools significantly advance the automation of software testing and the creation of complex test environments for software with many features requiring full regression tests. The tools overcome the test environment challenge by enabling the QA team to create multiple test environments simultaneously and perform new ephemeral tests on several branches(Abdulsalam et al. 13). Thus, the notable works toward solving test environment complications include innovative resources like Plutora and Testim, which enable QA teams to create multiple test environments and perform ephemeral test for several branches simultaneously.
Related Works Toward Solving Integration and Performance Issues
Testers and QA teams run into real-time monitoring and testing problems due to integration and performance issues arising from the inherent lack of control that comes with cloud-based resources. One of the notable developments in this area is the adoption of a multi-cloud strategy which involves utilizing cloud testing services from more than one third-party vendor. It involves splitting the software testing workloads between several public cloud providers to eliminate the risks of delays caused by bandwidth issues, network latency, database crashes and connectivity issues that developers can suffer due to reliance on one provider. Besides the multi-cloud strategy, large organizations are addressing integration, performance and interoperability issues by building private clouds (Ali and Hongqi 13). Building private clouds allows organizations full control over software and performance issues such as latency, bandwidth and server management. In addition, building private clouds enables organizations to have greater visibility and control over access and security issues. Finally, testers can overcome performance issues by utilizing cloud-based performance testing tools like LoadView that operate under varying bandwidths (Ali and Hongqi 14). However, using cloud-based performance and integration tools does not solve the challenge of vendor availability and reliability since the services are utilized by many users and are vulnerable to unscheduled downtimes and breakdowns. Thus, the main works towards solving performance, integration and interoperability issues in cloud-based testing include multi-cloud strategy and developing private clouds.
Related Works Toward Solving Financial and Legal (Compliance) Issues
Compliance is one of the major emergent problems of moving software testing activities from on-premise manual testing to automated cloud testing. Moving test data from internal storage to the cloud for product testing attracts compliance regulations and laws around data security, such as Payment Card Industry Data Security Standards (PCI DSS) for public retail companies in the US (Ali and Hongqi 10). One of the effective developments in this area includes increased responsibility for data security through increased awareness and encryption, especially in the multi-cloud setup. In addition, organizations are leveraging identity and access management tools (IAM)to provide more visibility and access control (Ali and Hongqi 11). Finally, organizations are managing ambiguity and overlap in compliance regulations by reducing the scope and implementing automated compliance as a strategy to reduce compliance fatigue. Similarly, organizations are implementing aggressive cloud cost optimization strategies like regular auditing and analyzing unused resources to reduce cloud computing costs. Additionally, testers apply test-driven developments with clear testing goals and requirements to minimize cost creeps due to complex and undefined test environments.
Conclusion
The integration of cloud computing services and solutions continues to accelerate across multiple industries due to the efficiency, collaboration and cost benefits associated with cloud computing. Software testing in cloud environments brings several benefits, such as automation, scalability and greater testing accuracy relative to on-premise software testing. However, cloud-based software testing has several emergent problems that limit its adoption and applicability, like security and privacy concerns, performance issues, challenges of creating complex test environments and incomplete testing as vendors may lack test configurations for new or diverse software functionalities. This indicates that cloud-based testing brings unique challenges that testers and QA teams must overcome to realize the benefits of performing tests in the cloud. The main developments toward solving the unique challenges of cloud-based testing include implementing strong security solutions such as symmetric encryption cryptosystems, role-based access, Zero-Knowledge Proof and cloud-based security features that provide backup and enhance organizations’ cybersecurity posture when utilizing cloud resources. Similarly, related works to mitigate integration and compliance issues include developing private clouds and adopting multi-cloud strategies to overcome delay risks from limited bandwidth, unscheduled system downtimes, database crashes and network latency. The related works to mitigate compliance risks include strong data information security responsibility, reducing the scope and automating compliance measures. Finally, the challenge of hidden costs from cloud-based testing can be addressed through proper cost auditing, leveraging test-driven developments and cloud-based test environment management tools like Plutora. Thus, based on this analysis, moving software testing activities to the cloud enables organizations to improve collaboration, cost control and minimize human errors, but the unique emergent challenges related to cloud computing must be addressed for the cloud to deliver its testing benefits.
Works Cited
Abdulsalam, Yunusa Simpa, and Mustapha Hedabou. “Security and privacy in cloud computing: technical review.” Future Internet 14.1 (2022): 11.
Ali, Sikandar, and Hongqi Li. “Moving software testing to the cloud: an adoption assessment model based on fuzzy multi-attribute decision making algorithm.” 2019 IEEE 6th international conference on industrial engineering and applications (ICIEA). IEEE, 2019.
Ali, Sikandar, et al. “Analyzing the interactions among factors affecting cloud adoption for software testing: a two-stage ISM-ANN approach.”
Chawla, Priyanka, Inderveer Chana, and Ajay Rana. “Framework for cloud‐based software test data generation service.” Software: Practice and Experience 49.8 (2019): 1307-1328.
Riungu, Leah Muthoni, Ossi Taipale, and Kari Smolander. “Research issues for software testing in the cloud.” 2010 IEEE Second International Conference on Cloud Computing Technology and Science. IEEE, 2014.
write