Introduction
Delivering software to market at speed is an important objective of the DevOps approach. The ability to deliver quality code at speed requires the adoption of automation. Test Automation plays a key role on the road to continuous integration/continuous delivery by following a Shift Left testing strategy, which involves introduction of testing early in the development cycle. Test Automation includes the practices of collaboration using CI/CD pipelines, execution of tests in an automatic fashion, managing test data, utilizing the test results and providing constant feedback. Test automation helps to improve a product’s quality, increase velocity and decrease time to market. It provides a reliable platform and efficiency for the teams to perform faster.
While Aether is an open source platform, ONF also runs a live production Aether network with more than a dozen sites around the world. We are using a DevOps approach operationalizing this network, and Test Automation is an essential component to achieve the uptimes we require while still providing a fluid and rapid development environment with new features being rolled out on the live Aether network on a continual basis.
The Aether project includes multiple complex components such as SD-Fabric, SD-Core, SD-RAN and ROC. All are quickly maturing, with additional help from automation processes. The software development process for the Aether project has become faster than ever, with high expectations for the user experience, overall software quality and network uptime. Test Automation at ONF has evolved into a highly collaborative discipline with an effective strategy that helps identify defects much faster, while delivering features and updates more frequently, thereby enhancing productivity and speeding innovation.
Aether Test Automation
The goal and objective of Aether test automation is to achieve quality software at speed. This requires building a framework that provides highly scalable and low maintenance code which will help cover various categories of tests utilized for different stages in the development life cycle. With the current automation framework, tests are integrated into CI/CD pipelines at various stages to provide faster feedback loops enabling the development teams to deliver updates in an efficient manner.
Test Strategy
In our vision to bring Aether to production grade quality, a comprehensive testing strategy has been put in place. Aether is a complex product made up of many mature subsystems. Testing such software requires a well-defined testing strategy. Some of our key strategies follow:
- Achieve a good return on investment by defining the right automation scope, KPIs for the project and aligning the priorities well
- Meet quality requirements by designing and selecting the right automation tools that meet the testing requirements
- Execute well by defining various test environments and providing a good execution plan
- Continuously improve through frequent analysis and fine tuning
Automation Framework
The automation framework must support Aether’s testing strategy which includes several types of test categories such as Unit, Smoke, Functional, Integration, Scale/Performance, Soak/Stability, Real-time IoT, User Acceptance Tests, System Tests, etc. To complement these categories, the automation framework needs to be robust and highly scalable.
Aether test automation framework includes libraries and tools that allow both component level and integration level tests. Framework supports tests to cover simulated as well as hardware environments. The following block diagram shows components of Aether at a high level and various automation frameworks that are used.
Figure 1. Aether Test Automation Frameworks
SD-Fabric is being tested by enhancing existing frameworks such as TestON and TestVectors. SD-Core is being tested using a framework called NG40 from NG4T company. This framework emulates 4G/5G RAN components required to test the SD-Core functionality. It can also emulate UEs for scale testing. Robot Framework, an open source test automation framework, is being used to cover ROC and SD-RAN level tests. By using Robot Framework, functional end-end integration test coverages have been achieved. We are expanding the coverage to GUI scenarios (enterprise portals) using Selenium with Robot Framework.
Test Execution Strategy
Aether’s test execution strategy includes testing at every stage of the CI/CD pipeline. Introducing testing early in the development process helps identify and facilitate fixing bugs at a much earlier stage which accelerates the delivery schedule. Code does not move forward until all the tests have passed and a feedback loop is established in case of a failure. Having this process in place creates a collaborative environment for developers and testers to improve and implement quality strategies.
A number of smoke tests have been created and integrated to the CI jobs based on their respective repos. As an example, smoke tests include component level verifications on repos as well as integration level verifications. The following diagram shows how smoke tests are invoked as part of the CI jobs.
Figure 2. Smoke Tests In CI Jobs
Quality strategy also includes execution of tests on various platforms/pods (Staging, QA) after a new image is deployed. Apart from the testing performed as part of the CI jobs, there are a number of tests that are executed after the images from the CD post-merge jobs are deployed. Testing on the staging platforms is an important part of the deployment process. Validating thoroughly on the staging environment is a critical step before deploying the images on the Production POD. After a successful upgrade of component(s) on the staging POD, tests from regression, functional and integration suites are executed. Apart from these, there are tests that run continuously to validate bandwidth utilization and ping tests to simulate real traffic. Staging POD is also soaked for a particular period of time where extensive scale, performance and soak tests are performed. The test automation team closely monitors for any issues during this phase and takes appropriate actions.
The following diagram shows various CD test pipeline jobs that are invoked on various pods after an image is published.
Figure 3. CD Test Pipeline Jobs
Aether is also being tested for Scale and Performance. Various tools and frameworks are used to collect data from multiple targeted runs. Data collected is analyzed to meet the necessary product KPIs. Soak testing is where the test platform is soaked for several weeks while executing chaos tests, failure/recovery tests and scaling the system there by validating the stability of the system.
For Aether Release 1.0, staging and production pods were integrated with sample IoT applications that simulate real-time traffic scenarios. Tests related to bandwidth utilization are executed to validate upstream and downstream latencies thereby measuring the performance of the network. Grafana based dashboards constantly monitor all the Aether Edge sites and report the status of automated KPIs.
Conclusion
By following a Shift Left testing strategy we’ve improved the quality and uptime goals for Aether during Release 1.0 by introduction of testing early in the development cycle. Also, integrating tests into CI/CD stages allowed us to fix bugs, create new builds and test builds productively. With effective test automation, developers are able to trigger tests and deliver code at speed. Our goals are to continuously improve the automation frameworks, improve test automation coverage and discover more bugs before deploying to production. As Aether matures, our vision is to continuously evolve and improve the test automation processes to deliver high quality software with large uptime goals.
Aether Test Automation Framework Resources
- https://github.com/opennetworkinglab/OnosSystemTest/tree/master/TestON
- https://github.com/stratum/testvectors
- https://www.ng4t.com
- https://robotframework.org
- https://www.selenium.dev
