Introduction
If you’ve been following our deployment and operationalization of the Aether Network, you know that ONF has been deploying private 5G connected edge sites across the globe in support of Project Pronto and various commercial partners.
We’re pleased to report that the deployment and turn up of each new successive Aether Edge site has been becoming easier, faster and more automated with the development and adoption of an ever-improving automation tool chain and associated processes. In most cases, configuration and initialization of an Aether Edge can now be completed in just a few hours with minimal on-site assistance.
In this blog, learn about the automation tools and processes that are being used to simplify and accelerate Aether Edge deployments, including:
- An overview of the Aether automation tool chain
- Lessons learned bringing up new Aether Edges, including what worked and challenges encountered
- A roadmap of planned enhancements for Aether Edge automation tools and processes.
Inside an Aether Edge
Each Aether Edge requires installation of several ONF open source components including SD-Core, SD-Fabric, ONOS, Stratum, and SD-RAN. Externally developed open source tools and technologies are also needed to orchestrate and automate the platform infrastructure, including Kubernetes, Rancher, Docker, Helm, Ansible, Prometheus, and Grafana.
The Aether Edge architecture consists of the following components:
- RAN: For RAN, integrated small cells (eNB/gNB) or disaggregated RAN (RU/DU/CU) provides the 4G/5G connectivity services to end devices, and connects to the User Plane Function (UPF) running on SD-Fabric. For most sites, radio hardware from Sercomm is used which includes an eNB providing LTE (in Q3’2021 when small cell radios become available) and 5G connectivity.
- Underlying edge cloud infrastructure: To simplify Aether Edge hardware installation and reduce operational costs, the on-premise equipment is standard COTS hardware. It includes:
- Compute servers for running ONF's ONOS SDN stack, and edge applications on x86-64-based server hardware.
- A network fabric built with Wedge100BF Switches from Edgecore, running Stratum and programmed using P4 on Intel Tofino silicon.
- A management server and switch to bootstrap and tie the deployment together.
- Run-time software: Software stack is deployed on top of the bare metal infrastructure with Kubernetes Cluster (K8s), and the Aether platform components such as SD-RAN, SD-Core, and SD-Fabric, shown in Figure 1.
- Services and Applications: Services and edge applications run on top of these platform components. Aether Edge users and administrators can onboard edge applications to their cluster using containers and K8s as an orchestrator.
Figure 1. Aether Software Components
To build an edge site, Aether has multiple components and subsystems that need to be configured. To simplify the installation and configuration process and reduce risk of human error, ONF invested in deployment automation tools and processes. These tools enable the configuration of complex components while minimizing human involvement. An additional advantage is that these tools and processes will facilitate efficient rollout of multi-Aether Edge site deployments on a large scale in the future.
Aether Edge Toolchain
To simplify the setup and cabling of hardware and reduce possible setup error, ONF leverages the NetBox IP Address Management (IPAM) and Datacenter Infrastructure Management (DCIM) tool to document every device, MAC address, cable, IP address, and so on at each Aether Edge. Once a site is fully described in this system, easy to follow instructions are generated that include equipment locations, power and network connections, and all the other necessary details that the on-site personnel need to perform the installation.
Figure 2: Aether Edge Hardware and Network as deployed for Project Pronto
After physical installation is complete, the information in NetBox is used to programmatically generate a complete set of configurations needed for bootstrapping the Aether Edge. The site is bootstrapped using a menu-driven USB installer that leverages iPXE, to load the base Ubuntu OS and perform security configurations over a mutual-TLS channel to provide a secure software foundation at the edge site. Once the OS is bootstrapped, Ansible playbooks are used to perform low-level configuration of the physical hardware, configure the management network and install software such as Docker, which is used to deploy higher level tools like Kubernetes and Rancher.
VPN and Runtime Deployment Automation
The first step in the deployment process is the setup of the VPN. The management server within the Aether Edge is used as a IPSec VPN gateway, which provides a secure connection to Aether Central for control plane traffic (user registration and MME) and centralized lifecycle management (CI/CD, monitoring, etc). Terraform and Ansible are used to automate the configuration of a VPN between the public cloud and management server. Once these are tested within the CI pipeline, Terraform creates the cloudside multi-homed VPN tunnels and adds required routing information.
The runtime environment consists of installing prerequisites to run K8s cluster on the compute servers and fabric switches. Centralized CI/CD is used to deploy the Aether runtime software. Configuration details for the new Aether Edge are added to the central cluster, the configuration is reviewed and automatically tested on our staging environment. Once the configuration passes continuous tests, the central CI/CD (Jenkins) runs Terraform deployment scripts for the runtime deployment, to check dependencies and perform component upgrades or installations if needed. Any further changes or runtime environment upgrades are managed through the same CI/CD process.
Connectivity Service Configuration
Connectivity service components are configured in two phases: configuration of the central connectivity services and configuration of the eNB radio base stations. A mobile core configuration is added for the new site and new subscribers are provisioned in the Aether central subscriber database (HSS). We currently automate this using a CI job to update the configuration, which is reviewed and verified in a staging environment before going into production. In the near future, a web GUI will provide the user provisioning through an administrative management portal and APIs for user provisioning will be provided to extend the system.
As a variety of eNB radios from various vendors can be used and are compatible with Aether, but there are challenges to automate the eNB configuration because of the unique Element Management System (EMS) requirements of each vendor. ONF is exploring options to use the CWMP (TR-069) protocol which standardizes this configuration process in order to automate the eNB deployment. Currently, we either pre-configure the eNB before installation or configure the installed eNB manually using a web UI.
Pronto Team Insights
Pronto project collaborators were the first sites to benefit from the automated Aether Edge deployment and they have provided valuable feedback that has helped us to improve the overall process. The first site at Stanford took over four days to bring up due to some final debugging of the automation process. Once these issues were addressed, the software installation at the remaining three Pronto Aether Edge locations each took less than a day.
In addition, Pronto collaborators identified several areas for enhancing the documentation, and this feedback has since been used to revise and improve the Aether Edge documentation.
Through continual improvement, the installation process continues to become more streamlined and more highly automated. COVID added to the initial challenge of rapidly scaling the network, but these restrictions and challenges have ultimately pushed us to create automations and tooling that are now resulting in a highly optimized deployment process.
The Aether team is excited for the rollout of more Aether Edges!
More Resources
- Blog: Aether: Automated CI/CD Systems Increase Reliability and Operational Efficiencies
- Blog: Aether - Rapidly Deploying New Edge Sites Around the World
Blog: Keeping Up with Aether - Aether Website
