• Approximately 12,000 routes moved without disruption
• Estimated $1.5million in savings for CMS

Customer Experience Overview

VSO provides a broad range of services to CMS for the entirety of the CMS cloud ecosystem. This includes application development, cloud migrations, shared cloud-services infrastructure, security, and networking. VSO provides expert AWS cloud infrastructure support, application migration, governance, security, and comprehensive architecture redesign for Centers for Medicare and Medicaid Services (CMS). The CMS AWS cloud hosts over 180 healthcare business owners utilizing more than 450 Virtual Private Clouds, thousands of EC2 instances, and petabytes of storage. VSO supports CMS Office of Technology’s (OIT) strategic advancement through important technology improvements such as the use of AWS Virtual Interfaces for large datacenter-to-cloud dataset migrations and Zero Trust Architecture for CMS executives.

Healthcare.gov (through AWS)

Perhaps most importantly, VSO provides senior experts each year during Open Enrollment to support operations for the CMS healthcare exchange, http://www.heathcare.gov. VSO works with AWS Technical Support team to ensure Healthcare.gov is available to 18 million users annually, which is particularly critical during annual Open Enrollment on CMS Marketplace as workloads surge. The transitions and migrations completed to date have increased availability, speed, and security, and have saved CMS over a million dollars annually.

Office of Infrastructure Technology (OIT) (through AWS)

VSO assists the CMS OIT office with strategic planning activity, design, and implementation of POCs to support strategic goals. Through our partner AWS, we help drive OIT strategy by modeling important technology improvements (i.e. use of Public VIFs for large datacenter to cloud datasets) to CMS Front Office and CTO. One such effort that proceeded to full MVP was the design and development for several components of Disaster Recovery as a Service, CMS' centralized/shared DR service, including VMC on AWS, Actifio and Luminex backup to AWS.

Problem

Cybersecurity Information Center (CCIC)

Through our partner Ironvine, VSO not only provides oversight and direction to the entire Security Engineering team, but also helps bridge the gap for the PMO by working closely with government task leads and the dedicated PMs from other groups

Agency for Healthcare Research and Quality (through AWS)

VSO designed and now runs an AWS-based data science as a service solution for AHRQ Maternal Morbidity and Mortality (3M) research. This solution allows the 3M team to use advanced data science without advanced technical knowledge

Problem

Healthcare.gov Project Highlight: AWS Transit Gateway Vignette

Problem

CMS had previously implemented an overlay network using Cisco Cloud Services Routers (CSR) in the CMS AWS environment. The design pattern required two or more Cisco CSRs in each tenant VPC, causing the CSR quantity to reach over 1,000. This incurred significant cost, complexity, and management overhead in order to maintain the CSR overlay network. Additionally, the Cisco CSR architecture could not utilize the full 45 Gbps AWS backbone. A better solution was needed before Open Enrollment 2020, but there was only a short window to implement changes before open enrollment on the website began.

VSO’s proposed solution was to move all network traffic from virtual images to the AWS Transit Gateway (TGW). The Transit Gateway greatly simplifies the current network architecture, provides better visibility and performance of the CMS network, while increasing redundancy and high availability.

Process

Implementation of AWS Transit Gateway was conducted in two phases:

Phase 1 focused on shared services such as Microsoft Active Directory, Splunk, Nessus, Nagios, and SaltStack. TGW was tested to ensure it met the CMS VPC traffic isolation requirement, while providing VPC access to the shared services. The test results were successful, and CMS approved the shift to TGW.

The first migrations were shared services traffic from 475 CSRs to AWS Transit Gateway in the AWS US East region. Phase 1 execution spanned two months consisting of a nine-wave transition from CSR to TGW. During this time, 27,039 routes were moved without disruption. This design for shared services was a first – it enforced CMS-required isolation between environments while providing central control via one Transit Gateway with multiple routing tables and an automated deployment utilizing Python.

Phase 2 focused on accounts with connectivity to both CMSNet and the Internet. Infrastructure as Code principles were used to migrate CMSNet traffic in 15 waves to the new architecture. The result of this activity was 2,138 routes were moved from 405 CSRs to TGW attachments.

Prior to the Transit Gateway transition, VPC-level CSRs were limited so that routes could only point to a single CSR. The impact was that there was no redundancy for auto-failover if a CSR went down. With Transit Gateway, the architecture is more reliable, cost-efficient, and highly available than the CSR solution.

There were significant costs, complexity, and management overhead to run and maintain this overlay network. There was a small window to implement changes before open enrollment on the website began. VSO provided planning, cost modeling and forecast, implementation, and migration from Cisco CSR to AWS Transit Gateway.

Outcomes of Project and Success Metrics

During this period, VSO transitioned CMS from 1022 to 14 Cisco CSR instances using AWS Transit Gateway without any disruption to business operations of the CMS site. By reducing licenses from over a thousand to fourteen, VSO was able to save CMS tens of thousands of dollars each month while increasing computing speed and security. The strategic effort realigned CMS to cloud best practices, mitigated performance issues, reduced application latency, offered reliable scaling and significant cost savings.

Additional benefits to CMS include items such as:

1. Approximately 12,000 routes moved without disruption
2. 1,017 route advertisements moved without disruption
3. 1,022 EC2 instances were terminated
4. Network Address Translation seamlessly moved to edge routers
5. Estimated $1.5million in savings for CMS

Hash Tags

#migration, #networking, #deployment, #orchestrated deployments, #cloud, #AWS, #healthcare, #federal, #cybersecurity