• VSO designed an extremely modular system, always up-to-date system to transform a legacy, outdated data warehouse for the Kennedy Center.
• Modular system’s data load time was reduced from 26-36 hours a day, to a 6-minute load time.
• “Turned the idea of a new data warehouse on its head”, achieved the concept of MVP, data is accessible to users, allowing users to do self-service of the data

Project Summary

VSO worked with Amazon Web Services (AWS) Professional Services to transform a legacy and outdated data warehouse for John F. Kennedy Center for the Performing Arts into a fully functioning modular data system. The old system was not fit for the high percentage of decisions being made which utilized Kennedy Center data. VSO leveraged the AWS suite of tools and services which assisted this transformative initiative to succeed. In addition to integration of AWS cloud services, VSO provided valuable expertise with data tools and architecture to build and develop code within the data warehouse.

Points for this technical requirement:

1. A high percentage of customer decisions were being made with data. That data is analyzed well, but the results of the data analysis lacked breadth.
2. There was not enough data to make these decisions which led to not knowing where to start or searching for proxy data.
3. Building data warehouses was one thing but moving everything in a data warehouse to the cloud was a whole different level of complexity.
4. Using VSO’s team and Amazon cloud solutions was essential to the success of this project.

Problem

Kennedy Center’s legacy data warehouse and hardware were outdated. Specifically, it was not designed to serve emerging needs or to match the plan of action for the next five years which was laid out by the CIO. Another data warehouse could be purchased as a potential solution but this offered no efficiencies and exceeded the Kennedy Center budget. The performance and stability of the data was not proficient for the growing needs of a modular system. Likewise, the functionality was poor - a single day’s worth of data took 24-36 hours to process.

Proposed Solution and Architecture

The proposed architecture was to plan on an MVP (minimum viable product) which was designed to do whatever it takes to get rid of the old data warehouse running on a physical server. Amazon Web Services has many tools available to help with this process, and VSO has experience with the data tools which led to building the code within the warehouse. AWS provided valuable help and resources with their training and tools while not interfering with the actual prototype itself.

Prior to AWS and VSO Kennedy Center made attempts to address this:

Step 1: Discern what the future holds for the Kennedy Center’s technology growth.

1. No intention of replacing the SAN (storage area network) hardware & want to move to the cloud but not sure where to start.
2. Hired a consultant to help which didn’t work.

Step 2: Spoke with AWS on many different topics having to do with this issue. AWS had many tools the Kennedy Center could use for their data.

Step 3: Do a “rapid prototype on a database to answer…are these tools the right tools, can we get to a working prototype of a new data warehouse that will solve existing reliability concerns and also set us (the Kennedy Center) up with a good foundation to take it to the future.”

Step 4: Wanted an MVP (minimum viable product) which was to do whatever it takes to get rid of the old data warehouse running on the physical server. It must be:

1. Reliable and Scalable
2. Modular Design to add data
3. Fix performance issues and act like more sophisticated tools where people could explore data
4. Do not engage with anyone who gave them a product that they did not know how to operate or expand. In the end they wanted it to be self-sufficient.

Step 5: Partnering with VSO and Amazon allowed the Kennedy Center to believe self-sufficiency was possible. Following the MVP instead of replacing the data warehouse gave them the following:

1. The prototyping approach ensured that they would have something usable that they understood how it worked by the end.
2. AWS trained them with the tools available but did not interfere with the actual prototype itself.
3. AWS offered project management and introduced the KC to experts for architecture and development. VSO offered experience with data tools, expertise and the architecture to help them build/develop their code within the warehouse.
4. Utilizing a simple clean design of the data warehouse architecture coupled with great documentation on functionality and the AWS tools means the Kennedy Center could build additional data points into the system, and act on the data.

Outcomes Of Project and Success Metrics

Using the AWS data migration services to on-premise databases while using change data capture to keep data in sync has effectively had the data warehouse team for the Kennedy Center go from building a four-terabyte database every night to maintaining all of the source tables up-to-date within seconds. As a result, the modular system data load time for a single day’s worth of data was reduced from 26-36 hours, to six minutes.Notes:

1. Use change data capture, inherent in Microsoft sequel server databases
2. Using AWS data migration services to connect to on premise databases using change data capture to keep data in sync.
3. Went from building a four-terabyte database every night to all of the source tables are kept up to date within milliseconds
4. No incremental load, always kept up to date
5. Extremely modular system
6. “Turned the idea of a new data warehouse on its head”, achieved the concept of MVP, data is accessible to users, allowing users to do self-service of the data
7. Able to add more data sources in the future, finally having a platform to do that
8. Stable, solid foundation with AWS
9. Reduced 26–36-hour day data load time to six minute load time.

Hash Tags

#design-thinking, #agile, #architecture, #migrate, #deploy, #cloud, #big_data, #data_warehouse, #deployment, #NotForProfit