SoundCloud: Amplifying Data Processing Efficiency
SoundCloud partners with Zencore for migration to Google Cloud Platform.
Migration to Google Cloud Platform
Project Location:
Berlin, Germany
Industry:
Audio Streaming
Use Case:
Migration of Apache Spark workloads from on-premise
Website:
soundcloud.com
In the pursuit of excellence, SoundCloud recognized the need to optimize their data processing infrastructure to meet the demands of a growing user base. They embarked on a journey to migrate their Apache Spark workloads processing data on HDFS from their on-premise data center to Google Cloud.
About SoundCloud
SoundCloud, an artist-first platform empowering artists and fans to connect and share through music, is dedicated to delivering captivating experiences to listeners globally. As a trailblazer in the music industry, SoundCloud constantly pushes the boundaries of innovation, seeking cutting-edge solutions to elevate their services and provide unparalleled value to their diverse community of artists, podcasters, and music enthusiasts.
Project Challenges
SoundCloud sought to leverage scalable resources and increase cost-effectiveness by migrating to the cloud. They needed a partner with expertise in data solutions and migrations to execute this ambitious migration seamlessly.
Scope of Work
Zencore partnered with SoundCloud to migrate 28 products/workloads, comprising 200 Spark Jobs, 400 DAGs, and 100TB of data from their on-premise data center to Google Cloud Platform. The migration involved transitioning business-critical HDFS/Spark jobs to Cloud Composer and Dataproc Serverless.
Key Achievements
- Reduced operational expenses and significantly decreased processing time
- Aligned with SoundCloud's overarching strategy of becoming fully cloud-native
- Enhanced scalability and flexibility of data processing infrastructure
Solution
Zencore engineers worked hand-in-hand with SoundCloud engineers to plan and execute the detailed migration process. The goal was to adopt a "lift & shift" strategy to seamlessly transition their Apache Spark data products and Apache Airflow DAGs to GCP's Cloud Dataproc and Cloud Composer, respectively.
Technologies
- Apache Spark
- Apache Airflow
- Google Cloud Platform
- Cloud Dataproc
- Cloud Composer
Business Goal
SoundCloud's primary goal was to optimize their data processing infrastructure, leveraging the scalability and cost-effectiveness of cloud computing. By migrating to Google Cloud Platform, they aimed to enhance their operational capabilities, reduce expenses, and align with their strategy of becoming fully cloud-native.
Business Value
The migration to Google Cloud Platform enhances SoundCloud's operational capabilities and reinforces their commitment to delivering a seamless and immersive audio experience to users worldwide. The partnership with Zencore lays the foundation for future growth and success, positioning SoundCloud to embrace the future of cloud computing and continue redefining the audio streaming landscape.
“I was extremely pleased with the performance and expertise demonstrated by Zencore throughout our migration to Google Cloud Platform. Their meticulous approach ensured a seamless transition. Zencore's partnership was instrumental in our success, and I look forward to continuing our collaboration on future endeavors.”