How Levels.fyi Cuts Cloud Bill By 15%

Author

Govardhana M K
October 02, 2024

TechOps Examples

Hey — It's Govardhana MK 👋

Along with a use case deep dive, we identify the top news, tools, videos, and articles in the TechOps industry.

IN TODAY'S EDITION

🧠 Use Case Deep Dive

  • How Levels.fyi Cuts Cloud Bill By 15%

🚀 Top News

📽️ Videos

📚️ Resources

🛠️ TOOL OF THE DAY

Bytebase - The GitHub/GitLab for database DevOps.

  • The only database CI/CD

  • A single place to manage environments, database instances, database users with optional Terraform integration

🧠 USE CASE DEEP DIVE

How Levels.fyi Cuts Cloud Bill By 15%

Levels.fyi was founded in 2017 to help people understand how levels are mapped across tech companies based on scope and responsibility, as well as employee transfers—not compensation.

The platform began with a simple goal but quickly grew to millions of users, powered by an unconventional yet effective backend—Google Sheets.

Chapter 1: Scaling with Google Sheets

Despite its limitations, Levels.fyi scaled to millions by employing several key strategies:

  • CDN / S3 and caching: These mitigated performance bottlenecks caused by the Google Sheets API.

  • Node.js (Nestjs): Core backend services were developed using this framework.

  • Static site generator: This approach allowed the platform to scale with minimal resources.

  • API Gateway and EC2: These services powered the API server, improving performance and reducing latency.

  • Lambda functions: Used for aggregating analytics and processing data.

Chapter 2: Tech Stack Evolution

As the platform grew, Levels.fyi’s tech stack evolved:

  • V2: Used Node.js for backend services, API Gateway and EC2 for the API server, and Lambda for data aggregation.

  • V3: Built fully on AWS, using AWS RDS (PostgreSQL) for storing critical data and Metabase for generating visuals.

By 2020, Google Sheets' limitations—such as lack of SQL-based analytics, data processing timeouts, and scalability challenges—became apparent, prompting the team to fully migrate to AWS.

Chapter 3: The Cloud Savings Journey

In 2024, Levels.fyi started a mission to reduce cloud costs. Here are the steps they took:

What’s Behind the Cloud Costs?

They used AWS Cost Explorer to review their cloud spending. In the last 12 months, they delivered 650 million pages to 26 million unique visitors, with nearly 25% of their costs linked to CloudFront. Rather than chasing obvious targets, they focused on uncovering hidden savings.

Ref: Levels.fyi

1. Unused EC2 Instances:

They identified unused EC2 instances set up for temporary employee access and promptly shut them down, along with their EBS volumes.

Leveraging reserved instances and group buying through Pump saved around $700/month on EC2 and ECS.

2. ElastiCache Instance:

A mysterious ElastiCache instance showed CPU activity despite not being in use.

After running DBSIZE and INFO commands, and checking metrics like connected_clients and total_commands_processed to confirm it wasn’t essential, they backed it up and decommissioned it, eliminating unnecessary costs.

3. S3 Historical Data:

To manage growing S3 storage costs, they set up lifecycle policies to move older data to Glacier Deep Archive.

By temporarily pausing S3 listeners, they avoided overwhelming the backend during the migration process.

4. ECS Autoscaling:

The team fine-tuned their AWS Fargate autoscaling configuration to better match traffic patterns, cutting down on over-provisioned resources.

5. RDS & Lambda Upgrades:

Upgrading RDS from t3 to t4g processors and gp3 disks improved performance and reduced costs.

Migrating Lambda functions to arm64 architecture also resulted in significant savings.

6. Cleaning Up ECR & CloudWatch:

They implemented ECR lifecycle rules to keep only the last 25 Docker image versions and adjusted CloudWatch log retention to save on storage costs.

ELBs and NAT gateways were also reviewed to eliminate unused resources.

Final Results:

By making these changes, Levels.fyi reduced their cloud spend by 15%.

Ref: levels.fyi

With ongoing plans like switching from CloudFront to CloudFlare, they continue to fine-tune their infrastructure for optimal performance and cost efficiency.

This use case is another testament to the hidden blind spots in cloud costs, and optimization is always a work in progress.

P.S. We helped enterprises with digital and cloud consulting, including Stanford University, Hearst Corporation, and 16+ clients around the globe.

Want help with technical consultation?

If you're a business, you can book time here

Looking to promote your company, product, service, or event to 20,000+ TechOps Professionals? Let's work together.