Wipro

Business Context

Alight Solutions runs HR and benefits products that generate large volumes of workforce and application data. The client needed a governed AWS foundation where that data could be ingested from its applications and SaaS systems, analysed by a data team, and served to Tableau and downstream consumers, next to a reliable platform for the supporting microservices.

The Challenge

Two different workloads had to share one secure account. The analytics side needed governed ingestion, validated data, and fast Redshift reporting. The application side needed repeatable EKS deployments with safe rollbacks. Both had to keep sensitive data private, stay inside approved regions, and be cheap enough to run every day without surprises on the bill.

Approach & Architecture

I led and mentored a seven-person team and was the on-call escalation point for production issues the rotation could not resolve. On the data side I set up the AWS Databricks environment: a customer-managed VPC in Terraform, clusters in private subnets with Secure Cluster Connectivity, S3 gateway and STS interface endpoints, and a cross-account IAM role for the control plane. Governance ran through Unity Catalog external locations backed by an S3 data lake, so analysts queried governed tables rather than raw files. Ingestion came from an EventBridge-scheduled Lambda that paged the Alight API into raw S3 and from AWS Transfer Family SFTP for flat-file exports. The data analyst and architect I worked with owned the Spark transformations that ran on top of that platform. Alongside it I built an event-driven analytics pipeline where AppFlow pulled Workday and ServiceNow data, a Glue crawler catalogued the RDS SQL Server source, Step Functions orchestrated the flow, a Python Lambda ran the ELT, Athena validated the output, and curated tables landed in Redshift for Tableau. On the application side I ran an EKS platform deployed through GitHub Actions over OIDC, with CodeDeploy canary releases that rolled back automatically on a CloudWatch alarm.

Tools & Stack

What I Owned

• The AWS Databricks platform: customer-managed VPC, private-subnet clusters with Secure Cluster Connectivity, S3 gateway and STS interface endpoints, cross-account control-plane IAM, Unity Catalog external locations, and JSON cluster policies for auto-termination, instance limits, autoscaling caps, and cost tagging
• An event-driven analytics pipeline from AppFlow, a Glue crawler, and Step Functions through a Python Lambda ELT into Redshift, validated with Athena and cached in Redis for Tableau
• Redshift performance work with distribution and sort key design to cut cross-node shuffle, scheduled VACUUM and ANALYZE, and RDS Performance Insights tuning of slow SQL Server queries
• An EKS delivery platform on Terraform and GitHub Actions with OIDC, CodeDeploy canary releases, and automatic rollback triggered by a CloudWatch alarm
• Account guardrails and secrets handling: AWS Organizations SCPs, KMS, WAF, ACM, SOPS with Secrets Manager, and S3 block-public-access, with Terraform state locked in S3 and DynamoDB
• GuardDuty threat detection off CloudTrail, VPC flow, and DNS logs feeding the wider Azure Sentinel SIEM, plus Prometheus, Grafana, Datadog, and CloudWatch for metrics and alerting

Outcomes

• Traced a climbing AWS bill to a Lambda re-reading the same S3 object in a loop on every run, fixed it, then right-sized instances and tuned autoscaling for about $70,000 a year in savings
• Gave the data team a governed lakehouse and gave analysts trusted Redshift and Tableau datasets to build on
• Made application releases safe to ship daily through canary rollouts that reverted themselves on failure with no manual step
• Defined a one-hour RPO and three-hour RTO with the application and product owners, then confirmed them by actually restoring from backup instead of assuming the backups worked
• Built a Python MCP server on EC2 so engineers authenticate once and reach Confluence, Jira, and ServiceNow through Claude, with results scoped to each person's existing permissions
• Wrote the team's setup SOPs in Confluence and mentored engineers on Terraform and CI/CD, which removed single-person bottlenecks

Lessons Learned

Running a data platform and an application platform on one account taught me that governance and cost are not afterthoughts. The two habits that mattered most were watching the bill like a metric and testing a restore instead of trusting a backup.

Explore more