Hadoop Operation Engineer-Level 1
Santa Clara Area, CA
MindSource is seeking Hadoop Engineers that are focused on operations to administer/scale our multi-petabyte Hadoop clusters and the related services that go with it. This role focuses primarily on provisioning, ongoing capacity planning, monitoring, management of Hadoop platform and application/middleware that run on Hadoop.
- Responsible for maintaining and scaling production Hadoop, Kafka, and Spark clusters.
- Deep understanding of Hadoop/ Spark stack and hands on experience in resolving issues with Hadoop/Spark Jobs
- Responsible for the implementation and ongoing administration of Hadoop infrastructure including monitoring, tuning and troubleshooting.
- Provide hardware architectural guidance, plan and estimate cluster capacity, and create roadmaps for the Hadoop cluster deployment.
- Able to support Shift plan with some odd hours coverage on a weekly basis
- Triage production issues when they occur with other operational teams.
- Conduct ongoing maintenance across our large scale deployments across the world.
- Write automation code for managing large Big Data clusters
- Work with development and QA teams to design Ingestion Pipelines, Integration APIs, and provide Hadoop ecosystem services
- Participate in the occasional on-call rotation supporting the infrastructure.
- Hands on to troubleshoot incidents, formulate theories and test hypothesis, and narrow down possibilities to find the root cause.
Hands on experience with managing production clusters (Hadoop, Kafka, Spark, more).
Strong development/automation skills. Must be very comfortable with reading and writing Python and Java code.
Overall 8+ years with at least 5+ years of Hadoop/ Spark debugging experience in production, in medium to large clusters. - Senior Resources
Overall 4+ years with at least 2+ years of Hadoop/ Spark debugging experience. - Junior Resources
Tools-first mindset. You build tools for yourself and others to increase efficiency and to make hard or repetitive tasks easy and quick.
Experience with Configuration Management and automation.
Organized, focused on building, improving, resolving and delivering.
Good communicator in and across teams.