Data Pipeline Engineer


Infrastructure/Networking/Solutions Architecture
Remote Work Option:
Jul 16, 2021
Job ID:
Employment Type:
Pay Rate:
Base Salary:

We're looking for a Data Pipeline Engineer to support our infrastructure team that is currently working on several pressing projects. The data that funnels through this team is critical and is used throughout the organization! This Engineer will be working to bridge some gaps between on-prem Hadoop infrastructure and the new AWS based infrastructure.


The responsibilities of this role will include building and operating the company's largest data infrastructure system supporting millions of users at 100+ PB scale, and scaling and operationalize big data technologies like Spark, Kafka, Presto, Flink, Hadoop in both on-premise and AWS environments.  This Engineer will be working to bridge some gaps between on-prem Hadoop infrastructure and the new AWS based infrastructure. The ideal candidate will have outstanding communication skills, proven abilities with Hadoop and AWS infrastructure  strong business acumen, and an innate drive to deliver results. Additionally, you will be providing feature implementation in Python using Hadoop client API's, monitoring the ingestion pipeline, collecting and adding metrics to existing dashboards, and answering customer and user questions through Slack about the data and latency.

Must Have: 

  1. Excellent expertise in Python.
  2. Experience in building data pipelines to migrate data from on-prem to AWS cloud.
    1. Basic understanding of Data Modeling and Good SQL Skills.
    2. Understanding of version control like Git and CI/CD workflow.
  3. Experience in Hadoop (YARN, Hdfs, Hive, distcp) and AWS Cloud ( S3 , IAM, Kubernetes) 
    1.  Experience in containerization (Docker, EKS)
  4. Experience in collecting metrics and building dashboards in DataDog.
  5. Expert in Scheduling Complex DAG's in Airflow. 


  1. Experience in search engines for logs like Splunk.
  2. Experience in Object oriented programming like Java  or Scala.
  3. Experience in Spark, Iceberg. 

Apply for “Sr. Java Engineer” position

Apply for “Sr. Java Engineer” position

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.