Lead, Big Data Administrator

Company Description See how Epsilon powers connected marketing by unlocking the full potential of data and identity. Overview Employer: Epsilon Data Management LLC Job Title: Lead, Big Data Administrator Job Requisition: 6630.7112.9 Job Location: Irving, Texas Job Type: Full Time Responsibilities Duties: Responsible for deploying and managing large-scale distributed systems, ensuring their performance, availability, and scalability; Oversee the installation, configuration, and maintenance of Hadoop ecosystems, including HDFS, Yarn, Hive, and Spark, as well as Kafka clusters; Implement and manage high-availability and disaster recovery strategies for both Hadoop and Kafka environments; Performance tuning, including resource allocation, cluster optimization, and monitoring, is a core responsibility to maintain system efficiency; Handle data pipeline orchestration and ensure seamless integration between Kafka producers, consumers, and downstream processing frameworks; Ensuring data security through encryption, access control, and auditing mechanisms; Troubleshoot and resolve complex issues related to cluster health, data loss, or Kafka offsets while maintaining minimal downtime; Automation of repetitive tasks using tools like Ansible, shell scripting in order to enhance the operational efficiency; Collaborate closely with development, data engineering, and analytics teams to support evolving business requirements and optimize workflows; and Staying updated with emerging technolo