The Role: Senior Data Engineer (Python, Spark)

The Location: Hyderabad

The Team:
The team works in agile environment and adheres to all principles of Agile. As a Senior Data Engineer, you will work with a team of intelligent, ambitious, and hard-working software professionals. The team is independent in driving all decisions and is responsible for the architecture, design, and quick turnaround of development of our products with high quality.

The Impact: 
As a Senior Python Data Engineer with Spark skills, your contributions will have a significant impact on our organization's data-driven decision-making processes and overall data infrastructure. Here are some key areas where your work will make a difference:

•    Scalable Data Processing
•    Data Quality and Consistency
•    Performance Optimization
•    Collaborative Problem-Solving
•    Continuous Improvement and Innovation
•    Knowledge Sharing and Mentorship

By leveraging your expertise in Python and Spark, you will contribute to the development of a robust, scalable, and high-performance data infrastructure, empowering our organization to extract valuable insights and drive data-informed decisions.

What’s in it for you? 
Be the part of a successful team which works on delivering top priority projects which will directly contribute to Company’s strategy. Drive the testing initiatives including supporting Automation strategy, performance, and security testing. This is the place to enhance your Testing skills while adding value to the business. As an experienced member of the team, you will have the opportunity to own and drive a project end to end and collaborate with developers, business analysts and product managers who are experts in their domain which can help you to build multiple skillsets.

Responsibilities: 
•    Design, develop, and maintain scalable and efficient data pipelines using Python and Spark to process large volumes of data.
•    Collaborate with cross-functional teams, including data scientists, analysts, and software engineers, to understand data requirements and translate them into technical solutions.
•    Implement data ingestion, transformation, and integration processes to ensure data quality, accuracy, and consistency.
•    Optimize Spark jobs and data processing workflows for performance, scalability, and reliability.
•    Troubleshoot and resolve issues related to data pipelines, data processing, and performance bottlenecks.
•    Conduct code reviews and provide constructive feedback to junior team members to ensure code quality and best practices adherence.
•    Stay updated with the latest advancements in Spark and related technologies and evaluate their potential for enhancing existing data engineering processes.
•    Collaborate with infrastructure teams to design and deploy Spark clusters, monitor system performance, and manage resource allocation effectively.
•    Develop and maintain documentation, including technical specifications, data models, and system architecture diagrams.
•    Stay abreast of emerging trends and technologies in the data engineering and big data space and propose innovative solutions to enhance data processing capabilities.

What we’re looking for: 

Basic Qualifications:
•    6-8 Years of experience in Software Development
•    Bachelor's or master's degree in Computer Science, Engineering, or a related field.
•    Strong experience in Python programming and software development, with expertise in building data-intensive applications.
•    Proven hands-on experience with Apache Spark, including Spark Core, Spark SQL, Spark Streaming, and Spark MLlib.
•    Solid understanding of distributed computing concepts, parallel processing, and cluster computing frameworks.
•    Proficiency in data modeling, data warehousing, and ETL techniques.
•    Experience with data serialization formats such as Avro, Parquet, or ORC.
•    Familiarity with big data technologies such as Hadoop, Hive, or HBase.
•    Knowledge of SQL and experience with relational databases.
•    Strong problem-solving and troubleshooting skills, with the ability to analyze complex data engineering issues and provide effective solutions.
•    Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams.

About S&P Global Commodity Insights:

At S&P Global Commodity Insights, our complete view of global energy and commodities markets enables our customers to make decisions with conviction and create long-term, sustainable value.
We’re a trusted connector that brings together thought leaders, market participants, governments, and regulators to co-create solutions that lead to progress. Vital to navigating Energy Transition, S&P Global Commodity Insights coverage includes oil and gas, power, chemicals, metals, agriculture and shipping.
Commodity Insights counts over 4,300 people in more than 30 offices worldwide.