Data Engineer

Full Time
Main Location
New York City, NY, United States
Open jobs
Data Engineer

The Data Analytics Development team is looking for a Data Engineer that will work on the acquisition, normalizing, and cataloging of large sets of data.

Currently, the team’s application is hosted entirely on AWS and uses Angular for the front end, Python/Django with AWS-managed Postgres for its API, Elasticsearch as a search engine, Python/Celery for back-end data processing, and various AWS offerings. This includes AWS CodePipelines, CloudFormation, EC2, and various other offerings. In the near future, the team is looking to work with SQS, AWS Lambdas, AWS SageMaker, and data streaming offerings.

Responsibilities:

  • Extraction, transformation, and loading of data from a wide variety of data sources using Python, SQL, Elasticsearch and AWS technologies.
  • Providing users access to datasets using REST and Python APIs
  • Categorizing, cataloging, cleansing and normalizing of datasets
  • Communicating with business users and technology stakeholders.

Required skills/experience:

  • Python
  • Relational SQL database development (SQL Server, Postgres)
  • Distributed queueing offerings like RabbitMQ, SQS or similar
  • 5+ years of experience working in IT
  • 3+ years of experience of Elasticsearch, Kibana and Logstash (ELK Stack)
  • 2+ years of experience of AWS Big Data offerings like EMR, Redshift, Athena
  • Masters or Bachelor’s degree in Computer Science, Statistics, Informatics, Information Systems or another quantitative field.

Desirable skills/experience:

  • Ansible, Chef, Java, etc.
  • AWS serverless offerings
  • Big data tools: Hadoop, Spark, Presto, Kafka, etc.
  • Broad understanding of Equity fixed income, derivatives, futures, FX, or other financial-services instruments

Other Qualifications:

  • Excellent listening and communication (both oral and written) skills.
  • Self-starter and critical thinker, takes ownership of own projects and makes improvement suggestions for the entire infrastructure.
  • Proactive, assertive and attentive to details.
  • Can work independently and in a collaborative environment.
  • Can handle several projects with different priorities at the same time in a fast-paced environment.
  • Excellent self-management and problem-solving skills.

Mission
We're a community of women leveraging our connections into top companies to help underrepresented women get the roles they've always deserved. Simultaneously, we work to build truly inclusive hiring processes and environments where women can thrive and not just survive.
Are you hiring? Join our platform for diversifiying your team
Data Engineer
Millennium Management