Join our team to develop and maintain a cutting-edge data collection, storage, and distribution platform designed to handle large-scale data from various sources, including vendors, research providers, exchanges, and web scraping initiatives. You’ll play a critical role in making this data accessible to key enterprise functions, including Operations, Risk, Trading, Compliance, as well as systematic and fundamental portfolio managers. In addition, you'll contribute to building internal data products and analytics to drive efficiency across the organization.
Key Responsibilities
- Web Scraping & Data Collection. Develop scripts, utilize APIs, and leverage web scraping tools to extract data from diverse sources.
- Greenfield Data Platform Development. Assist in building and maintaining a new data platform on Snowflake and AWS, ensuring scalability and reliability.
- Pipeline Enhancement. Understand existing data pipelines, enhancing and adapting them to meet evolving data requirements and new business needs.
- Onboarding New Data Providers. Collaborate with stakeholders to onboard new data sources, ensuring smooth integration with the platform.
- Data Migration Projects. Lead and support data migration initiatives, ensuring data is transferred accurately and efficiently.
Requirements
Technical Proficiency
- Strong expertise in SQL for querying, managing, and optimizing databases.
- Proficient in Python for automation, data processing, and scripting.
- Experience with Linux for system operations and automation.
- Solid understanding of containerization technologies such as Docker and Kubernetes.
- Knowledge of AWS cloud services for scalable infrastructure.
DevOps Skills
- Hands-on experience with Kubernetes for orchestrating containerized applications.
- Proficiency with Docker for container management.
- Familiarity with Jenkins or similar CI/CD tools for automation and continuous integration.
Other Skills
- Excellent communication skills to work across teams and with various stakeholders.
- Ability to work collaboratively with a focus on delivering robust and scalable solutions.