Lead Data Engineer
We’re looking for a smart, self-motivated Lead Data Engineer to join our great team and lead our data engineering effort. This role will be responsible for coordinating the current team of 6 and owning the data development process, setting best practice and standards, interpreting needs of stakeholders along with overall responsibility of programming and maintaining complex data pipeline.
About you:
- You’re comfortable with taking the lead in a group and helping others
- You care about getting the best possible outcome, you have a passion for what you do which you can clearly convey by your actions.
- You have an eye for detail and order, being able to spot problems in code or data which others might miss or take longer to find.
- You have a desire to explore and test concepts, ideas and theories.
- You have a strong sense of responsibility, and the ability to breakdown and manage workflow for a team of data engineers.
- You have a keen interest in cloud computing (we work with AWS) and have deep knowledge on cloud and data security, networking and running complex data pipelines in the cloud.
About Us:
We’re small (20+ people and growing), fast, innovative and varied group, solving big problems in real estate data and analytics. We’re growing at a rapid pace and are seeking enthusiastic, creative, intelligent and fun individuals to join us in helping build the best platform on the market. In return we can offer you a fun and hard working environment where you can instantly see your contribution to the company’s success.
What you’ll be doing:
The hire will be responsible for expanding and optimising our data and data pipeline architecture, as well as optimising data flow and collection. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys leading a small team who are optimising data systems and building them from the ground up. The Lead Data Engineer will support our software developers and data scientists on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects. They must be self-directed and comfortable supporting the data needs of multiple teams, systems and products. The right candidate will be excited by the prospect of supporting our next generation of products and data initiatives.
- Create and maintain optimal data pipeline architecture,
- Assemble large, complex data sets that meet functional / non-functional business requirements.
- Identify, design, and implement internal process improvements: automating manual processes, optimising data delivery, re-designing infrastructure for greater scalability and flexibility, etc.
- Continually improve the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using AWS big data processing technologies such as Glue.
- Work with stakeholders in the wider Product and Sales teams to assist with data-related technical issues and support their data infrastructure needs.
- Work with the wider team to keep our data secure across AWS.
- Work with existing data and analytics experts to strive for greater functionality in our data systems.
Our Stack
The current data processing and pipelines are written in PSQL, Spark (Python and Scala) using AWS Glue to orchestrate. We work mostly with batch data rather than streams.
Excellent SQL skills are required, and knowledge of window functions, optimising queries, geospatial queries, GIS, and general data wrangling are a must.
We are open to people from a diverse range of backgrounds and would have a preference for people with an understanding of most of the following;
- Postgres and PostGIS
- AWS Glue/S3/RDS/EC2
- Spark (Scala/PySpark) and GeoSpark/GeoMesa
- Java/Scala/Python
We are looking for someone with proven experience of managing a small data engineering team and working with stakeholders and directly with the CTO and Product Owner. We are still a small team, so you must be comfortable with conversing across all aspects of development and reacting quickly to new information.
If you’d like to apply, please send us your updated CV and a cover letter explaining why you feel that this is the right job for you, to hiring@realyse.com
