Python/Spark/AI Developer – Remote
per year
per year
Experience
4 years' professional Python development experience, 2 years' production experience developing Spark or comparable distributed data pipelines.
Description
Our client is seeking an experienced Python/Spark/AI Developer to join a technically driven team focused on modernising data platforms and building scalable AI-enabled solutions. This role is ideal for a developer who enjoys working with distributed data, clean and testable Python, APIs, and modern lakehouse technologies. Key Requirements Bachelor's degree in Computer Science, Engineering or equivalent experience.
4 years' professional Python development experience. 2 years' production experience developing Spark or comparable distributed data pipelines. Strong Python 3.12 skills, including type hints, Pydantic and modern packaging practices.
Solid Apache Spark/PySpark knowledge, including Spark SQL, DataFrame APIs, partitioning and performance optimisation. Experience with Delta Lake or equivalent technologies such as Iceberg or Hudi. Strong SQL skills and the ability to translate legacy T-SQL logic into Spark SQL.
Practical experience with pytest and automated testing. Experience developing REST APIs with FastAPI or equivalent technologies. Familiarity with Docker-based development environments.
Strong Git and CI/CD practices.
Advantageous
Experience integrating LLMs and AI services. Knowledge of data privacy, PII handling, POPIA/GDPR concepts and data governance. Exposure to Hive Metastore, Trino or Apache Ranger.
Azure data platform experience, including ADLS Gen2, Synapse or ADF. Experience with observability tools and structured logging. Previous data migration and pipeline parity experience.
Databricks or equivalent Spark certification. Should you meet the requirements for this position, please jobs@mspstaffing.co.za . You can also contact the IT team on XXX-XXXX or visit our website at https://mspstaffingza.co/ NOTE: When replying to the advert, also include the reference number in the subject line.
Correspondence will only be conducted with short listed candidates. Should you not hear from us within 3 days, please consider your application unsuccessful.
Submit your application for this position at MSP Staffing LTD
Submit your application for this position