Python/Spark/AI Developer

R750K - R1M

per year

Job Description

A company that runs a privacy-preserving data collaboration platform is seeking a AI Developer who will replatform legacy T-SQL systems onto modern Spark/Delta Lake pipelines, build robust FastAPI services around them, and integrate AI features.

Responsibilities

Data Engineering

Replatform T-SQL into Spark SQL and build production PySpark pipelines on Delta Lake.

API Development

Build secure, idempotent FastAPI services with robust job-status semantics and authentication.

Quality & Validation

Write type-hinted, tested Python (pytest) with CI gates; validate migrated pipelines against legacy systems with parity evidence.

Spec-First Delivery

Work autonomously from design docs and ADRs, keeping documentation current alongside code changes.

Minimum Requirements

Experience

4 years in professional Python development, with 2 years building production Spark/distributed data pipelines.

Education

Bachelors degree in Computer Science, Engineering, or equivalent practical experience.

Certifications

Databricks Certified Developer for Apache Spark (or equivalent) is advantageous. Python (3.12): Modern idiomatic Python, strict static analysis (mypy/pyright), Pydantic, ABC patterns, modern packaging ( uv /Poetry), and CLI tooling.

Apache Spark / PySpark

Production OSS Spark experience (DataFrame API, Spark SQL, tuning, driver/executor architecture, Spark Connect).

Lakehouse Formats

Delta Lake (or Iceberg/Hudi) including MERGE INTO , schema evolution, time travel, and idempotent writes.

SQL

Advanced, dialect-portable SQL skills to translate complex legacy T-SQL logic into Spark SQL.

Testing & CI/CD

Pytest (fixtures, tiered suites), Git/GitHub flow, PR-driven workflows with linting ( ruff ) and type-check gates.

APIs & Containers

FastAPI development and daily development using Docker Compose stacks.

Preferred Skills AI/LLM Engineering

Provider-neutral LLM integration, structured output validation, local/hosted inference, and guard-railing.

Governance & Security

PII tokenization, hashing schemes, and data privacy awareness (POPIA/GDPR).

Cloud & Platform

Azure data estate (ADLS Gen2, Key Vault), Trino, Hive Metastore, and OpenTelemetry observability.

Benefits

Competitive salary based on experience (salary can potentially be more based on experience/skills) IF you meet the above requirements and want to make a career-changing move, apply today by emailing your CV to itcareers@hireresolve.za.com

Apply for Python/Spark/AI Developer

Submit your application for this position at Hire Resolve

Add a few notes about your experience first for a more personal draft. You can edit the draft before submitting.

PDF, DOC, or DOCX (max 5MB)

Ready to Apply?

Submit your application for this position