Data Engineer
at Intuition IT Solutions UK Skilled Worker EmployerAbout the job
Role:
- Configure Databricks Auto Loader or Apache Spark Structured Streaming to securely subscribe to the relevant SAP AEM topics.
- Develop the PySpark/SQL data pipelines to process the incoming JSON payloads through the Delta Lake layers.
- Carry out data modelling in Databricks.
- Expose the Delta tables to the visualisation layer.
- Ensure that the data pipelines are optimised. Skillsets:
- Expert in Apache Spark Structured Streaming and Databricks Auto Loader.
- Fluent in PySpark and Databricks SQL.
- Proven experience building Databricks architecture patterns. Familiarity with Unity Catalog for data governance and access control.
Sign up for free or
Log in to view all job posting insights and get access to competitive insights.
Job ref blurredText
Posted on blurredText
Last checked on blurredText
Closing on blurredText
0 comments
0 requirements
Discuss this job
Share your intel on this vacancy and help others - anonymously. Get notified automatically of new comments when starred or saved to Tracker.
Log in to view and post comments.
My Private Notes
Saving private notes will automatically star the vacancy to your Jobs page.
Log in to view and save your private notes.