Data Engineer

Senior Data Engineer

Fortune 50 Healthcare

Brooksource

Remote

Overview

Our Fortune 50 Healthcare client is seeking a

Senior Data Engineer

to support our mission of improving the health and well-being of our members. This role will focus on

building scalable, secure, data centric solutions

and compliant data platforms that power analytics, clinical insights, and business decision-making across the enterprise.

The ideal candidate will have strong experience with

cloud-based data platforms, Databricks, PostgreSQL

, and healthcare data, with a passion for delivering high-quality, trusted data solutions in a regulated environment.

Key Responsibilities

Data Engineering \& Platform Development

• Design, develop, and

scalable data pipeline solutions

using

Databricks (Spark)

and cloud-native services

• Build and optimize

ETL/ELT workflows

for ingesting structured and unstructured healthcare data (claims, clinical, provider, and member data)

• Develop and maintain data models in

PostgreSQL and enterprise data warehouses

• Support

Lakehouse architecture

leveraging

Databricks, Delta Lake

, and cloud storage

• Improve performance, reliability, and cost-efficiency of data platforms

Healthcare Data \& Compliance

• Work with healthcare datasets, including producer/agent, broker, commission, and distribution data, ensuring proper ingestion, normalization, and optimization for analytics and reporting
• Ensure compliance with

HIPAA, HITECH

, and enterprise data governance policies

• Implement data security, encryption, masking, and access controls
• Maintain data lineage, auditability, and regulatory reporting readiness

Advanced Data Processing

• Build

real-time and batch pipelines

for analytics and operational use cases

• Develop

data transformations using PySpark and SQL

within Databricks

• Leverage

PostgreSQL

for transactional and analytical workloads where applicable

• Integrate data from APIs, third-party vendors, and internal systems

Collaboration \& Stakeholder Engagement

• Partner with business stakeholders to support data-driven initiatives and member acquisition strategies
• Translate insurance distribution,

agent/producer

, and

marketing requirements

into scalable, high-quality data solutions

• Support downstream consumers, including Power BI, marketing analytics teams, and operational reporting stakeholders, by delivering curated, analytics-ready datasets

Technical Leadership

• Lead design and architecture discussions for enterprise data solutions
• Establish and enforce

best practices

in data engineering, testing, and CI/CD

• Contribute to enterprise data strategy and platform modernization

AI \& Advanced Analytics (Databricks Genie)

• Leverage

Databricks Genie

(AI/BI capabilities) to enable natural language querying and democratize data access for business stakeholders

• Design and optimize semantic layers and governed datasets that power Genie-driven insights with trusted, high-quality data
• Collaborate with stakeholders to translate business questions into AI-assisted analytics workflows using Databricks
• Ensure AI outputs are accurate, explainable, and compliant with healthcare data governance and HIPAA requirements
• Leverage large language models (LLMs), including

Anthropic Claude

, to enhance data exploration, automate insight generation, and support conversational analytics use cases

• Integrate

Genie capabilities with Delta Lake

and curated data models to support near real-time insights and decision-making

• Partner with data scientists and analytics teams to enhance AI-driven use cases, including producer performance insights, marketing attribution, and member engagement analysis

Required Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field
• 5–8\+ years of experience in

data engineering

• Strong programming in

Python (PySpark)

and advanced

SQL

• Hands-on experience with:
• Databricks

(core requirement)

• PostgreSQL
• Distributed data processing frameworks (Apache Spark)
• Experience with

cloud platforms

(Azure preferred; AWS acceptable)

• Proficiency in building and maintaining

ETL/ELT pipelines

• Strong understanding of

data modeling and warehousing concepts

Preferred Qualifications

• Experience in

healthcare or insurance industry

(payer experience strongly preferred)

• Familiarity with healthcare standards (e.g.,

FHIR, HL7

)

• Experience with:
• Delta Lake / Lakehouse architecture
• Orchestration tools (Airflow, Azure Data Factory)
• Streaming (Kafka, Event Hubs)
• Knowledge of

DevOps and CI/CD pipelines

(Azure DevOps, GitHub Actions)

• Experience supporting

machine learning pipelines

Key Skills \& Competencies

• Deep understanding of

data pipelines at scale

• Strong experience with

Databricks ecosystem and Spark optimization

• Expertise in

PostgreSQL performance tuning and schema design

• Strong attention to

data quality, governance, and compliance

• Excellent communication skills, especially with non-technical stakeholders
• Ability to work in a

highly regulated healthcare environment

Typical Technology Stack

• Data Platform:

Databricks, Delta Lake

• Database:

PostgreSQL, Snowflake (optional)

• Cloud:

Azure, Google, AWS

• Languages:

Python, SQL

• Orchestration:

Airflow, Azure Data Factory

• Visualization:

Power BI

• Version Control:

Git

KPIs / Success Metrics

• Reliability and performance of

Databricks pipelines

• Data quality and compliance adherence (HIPAA standards)
• Time-to-delivery for new data products
• Query performance improvements in PostgreSQL and data warehouse systems
• Stakeholder adoption and satisfaction