Data Engineer VA

Responsibilities

• Design, build and maintain batch and streaming pipelines using tools such as Airflow, dbt, Fivetran, Kafka or Spark.
• Model data in a cloud warehouse such as Snowflake, BigQuery, Redshift or Databricks with clear layers for raw, cleaned and business-ready data.
• Write efficient SQL and Python for transformations, and version everything in Git with code review.
• Ingest data from APIs, databases, SaaS tools and files, handling schema changes and late-arriving data gracefully.
• Implement data quality tests, monitoring and alerting so broken pipelines are caught before stakeholders notice.
• Optimise warehouse performance and cost through partitioning, clustering, incremental models and query tuning.
• Manage infrastructure with Terraform or similar and deploy pipelines through CI/CD.
• Apply data governance including access controls, PII handling, retention and documentation in a data catalogue.
• Work with analysts and data scientists to understand what they need and expose well-documented datasets.
• Document pipeline architecture, data lineage and runbooks for on-call support.