A shortlist in 72 hours, contributing inside two weeks, and a 30-day guarantee if the fit is wrong.
72-hour shortlist · 30-day replacement guarantee · Month-to-month
Building a pipeline that moves data from A to B is not difficult. Building one that runs every night for two years without anyone thinking about it is a different discipline entirely.
The failures are predictable and expensive. A schema changes upstream and the job fails silently for a week. A late-arriving file means yesterday’s numbers were wrong and nobody noticed. A backfill runs twice and duplicates six months of records. Someone changes a transformation and three dashboards quietly disagree.
The engineers worth hiring are the ones who design for these before they happen — idempotency, data quality checks, alerting on absence rather than just on error, and lineage so you can answer “where did this number come from” without a two-day investigation.
Idempotency, incremental loading, and handling late or out-of-order data. The difference between a script and a system.
Window functions, CTEs, query plans. Data engineering is a SQL job first and a Python job second.
Airflow, Dagster, or Prefect. Dependency management, retries, and backfill strategy.
Dimensional modelling, slowly changing dimensions, and knowing when to denormalise. Snowflake, BigQuery, Redshift.
Increasingly the standard layer. Testing, documentation, and lineage.
Great Expectations or equivalent. An engineer who tests data as well as code is worth a premium.
Data failures are unusual in that the expensive ones don’t look like failures. The job succeeds. The number is wrong.
A concrete case. An upstream team renames a column and keeps the old one populated with nulls for compatibility. Your pipeline reads it, finds nulls, and loads them without complaint. Revenue by region now under-reports one market by 40%. The dashboard is green, the job is green, and the first person to notice is a regional manager three weeks later who thinks their team is underperforming. By then decisions have been made on the number.
The duplicate-backfill version is worse. A retry re-runs a non-idempotent load, and six months of history is doubled. Reconciling that means knowing precisely which rows came from which run — which you can only do if someone designed for it beforehand.
The real cost isn’t the engineering time to fix it. It’s that people stop trusting the numbers. Once a finance team has been burned, they rebuild their own spreadsheet, and your warehouse becomes an expensive system nobody uses for decisions.
The reputational cost compounds faster than the technical one. Once a leadership team has been shown two conflicting numbers for the same metric, they discount the next report by default — and rebuilding that trust takes considerably longer than fixing the pipeline that broke it.
Mid-level (3–5 years). Builds and maintains pipelines against an existing platform. Comfortable with SQL, Python, and an orchestrator. Needs direction on modelling.
Senior (5–8 years). Designs the warehouse model and pipeline architecture. Makes storage, orchestration, and cost decisions. Sets quality and testing standards.
Lead (8+ years). Owns data platform direction, handles governance, makes build-versus-buy calls on the stack.
Warehouse build or migration. Moving from ad-hoc reporting to a modelled warehouse, or between platforms.
Pipeline development. Ingesting from APIs, databases, and files into a central store on a schedule.
dbt implementation. Bringing transformation logic out of scattered scripts into a tested, documented layer. When that layer needs a permanent owner, the role you want is an analytics engineer.
Reliability rescue. Existing pipelines that fail often and nobody trusts. Usually a focused engagement.
Analytics enablement. Getting from spreadsheets updated by hand to dashboards people rely on, usually with BI developers building the reporting layer above. See data engineering and BI.
Warehouse cost control. Usually arrives as a surprise line on an invoice and resolves into a handful of unpartitioned tables plus a dashboard refreshing every fifteen minutes that nobody opens.
Week one. They trace one number end to end — from source system to dashboard — and document every hop. It sounds slow. It is the fastest way to find out what’s actually broken, and it usually surfaces at least one transformation nobody can explain. Expect questions about orchestration, where transformation logic lives, and whether anything currently tests the data rather than the code.
Weeks two to four. They add tests to an existing pipeline before building a new one — row counts, uniqueness on a key, not-null on the columns that matter, and freshness. Then they ship a pipeline that’s idempotent and can be safely re-run. Expect them to name the job most likely to fail silently and instrument it first.
The warning sign is a first month of new dashboards. Visible output early is tempting for everyone involved, but building more on an untested foundation multiplies the surface area of the problem you hired them to fix.
Ask which number they checked first. A data engineer who arrives and reconciles one important figure against its source — revenue for last month, active users last week — learns more about the state of your platform than any architecture review would tell them. It also surfaces the disagreements between teams that everyone has been quietly working around.
Alerting on failure but not on absence. Most teams get paged when a job errors. Almost none get paged when a job simply doesn’t run — a scheduler outage, a disabled DAG, a cron that silently stopped. The second failure is far more common and far more damaging, because yesterday’s data just quietly isn’t there. Ask candidates how they’d detect it.
Hiring a backend developer to build the warehouse. They will write competent Python and model the data like an application database — normalised, transactional, and painful to query analytically. Dimensional modelling is a distinct skill. The mismatch shows up as reports that take minutes to run.
Buying Snowflake or BigQuery before the modelling is settled. The platform is rarely the constraint at small and mid scale; Postgres handles more than people expect. Teams migrate to a warehouse, keep the same unmodelled tables, and discover the queries are still slow and now also metered. Fix the model first.
Measuring the pipeline instead of the data. A green run means the job finished, not that the numbers are right. Row counts that quietly halve, a source that starts sending nulls in a column nobody checks, a timezone change that shifts a day boundary — all of these pass every scheduler check you have. Tests belong on the data, not only on the execution.
Rates depend on seniority, warehouse platform, and engagement length. Tell us the role and we’ll give you a firm number.
Tell us the role and we will send a shortlist within 72 hours.
Tell Us the RoleTell us the role. Three to five profiles within 72 hours.