One API call a day. A pipeline the whole team could replay.

The API was supposed to feed a data product about LNG tanker movements. One test request revealed the real architecture constraint: our contract permitted one call in a 24-hour window.

The product needed roughly 700 tanker-movement records per day. A response covering the latest two days contained around 1,000 JSON objects. We needed to retain those raw observations, process them into structured records, and eventually publish aggregated LNG import and export information for analysts and customers.

Five developers and data scientists also needed the same input while building and validating different parts of the system. If each task depended directly on the vendor endpoint, one person’s experiment could consume the only request available to everyone for the next 24 hours.

The rate limit was not an API detail

My first call to the test endpoint returned a rate-limit response. I waited a few minutes and tried again, with the same result. Instead of treating it as a temporary test-environment issue, I emailed the vendor and copied our project manager and team. I asked whether the limit applied only to testing or to the production contract too.

The answer changed the design: both environments allowed one call per day. I raised the delivery and verification risk with the product owner, but increasing the limit was not commercially viable because of the API’s contract and cost. We had to design around the limit rather than wait for it to change.

A contractual limit belongs in the architecture, not in a footnote beside the client code.

Test the boundary before building around an assumption

The most valuable part of the first test was not the payload. It was discovering a boundary while the design was still cheap to change. Had I assumed that the test key was merely restricted, we could have coupled multiple development workflows to an API that production would never let them call independently.

The communication path mattered too. Bringing the vendor, project manager, and team into the same thread converted an ambiguous error into a confirmed constraint. It also made the cost trade-off explicit: a higher limit might have simplified the software, but it was not an option the product could purchase.

Why I chose a raw-data landing layer

I considered whether we could solve the problem at the source. The product owner explored a higher call allowance, but the vendor contract made that impractical. The remaining choice was where to materialize the one response we were permitted to receive.

Writing directly to a database would have coupled collection to assumptions that were still changing. The vendor JSON was neither clean nor shaped for the product’s final queries. Data scientists still needed to research it, and future non-developer users might need the original files for investigations we could not yet predict.

I therefore placed Google Cloud Storage between the vendor and every downstream consumer. The bucket became an append-oriented landing layer: inexpensive to retain, easy to inspect, and independent of the schemas used later for metadata, geospatial queries, and time-series analysis. A file could be replayed without spending another API call or asking the vendor to reproduce a historical response.

The daily pipeline stayed deliberately small

I wrote the Python collector and orchestrated it as an Airflow DAG. At 06:00 CET, Airflow invoked the endpoint for the latest two days of movement events. A normal run took approximately two to three minutes. The collector grouped returned objects by the datetime carried in each record and wrote raw JSON files to Google Cloud Storage.

Each filename began with its run date and a Unix timestamp. The naming made distinct ingestions visible even when they contained overlapping source dates. Downstream processing used the vendor’s unique event ID to recognize the same movement across snapshots rather than treating every appearance as a new fact.

PostgreSQL held pipeline metadata, while processed geospatial records lived in a PostGIS-enabled PostgreSQL database and movement series flowed to Cassandra. My responsibility ended at implementing and scheduling the raw collection path; teammates in data science owned research and processing of the collected data.

1. 01
   Acquire once

   Spend the constrained vendor request on a scheduled collection, not an individual developer’s transient task.

2. 02
   Preserve first

   Store the source response before cleaning it so downstream assumptions cannot erase the original evidence.

3. 03
   Interpret later

   Let geospatial, time-series, analytical, and verification workloads read the same snapshot on their own schedules.

The two-day window turned duplication into recovery capacity

Every run requested two days even though the product published daily data. That overlap was intentional. If one day’s collection was incomplete or its downstream processing failed, the following successful response could carry those events again. Because each event had a stable unique ID, downstream code could deduplicate the overlap instead of losing the repeated records or counting them twice.

This pattern separated two ideas that are easy to conflate: duplicate transport and duplicate facts. Receiving event E-1042 in two files is harmless when the processing boundary recognizes that both records describe the same movement. The overlap then becomes a small recovery buffer rather than a data-quality problem.

Alerts made failure visible; they did not make it disappear

The Airflow task had a retry policy of two. Failures and customer-SLA signals were connected to the team’s Slack channel so the team could respond without discovering a missing day later through an analyst’s query. A second API attempt inside the 24-hour window could itself receive HTTP 429, so a red DAG was treated as an operational signal, not as evidence that retries would always self-heal the run.

The two-day query bounded the durable recovery window. One missed collection could be covered by the next successful run. If collection failed for two consecutive days, the next ordinary response could no longer guarantee that every missing event remained available. At the project’s data volume—about 700 ship movements per day—the agreed fallback was to request the missing data manually from the vendor.

What changed—and what the evidence can support

The bucket became the project’s raw source of truth and historical working set. Developers and data scientists could inspect, replay, and reprocess captured files without competing for the vendor call. The same materialized response supported schema development, data-quality verification, research, backfills, and future investigations.

I estimate that this avoided a potential wait of up to 24 hours for each of the five people who needed the source data, because that was the contractual call window. It is not a measured productivity benchmark: not every developer would necessarily have made a conflicting request every day. The defensible outcome is operational—the team no longer needed another API call to reuse data we had already acquired.

The solution also shortened the distance between a collection failure and human awareness. Slack alerts did not guarantee recovery, but they gave the team time to act while the two-day source window could still repair a one-day gap.

What I would reuse—and where I would strengthen it

The reusable pattern is materialize once, consume many times. It is useful when an external source is expensive, slow, rate-limited, difficult to reproduce, or shared by several downstream workloads. Preserve the raw response at the boundary, give records stable identity, and let processing evolve on the other side.

I would not apply it blindly. Sensitive payloads need access controls, retention rules, and auditability. Large responses need a storage and lifecycle cost model. Sources without stable event identifiers need a more careful idempotency strategy. And if the acceptable recovery point is tighter than the source’s lookback window, a once-daily pull is the wrong acquisition contract regardless of how cleanly it is orchestrated.

If I extended this system, I would make the call window explicit in the collector’s state, distinguish acquisition failure from storage failure, and track the latest successfully covered source timestamp rather than relying only on DAG status. Those changes would make manual runs safer and turn the recovery boundary into something the system could measure directly.

Reliability began when the only permitted response stopped being disposable.

The important decision was not Airflow versus another scheduler or one database versus another. It was recognizing that a scarce external read should not remain a shared runtime dependency. By landing it once, preserving it intact, and designing a visible recovery limit around it, I turned a blocking contract constraint into a small and dependable data boundary.