Tony CantaveHow I Call REST APIs From Python Without Getting Burned

When I started writing Python scripts that talked to other services, my first versions worked fine on my desk and fell apart the first time a server was slow. A script would hang for minutes, or crash halfway through a batch and leave me guessing which records actually went through. Over time I settled on a small set of habits that make API calls boring in the best way. Here is what I do now, with code you can copy into your own projects.

Always set a timeout

The requests library will wait forever by default. That surprises a lot of people. If the other side accepts the connection and then goes quiet, your script just sits there. I pass a timeout on every call, as a pair of numbers: how long to wait to connect, and how long to wait for data.

response = session.get(url, timeout=(5, 30))

Five seconds to connect and thirty seconds to read is a sane starting point for most APIs. Tune it once you know how the service behaves.

Retry the right failures, not all of them

Some failures are temporary: a 429 rate limit, a 502 or 503 from a busy gateway, a dropped connection. Others are permanent: a 400 means my request is wrong, a 401 means my token is bad, a 404 means the thing does not exist. Retrying a permanent failure just wastes time and can get you rate limited. I let urllib3 handle retries with a growing delay between attempts.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

def make_session():
    retry = Retry(
        total=4,
        backoff_factor=1,
        status_forcelist=[429, 500, 502, 503, 504],
        allowed_methods=["GET", "PUT", "DELETE"],
        respect_retry_after_header=True,
    )
    session = requests.Session()
    session.mount("https://", HTTPAdapter(max_retries=retry))
    return session

Notice that POST is not in the allowed list. A POST usually creates something, and retrying it blindly can create it twice. If an API supports an idempotency key, use it before you retry a POST.

Check the status before trusting the body

A response coming back does not mean the call worked. I call raise_for_status so an error code becomes an exception I can handle, and I catch the cases I actually expect.

import logging
log = logging.getLogger("api")

def get_json(session, url, params=None):
    try:
        response = session.get(url, params=params, timeout=(5, 30))
        response.raise_for_status()
        return response.json()
    except requests.exceptions.HTTPError as err:
        log.error("HTTP %s for %s", err.response.status_code, url)
    except requests.exceptions.Timeout:
        log.error("Timed out calling %s", url)
    except ValueError:
        log.error("Response from %s was not valid JSON", url)
    return None

Returning None on failure keeps the caller simple. The caller checks one thing and decides whether to skip the record or stop the run.

Paginate on purpose

Most APIs return results in pages. The bug I see most often is a loop that never ends because the stop condition is wrong. I follow the cursor the API gives me and put a hard cap on pages so a surprise can never turn into an endless loop.

def fetch_all(session, url, max_pages=50):
    results = []
    params = {}
    for _ in range(max_pages):
        page = get_json(session, url, params=params)
        if page is None:
            break
        results.extend(page.get("items", []))
        cursor = page.get("next_cursor")
        if not cursor:
            break
        params["cursor"] = cursor
    return results

Field names like items and next_cursor change from API to API, so check the documentation and adjust.

Keep secrets out of the code

Tokens do not belong in a script, and they definitely do not belong in a Git repository. I read them from environment variables and attach them to the session once.

import os
token = os.environ["API_TOKEN"]
session.headers.update({"Authorization": f"Bearer {token}"})

If the variable is missing, the script fails right away with a clear KeyError instead of sending requests with no credentials.

Log enough to answer what happened

When a scheduled job runs at night, the log is the only witness. I log the start and end of each run, every failed call with its status code, and a count of records processed. A single line of setup gets you timestamps:

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")

The next morning I can read the log and know in a minute whether the run was clean.

Why I care about this

I spent more than fifteen years in logistics and manufacturing before moving into software. In route operations, a plan that only works when traffic is perfect is not a plan. Scripts are the same. The network will be slow sometimes, a service will go down, a token will expire. Building for that from the start is cheaper than cleaning up after it.

These habits show up everywhere in my work now, from the Python samples I write and debug for AI model training to the software I am building to manage the screens for Local Flyer Ads. None of it is fancy. It is just the difference between a script that works once and a script you can trust.

If you have a pattern that has saved you from a bad night, I would like to hear it. You can reach me at sales@localflyerads.com.