[gurchik]

I like using generators when querying APIs that paginate results. It's an easy way to abstract away the pagination for your caller.

  def get_api_results(query):
    params = { "next_token": None }
    while True:
      response = requests.get(URL, params=params)
      json = response.json()
      yield from json["results"]
      if json["next_token"] is None:
        return
      params["next_token"] = json["next_token"]
  
  for result in get_api_results(QUERY):
    process_result(result)  # No need to worry about pagination