[kiwicopple]

Yes, the data is queried live.

> unexpected N API calls when joining across my domain

I'm not sure why they would be unexpected (because it should displace some other API calls). I'll hazard a guess that you're worried about fetching the same data multiple times? If that's the case, then yes, you should materialize the data into your database.

The Wrapper itself handles pagination, but you'd also want to make sure you're using filters & limits in your SQL statements.

[abuehrle]

Thanks for the reply. I'll flesh out my thought process in case it's helpful. My immediate reaction was excitement about the abstraction. An example use case is joining my users to their corresponding Stripe Customers in SQL. The kinds of queries I can reasonably write depend on implementation details of the connector. For example, if Stripe has a bulk customer lookup (list of customer IDs -> Customers), and the connector uses it, I can estimate I'd be able to query on the order of 500 Users at a time in a performant way. But if the API only supports looking up one customer at a time, that 500 User query kicks off 500 API requests, which isn't going to work.

You're right -- it's not unexpected -- maybe more like a leaky abstraction.

[kiwicopple]

I understand now, and this is a similar problem to how some GraphQL engines work

I imagine you want to do something like:

    select 
        *
    from 
        public.users join stripe.customers
    on
        public.users.stripe_id = stripe.customers.id
    limit 100;

Then yes, it might make 100 consecutive calls to your stripe account. There are 3 options here:

1. Materialize your customers into your database (like I mention in the previous comment)

2. We build a "smart" FDW, so that it parses your query and fetches the Stripe data first, then performs the join.

3. Use a CTE:

    with
    customers as (
        select * from stripe.customers
    ),
    users as (
        select * from public.users
    )
    select 
        *
    from 
        users join customers
    on
        users.stripe_id = customers.id

[infogulch]

4. Use a local http cache in front of stripe's api. This is basically "external materialization".