Paginate: API Endpoint
This function card is only available in Connector Builder.
The specifics around paginating against an API endpoint depend heavily on the API in question. For this example, we’ll imagine a simple link header that includes a URL indicating the next page of records. The following builds on the concept and implementation introduced in Paginate: Set a Max Page Index.
To begin, add some new values to the pagination object:
-
url: Complete URL used to retrieve the first page of records. In many services, this will contain both a limit and an offset value.
-
limit: Determines how many records to retrieve starting from the offset.
-
offset: Position in the dataset of the last record retrieved.
-
-
results_list: This is an empty list of objects to which each page of new records will be added.
-
Connection: Pass the connection context along to the helper flow.
This is the resulting object:
{
"index":0,
"max_page_indext":10,
"break":false,
"url":”https://app.scaleft.com/v1/teams/demo/projects?limit=200&offset=0”,
"results_list":[],
"Connection":{
"blob":{
"api_key":"SlESyU45eGxqM8dDi3ucmodj3sJ1AGGQ"
}
}
}The inputs defined on the event card will need to match the keys defined in the object above: page_index, max_page_count, break, url, results_list, and Connection. The platform will extract the values from the pagination object and make them accessible in the flow.
Use the url value to make a call to the service using the HTTP Raw Request function. Once a response is received, the helper flow will manage the url and results_list to prepare for the next iteration of the pagination function. Extract the link header from the headers object and the page of results from the body object using Object Get cards.
Once a page of results has been extracted from the response, it needs to be added to any existing results from previous pagination iterations using the List Union function. Once that’s complete, it’s time to rebuild the pagination object with the new values collected within the flow.
This is the resulting object:
{
"index":1,
"max_page_index":10,
"break":false,
"url":”https://app.scaleft.com/v1/teams/demo/projects?limit=200&offset=200”,
"results_list":[
“1”,
“2”,
“3”,
“4”,
“5”,
"..."
],
"Connection":{
"blob":{
"api_key":"SlESyU45eGxqM8dDi3ucmodj3sJ1AGGQ"
}
}
}Finally, you’ll need to determine how the API indicates that there are no more pages of results and manage that much the same way you do with max_page_index. For our example, we’ll assume that once we’ve received the last page of results that the service drops the link key from the header. If the link key is missing or empty, then the break key/value pair should be removed from the pagination object using the Object Unset function. When the pagination object is returned to the parent flow without the break key, the pagination will stop.
This is the resulting object:
{
"index":6,
"max_page_index":10,
"url": "",
"results_list":[
“1”,
“2”,
“3”,
“4”,
“5”,
“6”,
“7”,
“8”,
“9”,
“10”,
"..."
],
"Connection":{
"blob":{
"api_key":"SlESyU45eGxqM8dDi3ucmodj3sJ1AGGQ"
}
}
}