Data streaming with action cards
You can use one or more action cards in a flow that perform searches for records. You can specify the search parameters in a card such that only one record is returned, a small number of records such 100 is returned, or run the flow such that a much larger set of records is returned. If you want to return a large number of records, you will often encounter a maximum limit on the number of records that can be returned in search result. Such a limit may be 10,000 records or less.
For larger sets of records, data streaming is a feature offering in Workflows connectors where a large data set is returned and processed not within a parent flow, but within a helper flow that’s in place to return a high number of records. This capability gives flow builders the ability to be scalable when handling large amounts of data, and it takes advantage of pagination-enabled API endpoints.
While lower limits help ensure proper system performance, additional custom user properties or other user data in the result set may cause a flow to exceed system limitations unexpectedly and cause an out-of-memory error to occur. To avoid these system limitations or possible memory errors, the Stream Matching Records option is available on many search and list cards.
You are most likely to take advantage of data streaming in Workflows when using an action card to search for or list records, such as the Search Tickets action card for the Freshservice connector or the List Users with Filter action card for the Okta connector. In those and other cards that offer data streaming, you must specify a helper flow that will be used to process the data. The Choose Flow dialog allows you to choose a processing flow as an input to the card by which your requested records are searched and returned.
With this option on an action card, a much larger set of records can be read, with these expected performance benchmarks:
500,000 records can be read in approximately 13 hours.
1 million records can be read in about 25 hours.
A maximum limit of 1 million records can be returned.
Limits and considerations
When using data streaming, there are some considerations to keep in mind. Once a flow has been executed, it cannot be stopped. Therefore, a helper flow that is in place to stream data from a search or list card will also continue to run until all records are returned from the API, or the specified maximum number of records has been reached, whichever is lower. Additionally, a flow will not stopping running even if it is subsequently set to be inactive.
The processing of data records into a helper flow will be halted if a page limit is set and reached for that connector. You can specify the maximum number of records that can be requested from a third-party API and returned by an action card. Native limits on a connector are also in place to prevent lengthy data returns.