Introducing: Async query result streaming

We’re always looking for ways to optimise our APIs, and the latest improvement is a new option for retrieving async query results. The old serialised JSON endpoint isn’t going anywhere, but integrators now have the choice of streaming result data instead, which is particularly useful for integrators handling documents. Read on for what that means for you and how to switch.

What’s the difference?

Serialised JSON

The old endpoint for fetching async query results serves the result directly in the response body. Returning query data serialised within the response body is convenient as it only requires integrators to deserialise the JSON and convert from base64. However, it requires loading entire datasets into memory, which is not the most efficient way of returning large amounts of data.

Streaming

In comparison, streaming result data requires a bit more work to set up but offers several advantages over returning serialized content, including:

• Memory Efficiency: Avoids allocating large in-memory objects, improving stability and performance under load.
• Reduced Latency: Enables immediate data transfer, lowering time-to-first-byte and improving perceived responsiveness.
• Scalability and Reliability: Performs better in constrained environments (e.g., serverless environments) and reduces the risk of failures due to size or resource limits.
• Client Flexibility: Allows integrators to process data incrementally, supporting large datasets without imposing heavy memory or parsing costs.

Most of the technical benefits more directly affect Halo Cloud (apart from the last one), however the flow-on benefits for integrators aren’t too shabby.

Why should I care?

The benefit of using the new endpoint is speed and reliability.

• Less overhead on Halo Cloud means faster transfer
• Less risk of Halo Cloud failing to process data means less errors and failures
• Data is transferred in a more robust manner

These benefits apply to anyone using async queries, but particularly for any integrators fetching large amounts of data, as the benefits multiply with increasing data size.

The most relevant use case is large documents. Halo tries to page async query results to 1MB max, but we also do not break up rows of database data. So if a single row of the database is >1MB, it will be returned as a single page, no matter how big it is. This is most likely to happen when dealing with documents, since they usually are a single row of data and can be several MB in size. Retrieving such large pages as serialised JSON can hit issues with memory limits or even tooling restrictions, e.g. if a tool or library has a 5MB limit.

It is recommended that integrators handling large documents switch to the new streaming endpoint.

How do I switch?

The API reference has been updated with details of the new Stream query results endpoint.

At a high level though:

• The format of the result data doesn’t change. It is still paged to 1MB and the data is still base64 encoded.
• The main difference in the data returned by the new endpoint is some extra headers.

As an example, the old endpoint returns something like this:

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 45

{
  "parameters": [],
  "data": "W3siSUQ...."
}

While the new endpoint will return this:

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Size: 45
Rows-Count: 10
Rows-Range-Start: 0
Rows-Range-End: 10

{
  "parameters": [],
  "data": "W3siSUQ...."
}

To use the new endpoint, integrators will need the ability to process streamed results, though most programming languages should have support for this.

As an example of how to use the new endpoint in C#:

// Make request to streaming endpoint
using var response = await _httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);        
response.EnsureSuccessStatusCode();

// Read custom headers
var size = response.Headers.GetValues("Size").FirstOrDefault();
var rowsCount = response.Headers.GetValues("Rows-Count").FirstOrDefault();
var rangeStart = response.Headers.GetValues("Rows-Range-Start").FirstOrDefault();
var rangeEnd = response.Headers.GetValues("Rows-Range-End").FirstOrDefault();

// Stream the JSON content directly
using var contentStream = await response.Content.ReadAsStreamAsync();

// Upload stream directly to blob - no intermediate buffering
await blobClient.UploadAsync(contentStream, blobUploadOptions, cancellationToken: default);