Introduction

In Part 1, we laid the groundwork for clear resource modeling, intuitive URLs, and robust single-resource interactions in our FlexiShop API. Now, Part 2 tackles a common challenge every growing API faces: how do you serve potentially thousands of resources without overwhelming your servers or frustrating your users?

This article is your toolkit for mastering collection management. We’ll focus on production-grade techniques for:

• Accessing collections effectively
• Filtering resources with precision
• Sorting results for relevance
• Paginating responses gracefully

Get these right, and your APIs will not only scale beautifully but also become a joy for developers to use.

FlexiShop’s Growth: New User Stories, New Challenges

As FlexiShop gains traction, its users naturally demand more sophisticated ways to interact with their data:

• “As a returning customer, I want to review my past orders.
• “As a shopper, I want to find products within a certain price range and filter them by brand and customer rating.”
• “As a shopper, I want to see products sorted from the newest to the oldest.”

These stories mark a shift — from simple single-resource operations to rich, scalable collection handling. With that shift come new design challenges: How do we support smart data retrieval How do we enable precise filtering, flexible sorting, and robust pagination? These are the cornerstones of effective collection API design we’ll explore.

Accessing Collections

User story: “As a returning customer, I want to review my past orders”

Design Challenge

How should we structure our URLs to make collections easy to access and flexible to extend—especially when differentiating between general data (e.g. all orders) and user-specific data (e.g. my orders)?

Options & Trade-offs

Two main models emerge for accessing collections related to a specific resource:

1. Nested URIs (e.g., /v1/accounts/{id}/orders)
2. Flatter URIs with Query Parameters (e.g., /v1/orders?accountId={id})

First, Nested URIs directly reflect an ownership model (orders belong to an account) using the URI. This approach is quite intuitive for clear parent-child relationships. However, it can lead to deeply nested URLs that become long and brittle, and they can feel awkward if you need to filter by multiple criteria not directly related to the nesting path.

Second, Flatter URIs with Query Parameters, you use top-level collection endpoints, and specific criteria like the accountId are passed as query parameters—the ?key=value part of a URL. This pattern offers much greater flexibility for combining multiple filter criteria, scales more gracefully as your API evolves, and is often easier to document consistently. The main trade-off is that the direct hierarchical link might seem less obvious at first glance compared to nested URIs.  

“My Data” Scenarios: Implicit vs. Explicit access

Let’s consider our shopper wanting their own orders. You could use an explicit alias, such as GET /v1/accounts/me/orders. The backend would then resolve me to the authenticated user’s ID, typically extracted from a Json Web Token (JWT). A JWT is a secure token, sent by the client with API requests, that the server verifies to confirm the user’s identity.

Alternatively, and often preferably, you can use an implicit context. When a user is logged in, their JWT already contains their verified accountId. A request to GET /v1/orders (with the JWT sent in the Authorization header) allows the backend to inspect the JWT, extract the accountId, and automatically filter orders for that user. This is simpler for the client, as they don’t need to know their own ID or use special keywords, and it’s generally more secure because the backend relies entirely on the server-verified JWT.

Pragmatic Recommendation

While shallow nesting (one level, like /posts/{postId}/comments) is perfectly fine, we recommend defaulting to flatter URIs with query parameters (e.g., /v1/orders?accountId={id}) for most collection access due to their flexibility and scalability. You should try to avoid deep nesting.

For “my data” scenarios, using the JWT to implicitly scope data is usually the gold standard. It’s simpler for clients and inherently more secure. While supporting an explicit accountId=me is acceptable, the implicit approach should be your default for personal data access.

FlexiShop Example: Accessing orders

A customer views their own orders: GET /v1/orders. The server uses the accountId from the JWT to return only their orders.

curl "https://api.flexishop.io/v1/orders" \
  -H "Authorization: Bearer $USER_JWT_TOKEN"

With patterns for accessing collections established, we need to consider how users pinpoint specific items within those, often large, lists. This brings us directly to filtering.

Filtering Collections

User Story: As a shopper, I want to find products within a certain price range and filter them by brand and customer rating.

Design Challenge:

How can we enable powerful and flexible filtering especially when dealing with multiple criteria, ranges, and different data types, without making the API cumbersome to use or slow to respond?

Options & Trade-offs

Query parameters are your primary tool for filtering in REST APIs. The simplest form is an exact match, like status=active. This is fine for basic, single-value criteria but doesn’t cover ranges or more complex conditions like those in our user story.

When you need more expressive power, several patterns emerge for encoding filtering logic into query parameters:

• Dedicated Named Parameters

  One approach uses dedicated parameters that imply the condition, such as priceMin=10&priceMax=50. This is very easy for clients to understand initially and simple for backends to parse. However, the number of query parameters can explode if you need many conditions per field (e.g., priceMin, priceMax, productNameContains, productNameStartsWith), making your API surface large and potentially inconsistent over time. This pattern is best for APIs with a small, fixed set of supported conditions.

• Operator in Parameter Name

  Another pattern, sometimes seen in frameworks like Ruby on Rails or Laravel, is putting the operator in the parameter name, like price[gte]=10&price[lte]=50. The [gte] signifies “greater than or equal to.” A benefit is that some backend frameworks handle parsing this natively. However, brackets ([ and ]) must be URL-encoded (e.g., price[gte]=10 becomes price%5Bgte%5D=10), which makes URLs longer and less readable. This can also be awkward to define in API specifications like OpenAPI and may lead to less elegant client code generation.

• Operator in Parameter Value

  For greater flexibility with URL parameters, we prefer encoding the operator in the parameter value, such as price=gte:10&price=lte:50. Here, the parameter name is the field (price), and the value (gte:10) contains both the operator (gte) and its value (10), separated by a chosen delimiter (here, a colon). This keeps parameter names simple and avoids special characters that require heavy URL encoding. It’s generally easier to model in OpenAPI, leading to cleaner generated client code, and it’s extensible—adding new operators involves defining new prefixes for the value string, not entirely new query parameter names. The main trade-off is that you need robust server-side parsing logic to split the operator from the value.

Pragmatic Recommendation

For most APIs that need flexible filtering, we recommend using the Operator-in-Value strategy as the default for GET requests — e.g., price=gte:10&brand=EcoGoods.

This format avoids special characters that require heavy URL encoding, keeps parameter names simple, and is easier to work with in OpenAPI — leading to cleaner, more maintainable client code. It’s also highly extensible: adding new operators is as simple as defining new prefixes in the value string, without introducing new parameter names.

Examples:

• Range: price=gte:10,lte:50 or price=gte:10&price=lte:50
• IN condition: status=available,pending (no special handling needed)

The trade-off: server-side parsing needs to handle splitting the operator from the value reliably. But that’s typically straightforward and well worth the flexibility and clarity this pattern provides. If the filters get complex or involve sensitive data, switch to a JSON body to a /search endpoint (more on that soon).

FlexiShop Example: Flexible Filtering

A shopper wants to “find products within a certain price range and filter them by brand and customer rating.” Here’s how that translates into a request using the Operator-in-Value strategy:

GET /v1/products?price=gte:20,lte:100&brand=FlexiBrand,MegaCorp&rating=gte:4

• price=gte:20,lte:100 defines a price range from $20 to $100.
• brand=FlexiBrand,MegaCorp filters by multiple brands (logical OR within the brand filter).
• rating=gte:4 ensures only products with a rating of 4 or higher are returned.

The backend applies all conditions using AND logic. This format keeps URLs clean and expressive while supporting powerful filtering capabilities.

However, there are situations where even this approach can feel constrained, especially as query complexity or length grows. Let’s explore what lies beyond standard URL query parameters.

Beyond Standard Query Parameters

Sometimes, your filtering requirements might become so complex or involve such lengthy criteria that cramming them into a URL becomes unwieldy or even hits practical limits.

Dedicated Query Languages (e.g., RSQL, FIQL)

For ultimate flexibility, some APIs opt to embed a mini-language directly within a single query parameter. A request might look like:

GET /v1/products?filter=(price=lt=50 and brand='EcoGoods') or (category=='electronics' and rating=ge=4)

The idea here is to provide a powerful, expressive syntax, often resembling SQL’s WHERE clause, supporting full Boolean logic, grouping, and diverse operators. While this offers unparalleled flexibility, it comes with significant downsides: a steep learning curve for your API clients and a major server-side development and maintenance effort to build a secure and performant parser (often involving Abstract Syntax Trees and robust input validation). This approach is rarely justified for most APIs due to its complexity and the risk of misuse leading to performance issues. Consider it only if your domain has an undeniable need for such dynamism and your client base can handle the complexity.

Filtering in the Request Body

A more common and pragmatic approach is to move the filter payload into the request body. Typically, this involves defining a dedicated search endpoint, for example, by allowing POST requests to a path like /v1/products/search.

The request would then look something like this:

POST /v1/products/search
Content-Type: application/json

{
  "filters": {
    "category": { "eq": "electronics" },
    "or": [
      { "price": { "lt": 50 }, "brand": { "eq": "EcoGoods" } },
      { "rating": { "gte": 4 } }
    ],
    "status": { "in": ["available", "backorder"] }
  },
  "sort": [{ "field": "name", "direction": "asc" }],
  "pagination": { "limit": 50, "cursor": "someOpaqueCursor" }
}

This JSON structure is far more expressive and cleaner for complex, nested queries than trying to URL-encode everything. It also naturally accommodates other aspects of a data request, like sorting instructions, field selection (projections), and pagination parameters, all in one structured payload.

Which HTTP Method Should You Use for Body-Based Filtering?

When you decide to send filter criteria in the request body, choosing the right HTTP method is a key consideration, and it’s a topic with some debate. Here’s a breakdown of your options:

• Using POST (The Pragmatic Choice):

  Many APIs use POST for endpoints like /search when filtering becomes too complex for query strings. While POST is traditionally for creating or modifying resources, using it for read-only search operations is a well-established, pragmatic compromise.

  This pattern supports rich, structured queries in the request body — including complex filters, projections, and multi-field sorting. It’s especially common in analytics systems, internal tools, and platforms with flexible search requirements.

  The trade-off: you’re stepping slightly outside traditional REST semantics (where reads use GET). And by default, POST responses aren’t cached by most HTTP intermediaries — caching must be configured explicitly if needed.

• Using GET with a Request Body (The Risky Path):

  While the HTTP/1.1 spec (RFC 7231) doesn’t prohibit a body in GET requests, it clearly states that such a body has “no defined semantics.” The appeal is understandable: keep the read-only, cacheable nature of GET while supporting structured, complex filters.

  In practice, though, this is a risky path. Many intermediaries — CDNs, proxies, load balancers, and even some web servers — will ignore, drop, or reject GET requests with a body. That can result in inconsistent behavior, broken caching, or filters that never reach your backend.

  Bottom line: unless you fully control the request chain (end-to-end), avoid using GET with a body. It’s not reliable in real-world environments.

• The Future: The HTTP QUERY Method (The Promising Newcomer):

  To overcome the limitations of GET (no reliable request body) and POST (non-idempotent, not cached by default), the IETF HTTP Working Group has proposed a new method: QUERY.

  QUERY is designed for complex, read-only operations. It behaves like GET—safe and idempotent—but explicitly allows a request body. That makes it ideal for structured search payloads that benefit from caching and don’t modify server state.

  However, as of mid-2025, QUERY is still a draft spec. It’s not supported in most frameworks, clients, or infrastructure yet. It’s one to watch, but not something you can depend on in production today.

A Note on Caching POST Requests for Search

By default, many systems treat GET requests as cacheable, assuming they meet standard criteria. POST requests, on the other hand, are generally not cached because they’re assumed to be non-idempotent.

However, if your POST-based search endpoint is idempotent — meaning repeated calls with the same input return the same result (assuming the data hasn’t changed) — you can enable caching.

To do so:

• Return appropriate Cache-Control headers from your server
• Configure your CDN or proxy to respect those headers for selected POST endpoints

This setup can significantly improve performance for repeat queries — just be sure caching rules are carefully scoped to avoid stale or unintended results.

When to Use Request Body Filtering (with POST):

This approach (using POST to a /search or /query endpoint) is generally your best bet for rich, complex queries. It’s often seen in internal tools, analytics dashboards, advanced search UIs, or any scenario where URL length limits or the sheer complexity of filter structures hinder usability and clarity when using GET parameters. The key trade-offs are that you lose default HTTP caching (requiring explicit setup if needed) and the simple shareability of queries via a URL.

Filtering Strategies Summarized: A tiered approach

So, to wrap up our discussion on filtering, here’s a pragmatic, tiered approach you can adopt:

1. Standard (via GET): Query Parameters. This should be your default method.
   • For Exact Matches and simple conditions: Use direct field=value (e.g., status=active).
   • For more complex conditions like ranges, multiple values for IN clauses, or various operators: Use the Operator-in-Value strategy (e.g., price=gte:10&status=available,pending). This offers a great balance of power and clarity for URL-based filtering.
2. Advanced (via POST): Request Body to a /search endpoint.
   • When your filter criteria become too elaborate, deeply nested, or too long for URLs, or if you need to send sensitive filter data that shouldn’t be in URLs or server logs. Use the POST method with a JSON payload as discussed above.
3. Edge-case (via GET): Dedicated Query Language String.
   • Only consider this if you have an extreme, domain-specific need for SQL-like query power directly in a GET request, and you’re prepared for the high implementation cost and client learning curve. This is rarely needed for most APIs.

Regardless of the method, always ensure you validate and sanitize all filter inputs rigorously on the server-side to prevent errors, security vulnerabilities (like injection attacks), and overly burdensome queries.

With a comprehensive understanding of how to let clients find exactly the resources they need, the next logical step is to ensure those resources are presented in a meaningful order. This brings us to sorting.

Sorting Collections

User Story: As a shopper, I want to see products sorted from the newest to the oldest

Design Challenge

How do we allow clients to specify the sort order of results in a clear and flexible manner, while also ensuring the API remains performant?

Options & Trade-offs

Several syntaxes exist for specifying sorting criteria via query parameters. You might see separate parameters like sortBy=name&orderBy=asc, but this can be verbose and potentially ambiguous if one parameter is provided without the other. Another approach uses a prefix, like sort=+createdAt (ascending) or sort=-createdAt (descending).

A common and clear method is using a colon notation, like sort=name:asc. Here, the field name and the direction (ascending or descending) are combined into a single parameter value. If you need to sort by multiple fields, these are often comma-separated, such as sort=createdAt:desc,name:asc, indicating the order of precedence for sorting.

Pragmatic Recommendation

We recommend using colon notation for sorting, like sort=field:direction, for a few key reasons:

• Clarity: Combining field and direction into one value makes the intent explicit and avoids ambiguity (e.g., sort=createdAt:desc).
• Consistency: If you’re already using colons for filter operators (price=gte:10), this keeps the query parameter style uniform.

For multi-field sorting, allow comma-separated criteria, like ?sort=primaryField:asc,secondaryField:desc.

However, the most crucial aspect of sorting is performance. You must only permit sorting on fields that have corresponding database indexes. Sorting unindexed fields on large tables is a primary cause of API performance degradation and can cripple your database. Additionally, ensure any dates used for sorting are stored and compared in Coordinated Universal Time (UTC) for consistency across different time zones.

FlexiShop Sorting Example

Let’s say our shopper wants to see the newest products first. The API call would look like:

GET /v1/products?sort=createdAt:desc

To break ties with a secondary sort (e.g., alphabetically by name when products have the same createdAt):

GET /v1/products?sort=createdAt:desc,name:asc

The backend should ensure proper indexes exist on createdAt and name. If this sort pattern is common, a composite index like (createdAt, name) is ideal for performance.

Once filtering and sorting are in place, the next challenge is handling large result sets. If a query matches thousands of items, returning them all at once isn’t feasible — that’s where pagination comes in.

Pagination

User Story: Considering our customer who wants to “review my past orders,” if they have hundreds or thousands of orders over several years, we can’t return them all at once. This scenario naturally leads to pagination.

Design Challenge

How do we implement pagination that is efficient for the server to process, reliable for the client (especially if new data is being added frequently), and provides a good user experience for navigating large datasets?

Options & Trade-offs

When a query could return many resources, you need to paginate the results. Two primary strategies are common:

1. Offset/Limit Pagination (also known as Page-Based Pagination):

   This is often the first approach developers think of. Clients specify a limit (how many items per page) and an offset (how many items to skip from the beginning). For example, GET /v1/orders?limit=25&offset=50 would aim to fetch items 51 through 75.

   The main advantage is its simplicity to understand and implement, and it allows clients to easily calculate and jump to specific page numbers (e.g., “go to page 5” can be calculated as offset = (5-1) * limit).

   However, it has significant drawbacks. Performance can degrade severely with large offsets (e.g., offset=100000) because the database still has to effectively count through all the skipped rows before finding the desired page. More critically, if items are added to or removed from the dataset while a user is paginating, items can be missed entirely or appear on multiple pages. This occurs because the “window” of items defined by the offset and limit shifts relative to the underlying data.

2. Cursor-Based Pagination (also known as Keyset Pagination):

   This method is more robust for large or frequently changing datasets. Instead of an offset, the client passes a cursor, which is an opaque string (meaning the client doesn’t need to know its internal structure) that acts as a pointer to the last item seen on the previous page. The server uses the information encoded in this cursor to fetch the next set of items that come after that specific item, based on the current sort order. For example, GET /v1/orders?limit=25&cursor=opaqueCursorString123.

   Cursor-based pagination offers excellent performance, even with massive datasets, because the database can typically use its indexes efficiently to “jump” directly to the start of the next page. It does this by using the decoded cursor values in a WHERE clause (e.g., WHERE (createdAt, id) < ('timestamp_of_last_item_from_cursor', 'id_of_last_item_from_cursor') assuming a descending sort by createdAt then id). This makes it much more resilient to data being added or removed concurrently, as it anchors to a specific item rather than a numeric offset.

   The trade-offs are increased implementation complexity on the server (to generate and parse these opaque cursors) and that it generally doesn’t allow arbitrary page jumps; users can typically only navigate to the “next” (and sometimes “previous”) page. Getting a total count of all items can also be an expensive separate query, so it’s often omitted or handled differently in cursor-based systems.

Implementation Note

Cursor-based pagination requires a stable, unique sort order, such as: sort=createdAt:desc,id:desc This prevents records from being duplicated or skipped.

Pragmatic Recommendation

While offset/limit is fine for small datasets or when users need to jump to exact pages, we recommend cursor-based pagination for most APIs — especially those working with growing or high-volume data.

Here’s why:

• It scales reliably — even with millions of records
• It plays nicely with dynamic datasets where data changes between requests
• It keeps DB load manageable by leveraging indexes instead of skipping rows

Your API response should make pagination intuitive:

• Offset-based: Return offset, limit, and optionally totalItems
• Cursor-based: Return the current items, a nextCursor (if more results exist), and optionally a hasMore flag for UI logic

Cursor pagination may take a little more effort upfront, but it pays off in performance and consistency as your data grows.

FlexiShop Example: Cursor-Based Pagination in Action

Let’s walk through how a customer views their order history — newest orders first, 20 per page — using cursor-based pagination.

First Page Request:

GET /v1/orders?limit=20&sort=createdAt:desc,id:desc

Response (Page 1 might look like this):

{
  "items": [
    // 20 order objects, newest first
  ],
  "pagination": {
    "limit": 20,
    "nextCursor": "eyJjcmVhdGV=",
    "hasMore": true
  }
}

• nextCursor is an opaque string representing the position after the last item on this page.
• hasMore tells the client whether another page is available.

Next Page Request:

GET /v1/orders?limit=20&sort=createdAt:desc,id:desc&cursor=eyJjcmVhdGV=

The server uses the cursor to fetch the next 20 items that come after the last item on page 1 — sorted by createdAt, then id.

Now that we’ve covered the core mechanics for handling collections, let’s see how these elements combine to form powerful API endpoints and discuss some overarching performance considerations.

Putting it all together

We’ve covered how to access collections, refine them with filters, order them with sorting, and split them into pages with pagination. Together, these patterns form a foundation for scalable, user-focused API design.

Here’s how FlexiShop applies these principles in real endpoints:

Performance Tips:

• Database Indexing matters All filter and sort fields should be backed by indexes. Use composite indexes when filter/sort combinations are predictable. Cursor-based pagination depends on sort-aligned indexes.
• Optimize query generation. Translate API params into efficient DB queries. Sanitize input to avoid misuse or slowdowns.
• Minimize data transfer by only selecting the fields truly needed from the database, if possible. While we haven’t dived deep into field selection (projections) in this part, it’s a related concept for optimizing responses.
• Continuously monitor and optimize your API’s query performance. Identify slow queries and take steps to improve them, often through better indexing or query refactoring.

Quick Reference (Cheat-sheet)

Here’s a quick summary of our practical recommendations for designing collection-based API endpoints:

Conclusion

Designing how your API handles collections is foundational to building systems that scale. When you combine clean access patterns, expressive filtering, reliable sorting (on indexed fields!), and efficient pagination (favoring cursors), you’re not just checking functional boxes — you’re crafting APIs that are robust, performant, and easy to integrate with.

These patterns lead to faster APIs, smoother client experiences, and fewer surprises in production. They also future-proof your system as traffic and data grow.

Coming Up Next: Part 3 — Robustness and Evolution

In Part 3, we’ll shift focus to the practices that take your API from good to production-grade:

• Versioning strategies that let you evolve without breaking clients
• Rate limiting to protect your service and ensure fairness
• Idempotency for safe, retryable operations
• Async workflows for long-running jobs with trackable progress

See also

• Pragmatic API Design: Balancing Best Practices and Real-World Needs (Part 1)