Add v1/orders POST endpoint to get orders in batches#4048
Conversation
Adds POST handler for `v1/orders` endpoint that requires a list of order uids and responds with a vector of their data. Has a hardcoded limit of 5000 orders per request.
There was a problem hiding this comment.
Code Review
This pull request introduces a new endpoint v1/get_orders to retrieve orders in batches using a POST request with a limit of 5000 orders per request. The code adds a new module get_orders_by_uid.rs, modifies api.rs to include the new route, and updates database/orders.rs and orderbook.rs to support fetching multiple orders. I found a missing test case for the MAX_ORDERS_LIMIT validation.
|
Tested on sepolia staging: (queries twice for the same order) response |
|
By the way, we can have a discussion on what the endpoint itself should be. I got a couple suggestions from Claude
based on the above suggestions I have opted for a custom not-order-id-like name |
There was a problem hiding this comment.
Code Review
The implementation adds a new endpoint to fetch orders in batches. A critical issue was found in crates/orderbook/src/database/orders.rs where await is used on a stream, which will cause a compilation error. The implementation also inefficiently collects two full vectors into memory; a suggestion is provided to fix the bug and improve performance by chaining the streams.
fafk
left a comment
There was a problem hiding this comment.
LGTM. The endpoint should be documented in openapi.yml too I think.
jmg-duarte
left a comment
There was a problem hiding this comment.
I didn't notice these in the last review
crates/orderbook/openapi.yml
Outdated
| - `{"error": {"orderUid": "<UID>", "description": "<message>"}}` for orders that failed conversion | ||
| Orders that do not exist in the database will be missing from the response. | ||
| content: | ||
| application/x-json: |
There was a problem hiding this comment.
Ahhh, Claude's artifact. Good catch!
MartinquaXD
left a comment
There was a problem hiding this comment.
Looks alright overall. Just some small nits.
Co-authored-by: Martin Magnus <martin.beckmann@protonmail.com>
| pub const ORDER_UID_LIMIT: usize = 1024; | ||
| /// OrderUid is 56 bytes. When hex encoded as 0x prefixes Json string it is 116. | ||
| /// Chosen to be under the MAX_JSON_BODY_PAYLOAD size of 1024 * 16 | ||
| pub const ORDER_UID_LIMIT: usize = 128; |
There was a problem hiding this comment.
This constant is also used here:
1024 -> 128 is a breaking change.
There was a problem hiding this comment.
It is not truly a breaking change since the MAX_JSON_BODY_PAYLOAD was limiting the cancellation data order uids anyway to 139 (The amount of order uids that fit with the SignedOrderCancellations). I decided to reuse the same limiting constant for both requests and rounded it up to 128.
If the 1024 (original) limit needs to be effective, We will need to increase the MAX_JSON_BODY_PAYLOAD, or do away with any order count limiting and just use MAX_JSON_BODY_PAYLOAD.
There was a problem hiding this comment.
Not sure I am following. From the main branch:
services/crates/model/src/order.rs
Line 505 in 1c28e49
It is literally the maximum number of UIDs in the requests. It was 1024, and now it is 128. Did I miss anything?
There was a problem hiding this comment.
Yeah, there is a default body limit at the api_router layer:
api_router
.layer(DefaultBodyLimit::max(MAX_JSON_BODY_PAYLOAD as usize))which limits the request body to 16kb which equals to about 139 orders in the case of order cancellation payload. Which ultimately was the breaking change limiting the effective amount of order uids to 139.
There was a problem hiding this comment.
Either we can also discuss removing the ORDER_UID_LIMIT altogether in both instances since we have the body limit in place.
There was a problem hiding this comment.
Ah, thanks! I missed that. Does it mean that with this PR it will now be limited to 128 instead of 1024?
There was a problem hiding this comment.
In this PR it will be limited to 128 instead of 139 effectively. The previous limit of 1024 was unrealistically high considering the size of hex encoded OrderUid and the body limit of 16kb.
There was a problem hiding this comment.
By the way, You can try it out for yourself as is currently:
This will fail, the body is larger than 16kb (despite the 1024 max order limit, we are sending 133)
curl -v -X DELETE \
-d "`jq -n '{"orderUids": [range(133) | "0xa8061d917531484d528329e6d6b99cc8bb7020d3436cdd8f4def4c6a8e78203604501b9b1d52e67f6862d157e00d13419d2d6e95ffffffff"], "signature": "0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", "signingScheme": "eip712" }'`" \
https://api.cow.fi/mainnet/api/v1/orders
132 orders will not fail the size check:
curl -v -X DELETE \
-d "`jq -n '{"orderUids": [range(132) | "0xa8061d917531484d528329e6d6b99cc8bb7020d3436cdd8f4def4c6a8e78203604501b9b1d52e67f6862d157e00d13419d2d6e95ffffffff"], "signature": "0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", "signingScheme": "eip712" }'`" \
https://api.cow.fi/mainnet/api/v1/orders
There was a problem hiding this comment.
I mean, the code you share will work with 128 limit instead of 1024, no?
api_router
.layer(DefaultBodyLimit::max(MAX_JSON_BODY_PAYLOAD as usize))
Before this PR, the MAX_JSON_BODY_PAYLOAD was 1024, which means this layer allowed a body with a size up to 1024. And right now this is 128.
There was a problem hiding this comment.
MAX_JSON_BODY_PAYLOAD was 16kb, and ORDER_UID_LIMIT was 1024 those are distinct. MAX_JSON_BODY_PAYLOAD effectively limits every request to every endpoint to be under 16kB, while ORDER_UID_LIMIT is a custom constant used today in cancellations, and with this PR in bulk order querying to limit amount of orders that can come in a request. The MAX_JSON_BODY_PAYLOAD or ORDER_UID_LIMIT both have a chance to fire at a specific request. If the request containing a large amount of orders exceeds the body payload limit, it will not even go into the handler.
Then in the handler the parsed orders are checked if they fit under the ORDER_UID_LIMIT.
As discussed on Slack, let's keep both limits in place, since the ORDER_UID_LIMIT tells the user exactly what is wrong and what is the amount of orders that can be submitted at once.
Co-authored-by: ilya <ilya@cow.fi>
Moves domain level error handling to domain specific code. API handler now returns Internal server error generic message. Added de-duplication logic to requested orders.
squadgazzz
left a comment
There was a problem hiding this comment.
LGTM, thanks for addressing all the comments!
Description
Aave wants to track specific orders in bulk, knowing their ids.
Changes
Adds POST handler for
v1/orders/lookupendpoint that requires a list of order uids and responds with a vector of their data. Has a hardcoded limit of 128 orders per request (to fit the MAX_JSON_BODY_PAYLOAD size).How to test
Test on staging, query multiple orders using this API.