Should we remove redundant endpoints from Catalyst? ll
Proposal Details
Description
### Introduction The Lambdas service from Catalyst was originally designed to support various independent operations required by the reference implementation client. However, as time progressed, newer and more effective endpoints with identical functions were introduced, yet the service retained all previous iterations. This is the second of a serie of proposals that aim to enhance the resilience and efficiency of Catalyst's Lambdas service by optimizing its endpoints and removing old implementations that didn't take performance into consideration, making inefficient use of resources. You may find the first proposal of this serie [here](https://governance.decentraland.org/proposal/?id=24f524f0-eb50-11ed-ac2d-876c6fc9416f). The main objective of this proposal is to decrease the overhead of the Lambdas service and improve Catalyst response times by removing the redundant endpoints GET /lambdas/contentv2/contents/:cid, GET /lambdas/contentv2/scenes, and GET /lambdas/contentv2/parcel_info. ### Proposal **GET /lambdas/contentv2/contents/:cid** This endpoint is currently acting as a proxy, forwarding the initial request to the _GET /contents/:cid_ endpoint of the Content service and returning its response without any additional formatting. Going forward, clients should access the [GET /content/contents/:cid endpoint](https://decentraland.github.io/catalyst-api-specs/#tag/Content-Server/operation/getContentFile) from content-server directly. **GET /lambdas/contentv2/scenes** This endpoint is currently acting as a proxy, forwarding the initial request to the _POST /content/entities/active_ endpoint of the content-server and returning its response without any additional formatting. Going forward, clients should access the [POST /content/entities/active endpoint](https://decentraland.github.io/catalyst-api-specs/#tag/Content-Server/operation/getListOfEntities) from content-server directly. **GET /lambdas/contentv2/parcel_info** This endpoint is also acting as a proxy, forwarding the initial request to the _POST /content/entities/active_ endpoint of the content-server and returning its response without any additional formatting. Going forward, clients should access the [POST /content/entities/active endpoint](https://decentraland.github.io/catalyst-api-specs/#tag/Content-Server/operation/getListOfEntities) from content-server directly. Provided that this proposal gets general acceptance level, the specified endpoints will be removed one month subsequent to its approval.
Hey @dax, No changes were introduced on the **/content/entities/scene?** endpoint on the scope of this serie.
Hello @MorrisMustang, Currently, **GET /lambdas/contentv2/scenes** is acting as a proxy of **POST /content/active/entities**. This means, that you will be able to get the same data by consuming **POST /content/active/entities** from now on.
i'm using `/content/entities/scene?pointer=x,y` is that still good to use?
Should we remove redundant endpoints from Catalyst? ll This proposal has been PASSED by a DAO Committee Member (0xbef99f5f55cf7cdb3a70998c57061b7e1386a9b0)
Does changing the endpoint GET /lambdas/contentv2/scenes affect our ability to query historical scene data?
Should we remove redundant endpoints from Catalyst? ll This proposal is now in status: FINISHED. Voting Results: * Delete the endpoints 100% 2,174,308 VP (57 votes) * Keep the endpoints 0% 0 VP (0 votes) * Invalid question/options 0% 0 VP (0 votes)
[quote="ph33bs, post:7, topic:19209"] For example, the content served by the **GET endpoints /lambdas/contentv2/scenes** is highly mutable, and the content pointed to by the coordinates usually changes. In such cases, caching these request would be ineffective. Instead, *POST /active/entities* can be used to request the same amount of information at once by taking a list of pointers, and the client could calculate the pointers in the square, which, if it is a much-needed functionality it can be implemented on the Catalyst client. [/quote] I'd love to delve into this a bit further with you. When we talk about the content served by the **GET endpoints /lambdas/contentv2/scenes**, you mentioned that it's "highly mutable." This basically means that the content can change quite frequently. But to break it down, there are two types of mutability at play here: intrinsic and accidental (to paraphrase Fred Brooks). The intrinsic mutability comes from the changing nature of the scene deployments themselves - it's just part of the game. Accidental mutability, on the other hand, is a result of how the endpoint is consulted, with different clients asking for different X, Y ranges. So, if I'm hanging out in parcel 0,0 and you're in parcel 0,1, we might be asking for slightly different information even though the answers might be largely the same. Now, thinking about scenarios where multiple requests are made with the same parameters, like when spawning on the genesis plaza or teleporting to a popular area. In these cases, we could potentially reduce the accidental mutability factor. But to do this effectively, we would need a more nuanced approach to caching, something that might not be fully optimized in the Catalyst client right now. It would definitely be worth considering some empirical data to help us understand the trade-offs better. Metrics such as response times, cache hits and misses, network and database bandwidth saved could provide some valuable insights. This could help us determine if the added complexity of this caching mechanism is truly worth it. The ultimate goal of `/lambdas/contentv2/parcel_info` is to streamline things so that we only need one request to fetch the parcel deployment CID, the scene.json CID, and its contents, specifically the main script's CID. This could be particularly beneficial when latency to the server is high (around ~200ms), as it would likely save us from unnecessary round-trips. - The resolution of (x,y) to fetch the parcel deployment CID - The resolution of the parcel deployment CID to fetch the scene.json CID - The resolution of the scene.json CID to its contents, particularly, the main script's CID I'm mainly curious about why the proposed solution involves a POST request, which as you pointed out, can't be cached. **Could we not explore a method that allows caching via a GET request?** I'd love to hear your thoughts on this.
Hello @esteban, Thank you for raising this interesting topic. We agree that it is crucial to make the Catalyst API cache-friendly, and we are actively working to achieve this goal. However, in certain cases, relying solely on cloud caches may not be practical. For example, the content served by the **GET endpoints /lambdas/contentv2/scenes** is highly mutable, and the content pointed to by the coordinates usually changes. In such cases, caching these request would be ineffective. Instead, *POST /active/entities* can be used to request the same amount of information at once by taking a list of pointers, and the client could calculate the pointers in the square, which, if it is a much-needed functionality it can be implemented on the Catalyst client. Regarding the **GET /lambdas/contentv2/parcel_info** endpoint, the client could use *GET /content/contents/:cid* if it is interested in retrieving a single entity, which would be automatically cached by CloudFlare. Although we don't have a CloudFlare cache for this *parcel_info endpoint* (it only relies on an internal cache), it's unlikely that caching many entities at once would be helpful, as the existing CloudFlare cache has a 100% miss rate since it's rare to request the same entities in the same order repeatedly. Please let us know if you have further concerns about this and thanks again.
To be honest, I'm worried about the change of GET to POST of the last two endpoints. That prevents any chance of caching those results. How much are these endpoints used, and how many of those requests are being cached? Going forward, I'd love to try to design the catalyst APIs so that they are as cache-friendly as possible. I understand it's pretty hard, but for example, **GET /lambdas/contentv2/scenes?x1&x2&y1&y2** could be replaced with *some obscene amount of parallel HTTP requests* that could be cached/pipelined with some redirects instead of using POST requests (maybe I'm too optimistic here).
Thanks, all is clear. :handshake:
Hello @web3nit, As the changes we are making to the Catalyst API are aimed at enhancing functionality rather than causing any loss, we are just seeking a general level of acceptance **instead of going through a 3 stages poll**. If anyone is currently utilizing these endpoints or wish to retain them, we would appreciate gaining a deeper understanding of their use cases. This will enable us to determine whether it is necessary to maintain the functionality by potentially re-implementing them with a focus on performance, or if the proposed endpoints already provide solutions for the identified use cases. Thank you so much for your interest on this serie, it is highly appreciated. Please let us know if you have any further questions or concerns.
Hi, thanks for proposal. Are you planning to move your [previous poll](https://governance.decentraland.org/proposal/?id=24f524f0-eb50-11ed-ac2d-876c6fc9416f) to Draft stage ?
**About the author** I’m Alejo Ortega, a back-end developer contributing to the Decentraland Foundation in the area of Catalyst Development. This proposal was co-authored with Matias Penthreath and Hugo Arregui.
13 Comments