Stellar Community Fund

By Mercury

The Mercury Ecosystem

SCF #25

$100.0K

Legacy v5.0 Community Award

Awarded

Mercury is an efficient and already-running (testnet and mainnet) indexing service which we're looking to scale.

Products & Services

The introduction of protocol 20 led to new challenges with respect to on-chain data access, signaling the need for complex, granular and customizable data retrieval. That’s where indexers come into the picture, trying to fill the gap between on-chain projects and efficient data fetching. Mercury is a managed indexer specifically tailored for the Stellar-Soroban ecosystem built by the Xycloo Labs team. Its architecture allows for high efficiency which is reflected on the end user experience as fast, reliable, granular and cheap data indexing. To learn more about Mercury’s advantages and how we tackle the new challenges of data needs on the network check out https://blog.xycloo.com/blog/explaining-mercury-stellar-event.

Many important projects being built on Stellar and Soroban are already relying on Mercury for their data indexing: the Freighter Wallet, Soroswap Finance, Soroban Governor, Kwickbit, SmartDeploy, xycLoans and individuals. Their teams are working closely with Tommaso aka tdep (our lead developer), especially for what concerns the integration with Zephyr, as we still need to integrate a more complete SDK and facilitate the developer experience (more on that later). We believe that this is already a good amount of traction given that Soroban has just now started getting traction and the service has been operational only for a couple of months. However, we’ve noticed that:

Once clients start relying on Mercury and get past the initial learning curve, they always continue working with us.

We’ve only worked closely with projects whose developers were already very familiar with Stellar and Soroban.

There’s a good percentage of customers we recognized to have less experience with Soroban that signed up or requested access and then didn’t persist with using the service.

The above insights have led us to formulate what is one of the two main components of this submission: making Mercury a fully-featured data-related ecosystem. With this submission we propose to implement and outline Mercury's significant potential to capture a substantial market share and establish itself as a leader in developing services that bridge the gap between the network’s data and various user groups, including developers, traders, and general users. We want to make Mercury the one-stop service to plug-in projects, set up simple or complex data workflows, simulate transactions, build bots, explore and test contracts, and possibly other data-related features that can scale up the workflows of projects building on Soroban and Stellar. All of this would be accessible to all users in a user-friendly Mercury ecosystem dashboard that doesn’t necessarily require a deep understanding of the underlying network data structure or format, and providing features such as auto-generated code for frontends when interacting with Mercury.

We also believe that it is crucial in this phase of users and developers onboarding for Soroban to provide the network with the best infrastructure possible in order to reduce at maximum the entry barriers. Having a powerful, managed, omni-comprehensive and user-friendly environment to deal with network’s data is a huge step forward in that direction, making available a full suite of components that would significantly simplify the processes.

More concretely, the submission can be split in two components (not necessarily of equal amount of work):

Scaling the ZephyrVM integration.

Building the “Mercury ecosystem”.

Scaling the ZephyrVM integration

The current version of the ZephyrVM works well, but only for teams that are much familiar with Soroban and Rust (outlining the need for better tooling and more work on the SDK). Additionally, the ZephyrVM is meant to replace the need of running your own infrastructure with an indexer and retrieve in real time what you want (with total customizability) from the network by writing a “simple” program, and there are still many features missing to achieve real feature completion and have the service be in fact comparable with running your own infrastructure, including database-related features (full control over your tables or defining your own queries, which is currently done manually), and VM-specific improvements on the host environment to allow for more functionality (testing toolkit, better database client, env objects for optimization, websocket alerts, etc). Another very powerful feature we also want to implement is to allow for a serverless-alike functionality for certain defined functions, so that they can also potentially process their own queries in case the custom querying functionality we already aim to implement isn’t enough. The ability to define such functions within your Zephyr program allows you to have full control over your tables (for example, have authorized functions that load data in the tables if required), over the querying (for if you need to for instance process scvals in the queries or simply need to form highly complex queries). Additionally, for programs that only need to access the current state (imagine an AMM router) you can write zephyr serverless functions to give the best path by simply accessing the ledger state and processing it, making it cheaper as it won’t be executed for every ledger close. Specifics about these improvements can be found in the architecture document. This part of the submission, while probably the most “boring” one (thus why most of the details are in the architecture docs), is one of the most important and expensive in terms of time required. A complete ZephyrVM integration can be extremely powerful for the Stellar/Soroban ecosystem, extensively tightening the gap between projects and custom processed data. Once the code is safe we will also open-source the ZephyrVM (the code is currently available for a few trusted testers) and publish an open-source indexer reliant on zephyr programs which you can connect to your database. We still strongly believe that one of the biggest advantages of using Zephyr is for it to be no-setup without having to bear infra costs and maintenance, but some services might prefer their indexing to run on their own servers with the ability to build the ingestion logic leveraging the Zephyr SDK along with its tooling. The ZephyrVM integration is also part of the Mercury Ecosystem we describe below, having also it’s management panel in the Mercury Dashboard we will later describe:

Building the Mercury Ecosystem

Completing the ZephyrVM integration means that the ecosystem can now rely either on Mercury to easily subscribe and retrieve data directly from stellar core, or on a Zephyr program for when Mercury’s pre-built structures aren’t enough for the use-case. However, as mentioned before, we see the potential for Mercury to grow beyond its current state. To make Mercury the one-stop for projects to scale their on-chain apps, we propose 3 macro-components:

The Mercury dashboard and related components.

The Mercury “simulator” and ledger API.

Testing studio.

Simulator and Ledger API

We want projects and users that rely on Mercury to avoid the necessity of depending also on additional services to simulate their transactions or to access ledger entries (the two most common practices of the RPC). Because we’re obviously already running our stellar core nodes to provide customers with data directly from the network, there’s no added cost for us to implement a very lightweight and graphql version of the RPC’s getLedgerEntries method. This ledger API can be easily integrated into a Mercury-specific fork of the new versions of Soroflare, an open source wrapper of Soroban’s simulation with the possibility of customizing the ledger in the VMs state, also built by Tommaso (note: the work on Soroflare is not part of this submission). The resulting integration, along with a very light Mercury SDK (wrapper of stellar-sdk tx assembly for working with Mercury), will enable our customers to simulate their Soroban transactions and obtain the current ledger entries with expiration as well (note that the entries functionality already exists in Mercury, but will store the entire history of the entry). Note that the simulator is not a Soroban RPC and due to its design has also an important part of Mercury’s “testing studio”, which as you’ll see below powers both Zephyr and Soroban testing activities.

Testing Studio

While it’s relatively easy to write Zephyr code for Rust-familiar folks with a good understanding of the network’s data structures, a big drawback that currently exists on Mercury is the inability to test the programs against meaningful data for your testing flows. To this purpose we will create a testing studio, both on the mercury dashboard and as helpers in the Zephyr SDK. You will have the possibility of simulating calls to an existing contract or load test snapshots from your contract’s tests. Our sdk will do the heavy lifting to construct and translate the tests/execution results into ledgerclose metas and you’ll be able to test Zephyr out, at least for what regards Soroban interactions. Besides being able to test Zephyr programs against real/tweaked data, the testing studio is also meant for users and developers to test Soroban itself, always with data-centered applications:

Snapshot the network entries you’re interested in within your Soroban rust tests through our SDK. This is particularly useful for contracts that are relying on other contracts in their workflows. For example, for testing out flash loans or oracle contracts without having to install and build them and by relying on real protocol data.

From the dashboard, save and manage checkpoints of the ledger state you’re interested into to replay them either in soroban tests or Zephyr programs.

From the dashboard, interact with on-chain contracts without actually submitting transactions (a gadget that can be potentially embedded also by dapps for witnessing and visualizing actual effects of the transaction without submitting it).

Again, this non-zephyr part of the testing studio is specifically designed for Soroban applications that are relying on other contracts, so besides the testing ease of trying out your code interacting with real chain data within your cargo tests (which is cool, but unit testing for production protocols should generally also rely on a “blank” or no-state version of the chain, i.e the tests-only snapshots), it’s perfect for things like arbitraging contracts, flash loan liquidation contracts, etc which will generally also need to index data off-chain to trigger execution. We don’t aim to provide a full testing framework for Soroban, but rather to provide our clients (or whoever needs to index data) with the best experience possible given the resources we have already available from the other features of the environment.

Dashboard

The dashboard is the UI that enables access to all of the features offered by Mercury, while also helping customers manage data subscriptions, Zephyr activity, simulation and testing under named projects. Similarly to Web2 developer-oriented services, the dashboard would serve the purpose of creating projects and managing all of the components. The dashboard in Mercury also has the purpose of highly lowering the entry barrier required for working with Mercury. For example, developers will be able to rely on our Subscriptions Builder: this will be integrated with the Mercury Transaction Simulator, so users will be able to simulate transactions from our dashboard (or they will also able to import test snapshots from your Soroban rust tests) and display all emitted events and changed/created entries, which you can click on to open a dialogue for creating subscriptions starting from that structure’s layout. The dashboard is also responsible for all the generated code snippets that we provide the users with to immediately get started into integrating Mercury’s indexing and transactions simulation, testament to the fact that we plan for indexing to be easy for all developers, even entry-level developers trying to test Soroban/Stellar. Below is an example of how we envision combining some of these dashboard features to create a new Mercury-only (no Zephyr) project, set up subscriptions and get your code snippets to immediately provide your front ends with Mercury’s indexing and simulation capabilities:

User defines the initial settings for their lending protocol data workflow:

User imports test snapshots from rust SDK to display data segments their can subscribe to (with all the required customization for smart contract objects, so for instance in this example subscribe to all entry changes that have a vector that begins with the balance symbol as contractdata key scval):

Preview all the snippets to retrieve and parse the data you’re about to subscribe to with Mercury simulation snippets for your methods as well if needed.