Async Subgrounds

Async Subgrounds

The Power of Concurrent Computing

The Power of Concurrent Computing

Mochan

Sept 6, 2023

Founding Engineer

Founding Engineer

From Intricacies To Insights

Since its launch, Subgrounds has amassed over 36K downloads and has been adopted by leading web3 projects like Optimism, Messari, KlimaDAO, and more. We are proud of the Playgrounds community and our close relationships with users. If you're new to Subgrounds, feel free to start your journey with this introduction. Through weekly office hours, live workshops, and community discussions, we collaborate with data analysts to pinpoint challenges and innovate solutions that push the boundaries of web3 data analytics.

Within Playgrounds, we’ve observed a recurring theme amongst our users: the pressing need to quickly and efficiently access multiple and large datasets, also known as subgraphs. Subgraphs are open APIs on The Graph that provide native access to web3 data. However, the sheer volume of information available can sometimes be overwhelming. As a result, data analysts often find themselves repeating tedious processes to get the full data set they need. Traditional Python scripts, with their linear execution, can hinder efficiency, especially when handling really extensive datasets.

Sequential computation often results in bottlenecks, causing delays in data retrieval and insights extraction. Recognizing this challenge, and after 40,000 lines of code, we are excited to unveil Async Subgrounds — the ultimate solution for concurrent subgraph querying, providing a significant boost to data processing speeds.


The World of Async

To truly grasp the unlock that Async Subgrounds brings, let's draw an analogy. Most Python scripts can be likened to a single-lane road. Every action in your script, whether importing a library, calling an API, reading a file, or even simple data processing, is akin to a car on this road. When actions are sequential, they line up one behind the other, much like cars in slow-moving traffic, causing congestion and limiting speed.

This sequential approach becomes particularly problematic when making API calls, such as with subgraphs, as we need to wait for the request to come back from the server. There's no actual driving being done on our road — the car sits idle, halting all other cars behind it. The more requests we make to the server, the more our single-lane road becomes congested.

Imagine transforming this single-lane road into a multi-lane expressway for your data analytics work. This is what Async Subgrounds offers. Asynchronous programming, or Async for short, allows for non-linear execution. Instead of tasks waiting in a queue, multiple tasks can be executed concurrently. Even if one task (like an API call) is waiting for a response, other tasks can continue unabated. In the following code snippet, we’re simultaneously querying Curve Finance pools, candles, and swaps.

import asyncio
from subgrounds import AsyncSubgrounds

# Define the main asynchronous function.
async def main():
    # Initialize the AsyncSubgrounds instance.
    sg = AsyncSubgrounds()
    
    # Load the "curve-subgraph" using AsyncSubgrounds.
    curve = await sg.load_subgraph("curve-subgraph")
  
    # Concurrently query pools, candles, and swaps from the curve subgraph.
    pools, candles, _ = await asyncio.gather(
        sg.query_df(curve.Query.pools), 
        sg.query_df(curve.Query.candles),
        sg.query_df(curve.Query.swaps),
    )

# Execute the main asynchronous function.
asyncio.run(main())


Expand Your Highways

Using Async, you can design your Subgrounds to handle vast numbers of requests with ease, streamlining your workflow. The efficiency improvements are remarkable, whether you're managing thousands of requests or integrating Subgrounds with other asynchronous frameworks. For example, it is now possible to integrate Subgrounds with a GUI framework or any application incorporating async logic to ensure Subgrounds’ processing speed doesn't disrupt user interactions or other critical front-end logic. In the code snippet below, we interface Subgrounds with a Discord bot.

from subgrounds import AsyncSubgrounds

# Initialize the AsyncSubgrounds instance.
sg = AsyncSubgrounds()

# Define an asynchronous function to handle user interactions with discord bot.
async def handle_interaction(int, subgraph_id):
    # Defer the response to the interaction.
    await int.response.defer()
    # Indicate that the bot is typing in the channel.
    async with int.channel.typing():
        # Load the specified subgraph using AsyncSubgrounds.
        subgraph = await sg.load_subgraph(subgraph_id)
        # Query the pools from the subgraph, sorted by tvl in descending order.
        data = await sg.query(
            subgraph.Query.pools(first=1, sort=subgraph.Pool.tvl, sort="desc")
        )
        # Send the queried data to the channel.
        await channel.send(data)

With that advanced functionality in place, our mission at Playgrounds remains the same: providing easy and reliable integration tools for anyone interested in leveraging on-chain data. Using Async Subgrounds, it's now possible to query any number of subgraphs across multiple blockchains simultaneously and integrate that data seamlessly into any environment.


Unlock New Horizons

While refactoring Subgrounds to support async, we needed to completely reimagine how we handled the Subgrounds client. In the showcases above, we introduced a brand new client, AsyncSubgrounds, to be able to handle async/await. Now, in Subgrounds v1.7.0, you can create your own client class to handle all of the logic that Subgrounds organizes.

This means you can use a completely different HTTP client (such as requests), implement a completely new set of query functions, or switch the actual data you return (such as returning polars DataFrames instead of pandas).

While we're still in the experimental phase with this feature, the horizon looks promising, and we're excited to see the incredible solutions our community will build on this foundation.

Join The Evolution

As we continue to enhance data analytics on The Graph via Subgrounds' powerful tooling, we eagerly look forward to seeing both new and veteran Subgrounds users dive into the updates and start tinkering!

As always, we are active in our Discord community to answer any questions or ponder any ideas you might have! Together, let's shape the future of web3 data analytics.

Check out the docs to learn more on how to integrate Async Subgrounds into your project!


Subgrounds Docs

Subgrounds Repo

Playgrounds App

Twitter

Discord

From Intricacies To Insights

Since its launch, Subgrounds has amassed over 36K downloads and has been adopted by leading web3 projects like Optimism, Messari, KlimaDAO, and more. We are proud of the Playgrounds community and our close relationships with users. If you're new to Subgrounds, feel free to start your journey with this introduction. Through weekly office hours, live workshops, and community discussions, we collaborate with data analysts to pinpoint challenges and innovate solutions that push the boundaries of web3 data analytics.

Within Playgrounds, we’ve observed a recurring theme amongst our users: the pressing need to quickly and efficiently access multiple and large datasets, also known as subgraphs. Subgraphs are open APIs on The Graph that provide native access to web3 data. However, the sheer volume of information available can sometimes be overwhelming. As a result, data analysts often find themselves repeating tedious processes to get the full data set they need. Traditional Python scripts, with their linear execution, can hinder efficiency, especially when handling really extensive datasets.

Sequential computation often results in bottlenecks, causing delays in data retrieval and insights extraction. Recognizing this challenge, and after 40,000 lines of code, we are excited to unveil Async Subgrounds — the ultimate solution for concurrent subgraph querying, providing a significant boost to data processing speeds.


The World of Async

To truly grasp the unlock that Async Subgrounds brings, let's draw an analogy. Most Python scripts can be likened to a single-lane road. Every action in your script, whether importing a library, calling an API, reading a file, or even simple data processing, is akin to a car on this road. When actions are sequential, they line up one behind the other, much like cars in slow-moving traffic, causing congestion and limiting speed.

This sequential approach becomes particularly problematic when making API calls, such as with subgraphs, as we need to wait for the request to come back from the server. There's no actual driving being done on our road — the car sits idle, halting all other cars behind it. The more requests we make to the server, the more our single-lane road becomes congested.

Imagine transforming this single-lane road into a multi-lane expressway for your data analytics work. This is what Async Subgrounds offers. Asynchronous programming, or Async for short, allows for non-linear execution. Instead of tasks waiting in a queue, multiple tasks can be executed concurrently. Even if one task (like an API call) is waiting for a response, other tasks can continue unabated. In the following code snippet, we’re simultaneously querying Curve Finance pools, candles, and swaps.

import asyncio
from subgrounds import AsyncSubgrounds

# Define the main asynchronous function.
async def main():
    # Initialize the AsyncSubgrounds instance.
    sg = AsyncSubgrounds()
    
    # Load the "curve-subgraph" using AsyncSubgrounds.
    curve = await sg.load_subgraph("curve-subgraph")
  
    # Concurrently query pools, candles, and swaps from the curve subgraph.
    pools, candles, _ = await asyncio.gather(
        sg.query_df(curve.Query.pools), 
        sg.query_df(curve.Query.candles),
        sg.query_df(curve.Query.swaps),
    )

# Execute the main asynchronous function.
asyncio.run(main())


Expand Your Highways

Using Async, you can design your Subgrounds to handle vast numbers of requests with ease, streamlining your workflow. The efficiency improvements are remarkable, whether you're managing thousands of requests or integrating Subgrounds with other asynchronous frameworks. For example, it is now possible to integrate Subgrounds with a GUI framework or any application incorporating async logic to ensure Subgrounds’ processing speed doesn't disrupt user interactions or other critical front-end logic. In the code snippet below, we interface Subgrounds with a Discord bot.

from subgrounds import AsyncSubgrounds

# Initialize the AsyncSubgrounds instance.
sg = AsyncSubgrounds()

# Define an asynchronous function to handle user interactions with discord bot.
async def handle_interaction(int, subgraph_id):
    # Defer the response to the interaction.
    await int.response.defer()
    # Indicate that the bot is typing in the channel.
    async with int.channel.typing():
        # Load the specified subgraph using AsyncSubgrounds.
        subgraph = await sg.load_subgraph(subgraph_id)
        # Query the pools from the subgraph, sorted by tvl in descending order.
        data = await sg.query(
            subgraph.Query.pools(first=1, sort=subgraph.Pool.tvl, sort="desc")
        )
        # Send the queried data to the channel.
        await channel.send(data)

With that advanced functionality in place, our mission at Playgrounds remains the same: providing easy and reliable integration tools for anyone interested in leveraging on-chain data. Using Async Subgrounds, it's now possible to query any number of subgraphs across multiple blockchains simultaneously and integrate that data seamlessly into any environment.


Unlock New Horizons

While refactoring Subgrounds to support async, we needed to completely reimagine how we handled the Subgrounds client. In the showcases above, we introduced a brand new client, AsyncSubgrounds, to be able to handle async/await. Now, in Subgrounds v1.7.0, you can create your own client class to handle all of the logic that Subgrounds organizes.

This means you can use a completely different HTTP client (such as requests), implement a completely new set of query functions, or switch the actual data you return (such as returning polars DataFrames instead of pandas).

While we're still in the experimental phase with this feature, the horizon looks promising, and we're excited to see the incredible solutions our community will build on this foundation.

Join The Evolution

As we continue to enhance data analytics on The Graph via Subgrounds' powerful tooling, we eagerly look forward to seeing both new and veteran Subgrounds users dive into the updates and start tinkering!

As always, we are active in our Discord community to answer any questions or ponder any ideas you might have! Together, let's shape the future of web3 data analytics.

Check out the docs to learn more on how to integrate Async Subgrounds into your project!


Subgrounds Docs

Subgrounds Repo

Playgrounds App

Twitter

Discord

From Intricacies To Insights

Since its launch, Subgrounds has amassed over 36K downloads and has been adopted by leading web3 projects like Optimism, Messari, KlimaDAO, and more. We are proud of the Playgrounds community and our close relationships with users. If you're new to Subgrounds, feel free to start your journey with this introduction. Through weekly office hours, live workshops, and community discussions, we collaborate with data analysts to pinpoint challenges and innovate solutions that push the boundaries of web3 data analytics.

Within Playgrounds, we’ve observed a recurring theme amongst our users: the pressing need to quickly and efficiently access multiple and large datasets, also known as subgraphs. Subgraphs are open APIs on The Graph that provide native access to web3 data. However, the sheer volume of information available can sometimes be overwhelming. As a result, data analysts often find themselves repeating tedious processes to get the full data set they need. Traditional Python scripts, with their linear execution, can hinder efficiency, especially when handling really extensive datasets.

Sequential computation often results in bottlenecks, causing delays in data retrieval and insights extraction. Recognizing this challenge, and after 40,000 lines of code, we are excited to unveil Async Subgrounds — the ultimate solution for concurrent subgraph querying, providing a significant boost to data processing speeds.


The World of Async

To truly grasp the unlock that Async Subgrounds brings, let's draw an analogy. Most Python scripts can be likened to a single-lane road. Every action in your script, whether importing a library, calling an API, reading a file, or even simple data processing, is akin to a car on this road. When actions are sequential, they line up one behind the other, much like cars in slow-moving traffic, causing congestion and limiting speed.

This sequential approach becomes particularly problematic when making API calls, such as with subgraphs, as we need to wait for the request to come back from the server. There's no actual driving being done on our road — the car sits idle, halting all other cars behind it. The more requests we make to the server, the more our single-lane road becomes congested.

Imagine transforming this single-lane road into a multi-lane expressway for your data analytics work. This is what Async Subgrounds offers. Asynchronous programming, or Async for short, allows for non-linear execution. Instead of tasks waiting in a queue, multiple tasks can be executed concurrently. Even if one task (like an API call) is waiting for a response, other tasks can continue unabated. In the following code snippet, we’re simultaneously querying Curve Finance pools, candles, and swaps.

import asyncio
from subgrounds import AsyncSubgrounds

# Define the main asynchronous function.
async def main():
    # Initialize the AsyncSubgrounds instance.
    sg = AsyncSubgrounds()
    
    # Load the "curve-subgraph" using AsyncSubgrounds.
    curve = await sg.load_subgraph("curve-subgraph")
  
    # Concurrently query pools, candles, and swaps from the curve subgraph.
    pools, candles, _ = await asyncio.gather(
        sg.query_df(curve.Query.pools), 
        sg.query_df(curve.Query.candles),
        sg.query_df(curve.Query.swaps),
    )

# Execute the main asynchronous function.
asyncio.run(main())


Expand Your Highways

Using Async, you can design your Subgrounds to handle vast numbers of requests with ease, streamlining your workflow. The efficiency improvements are remarkable, whether you're managing thousands of requests or integrating Subgrounds with other asynchronous frameworks. For example, it is now possible to integrate Subgrounds with a GUI framework or any application incorporating async logic to ensure Subgrounds’ processing speed doesn't disrupt user interactions or other critical front-end logic. In the code snippet below, we interface Subgrounds with a Discord bot.

from subgrounds import AsyncSubgrounds

# Initialize the AsyncSubgrounds instance.
sg = AsyncSubgrounds()

# Define an asynchronous function to handle user interactions with discord bot.
async def handle_interaction(int, subgraph_id):
    # Defer the response to the interaction.
    await int.response.defer()
    # Indicate that the bot is typing in the channel.
    async with int.channel.typing():
        # Load the specified subgraph using AsyncSubgrounds.
        subgraph = await sg.load_subgraph(subgraph_id)
        # Query the pools from the subgraph, sorted by tvl in descending order.
        data = await sg.query(
            subgraph.Query.pools(first=1, sort=subgraph.Pool.tvl, sort="desc")
        )
        # Send the queried data to the channel.
        await channel.send(data)

With that advanced functionality in place, our mission at Playgrounds remains the same: providing easy and reliable integration tools for anyone interested in leveraging on-chain data. Using Async Subgrounds, it's now possible to query any number of subgraphs across multiple blockchains simultaneously and integrate that data seamlessly into any environment.


Unlock New Horizons

While refactoring Subgrounds to support async, we needed to completely reimagine how we handled the Subgrounds client. In the showcases above, we introduced a brand new client, AsyncSubgrounds, to be able to handle async/await. Now, in Subgrounds v1.7.0, you can create your own client class to handle all of the logic that Subgrounds organizes.

This means you can use a completely different HTTP client (such as requests), implement a completely new set of query functions, or switch the actual data you return (such as returning polars DataFrames instead of pandas).

While we're still in the experimental phase with this feature, the horizon looks promising, and we're excited to see the incredible solutions our community will build on this foundation.

Join The Evolution

As we continue to enhance data analytics on The Graph via Subgrounds' powerful tooling, we eagerly look forward to seeing both new and veteran Subgrounds users dive into the updates and start tinkering!

As always, we are active in our Discord community to answer any questions or ponder any ideas you might have! Together, let's shape the future of web3 data analytics.

Check out the docs to learn more on how to integrate Async Subgrounds into your project!


Subgrounds Docs

Subgrounds Repo

Playgrounds App

Twitter

Discord