SQD (fka Subsquid) - Public Data Indexing Infrastructure for Polkadot and Kusama (Q4 2024)
Summary
Proponent: Subsquid Lab Official - 13bfKSQXoBn3AMLtZaW6BKv797fqZzsD3PYF6xpJDir3tdoE
Beneficiary: Subsquid Lab Official - 13bfKSQXoBn3AMLtZaW6BKv797fqZzsD3PYF6xpJDir3tdoE
Contact Details:
Subsquid Labs GmbH
6300 Zug, Switzerland
Short description: Ongoing development and maintenance costs for public SQD Archives, providing critical data infrastructure for Polkadot and Kusuma and other public Substrate-based chains in Q4 2024.
Archive raw data: Archive Infrastructure metrics - October, November, December
Requested amount: $308,704 USDC
Previous proposals: https://polkadot.polkassembly.io/referenda/1526
Motivation
A) Archives
SQD Archives are a vital component of the Polkadot and Kusama ecosystem’s data infrastructure, offering scalable access to historical on-chain data for public Substrate-based chains. These archives serve as performant data pipelines for major dApps, APIs, and developer tools, supporting critical ecosystem projects such as:
- Tanssi Network
- Talisman Wallet
- Polimec
- Polkassembly
- Apillon
- Hydration
- Giant Squid API (maintained by LimeChain, see details below)
- SubWallet
- FiDi
- KodaDot
- RMRK
- ChainSafe (in particular the Multix multi-sig, as well as other projects)
- Phala Network
- Stellaswap
- Polkascan
By providing free indexing services for the following public chains, SQD lowers barriers for developers and projects building on Substrate, fostering innovation and adoption:
- polkadot
- kusama
- asset-hub-kusama
- asset-hub-polkadot
- bridge-hub-polkadot
- collectives-polkadot
- Westend
Despite transitioning to a fee-based model for private data services to ensure sustainability, SQD remains committed to covering infrastructure costs for public archives, ensuring open access to indexed Substrate Events, Extrinsics, Storage Items, and EVM logs. This commitment aligns with Polkadot’s vision of a decentralized, accessible ecosystem.
B) Contribution
During Q4’2024, we made significant contributions to the Polkadot ecosystem:
- Infrastructure Scaling: Expanding archive capacity to meet growing data demand.
- Squid SDK Enhancements: Introducing advanced indexing features, including support for new Substrate pallets and improved error-handling mechanisms.
- Decentralized Network Growth: Increasing SQD Worker Node participation to enhance network resilience.
Achievements
Growth in Archival Data Demand, Network Expansion and Further Decentralization
In Q4 2024, demand for archival data grew steadily, rising from 11.35 TiB to 11.68 TiB. At the same time, the number of active Worker Nodes in the SQD network increased from 1,139 in October to 1,600 by the end of December—a 40.47% increase. This expansion strengthens the SQD network’s infrastructure, contributing to greater decentralization and resilience across the Polkadot ecosystem.
By the end of 2024, the SQD network had scaled to approximately 1,600 active Worker Nodes, securely storing around 900 TB of data and serving roughly 11 TB daily. This robust, decentralized infrastructure significantly enhances data processing and storage capabilities, delivering improved security, fault tolerance, and scalability for the Polkadot network.
Squid SDK improvements empowering Polkadot Developers:
- Additional unique support for Revive pallet: Enabling the tracking of contracts deployed to the PolkaVM.
- Indexing efficiency upgrades: Implementation of a novel mechanism to skip and log problematic blocks-those containing invalid or non-decodable data on Substrate based chains, ensuring uninterrupted indexing and improved resilience of the data pipeline.
Statistics and data
Over the course of Q4 2024:
- Served a total of 11,668,177,888,249 (~10.06 TiB) of data from substrate based chains.
- Served 2,336,249 archive requests from substrate based archives.
- The highest daily total data transfer recorded for Substrate was 382.57 GiB within a single 24-hour period (2024-10-05).
Cost Breakdown
This proposal encompasses the combined costs for running Archives for Polkadot, Kusama, and other public chains, along with the development contributions, amounting to a total of $308,704 USDC Raw data can be seen here
The proposal is submitted towards the Polkadot treasury. Here is a summary of the running costs for Archives, in a tabular format:
| Service | Cost | VAT 8.1% | Subtotal |
|---|---|---|---|
| Compute Engine (GCE) | $92,630.90 | $100,134.00 | $100,134.00 |
| OVHCloud | $28,419.03 | $30,720.97 | $30,720.97 |
| Networking | $2,963.02 | $3,203.02 | $3,203.02 |
| Kubernetes Engine (GKE) | $436.23 | $471.56 | $471.56 |
| Cloud Storage (GCS) | $63.21 | $68.33 | $68.33 |
| Cloud SQL | $396.38 | $428.49 | $428.49 |
| Cloud DNS | $18.01 | $19.47 | $19.47 |
| Cloud Logging | $13,506.84 | $14,600.89 | $14,600.89 |
| Cloud Key Management Service (KMS) | $2.99 | $3.23 | $3.23 |
| Artifact Registry | $336.01 | $363.23 | $363.23 |
| Engineers & Maintenance* | $144,000.00 | $155,664.00 | $155,664.00 |
| Total | $305,677.20 |
Contribution costs:
| Service | Cost / h | Quantity | VAT 8.1% | Subtotal |
|---|---|---|---|---|
| Revive pallet support | $100.00 | 16.00 | $1,729.60 | $1,729.60 |
| Indexing efficiency upgrades | $100.00 | 6.00 | $648.60 | $648.60 |
| Product management | $100.00 | 3.00 | $324.30 | $324.30 |
| Project management | $100.00 | 3.00 | $324.30 | $324.30 |
| Total | $3,026.80 |
Comments (7)
Requested
Proposal Failed
Summary
0%
Aye
0%
Nay
Aye (25)0.0 DOT
Support0.0 DOT
Nay (39)0.0 DOT
PolkaWorld votes NAY
We appreciate the continued work and effort put into maintaining and supporting the network. That said, we have some concerns regarding the scale and clarity of the proposed budget for Q4, which totals approximately $310,000. This includes cloud infrastructure and engineering costs — with $155,664 allocated to engineering and maintenance (including tax), and around $135,000 to server and network expenses.
From our perspective, a $100/hour rate for ongoing maintenance appears to be on the higher end. For reference, other Treasury-funded maintenance proposals — such as Referendum #1640 — have operated at a much lower rate (around $23/hour) while still ensuring reliable delivery.
We also believe that the “Engineers & Maintenance” cost would benefit from more detailed breakdowns — for example, how many engineers are covered by this budget and what specific tasks or responsibilities are included.
We share this feedback with respect and in the spirit of constructive discussion.
You can view our full feedback [here].
Hi @polkaworld
Thank you for taking the time to review our proposal and for sharing your thoughtful feedback. We genuinely appreciate the role you play in maintaining fiscal responsibility within the Polkadot ecosystem.
On the $100/hour rate: we understand the comparison to Referendum #1640, but the scope and delivery model differ considerably. PAL uses bounty-style engagements for audits and tool maintenance, while our funding supports a dedicated, full-time engineering team maintaining live infrastructure, with 24/7 incident response.
This team — three senior engineers skilled in Rust, networking, and DevOps — has already delivered critical contributions, such as these patches to the squid-sdk, often in response to production issues faced by ecosystem teams:
https://github.com/subsquid/squid-sdk/commit/209f88cf3b8f320dc62a3e44f713562e30283fdd
https://github.com/subsquid/squid-sdk/commit/e6ccd9b15a34086543702415d851ee66403ea937
https://github.com/subsquid/squid-sdk/commit/61d03fedd9cf9dc432a323997dbc9f3091751205
https://github.com/subsquid/squid-sdk/commit/bcc06a6eb9f5e4eec5147a67647216a506bd764c
https://github.com/subsquid/squid-sdk/commit/90e4a788a07abd19f702495823709786f466774c
https://github.com/subsquid/squid-sdk/commit/bb87496147e5a05822be4355ca99a9e7f9593d49
https://github.com/subsquid/squid-sdk/commit/03736c5657c133d04fa58269014807f88f573660
https://github.com/subsquid/squid-sdk/commit/1e3a144544402c7f3dd9461364a547fff227d8ba
https://github.com/subsquid/squid-sdk/commit/c1b9fa390e6f79e539eeaa9dce2a3509cc57998b
https://github.com/subsquid/squid-sdk/commit/366b6658c916468cf39fdd5cfc201027faa9331b
This excludes the off-the-record, real-time support these engineers have provided via Telegram, and Slack, often jumping in at short notice to help resolve infra or indexing issues for Polkadot builders.
The rate reflects standard benchmarks for senior blockchain engineers in Western Europe. In contrast, $23/hour would fall below minimum wage in our jurisdiction and make it impossible to attract or retain talent of this caliber.
Lastly, we kindly ask that you consider this in light of the fact that this is retroactive funding for work already completed. As a bootstrapped team committed to serving Polkadot’s infrastructure layer, Treasury support is essential for us to continue providing high-availability services to the community.
If there's anything we can clarify further, we’re here and happy to engage.
Best,
Thomas
Operations Lead, SQD
🏛️ REEEEEEEEEE DAO
REEEEEEEEEE DAO is a decentralized collective body committed to serve Polkadot Opengov. Our mission is to provide high-quality assessments on referenda to ensure outcomes that strengthen the Polkadot ecosystem. Each referendum is reviewed carefully by our DAO members through the scope of technical, strategic, and governance.
📋 Community Feedback
💬 Open Communication Channel
For further discussion and detailed feedback, please engage directly with our DAO members on our secured, on-chain authenticated communication platform
👉 Continue discussion with the DAO
This feedback represents the collective voice of REEEEEEEEEE DAO members participating in Polkadot governance.