substrate-etl 2.0: Polkadot + Kusama Data on Dune (12 months, 3/15/24-3/14/25)
[Dune](https://dune.com/browse/dashboards) is the industry leader in highly shareable dashboards widely used by industry leaders, data analysts, marketers, and builders across the broader Web3 ecosystem. For example here is [Brian Armstrong sharing a Dune dashboard about Base](https://twitter.com/brian\_armstrong/status/1690158283632586752), compared to [Moonbeam sharing dolpha.com](https://twitter.com/MoonbeamNetwork/status/1702721492834755061), [Astar sharing subscan](https://twitter.com/WatanabeSota/status/1731863081943277787) or [Polkadot sharing DotLake](https://twitter.com/Polkadot/status/1722980186189070760):
 |  |
 |  |
Common to the above is that Web3 ecosystem participants want to share on-chain data, which Dune is optimized for, better than a block explorer ([Subscan - #311](https://polkadot.polkassembly.io/referenda/311)) or expensive dashboards ([Dolpha.com - #341](https://polkadot.polkassembly.io/referenda/341)). Colorful Notion has built [substrate-etl - #248](https://polkadot.polkassembly.io/referenda/248), here proposed to be ingested by Dune as *substrate-etl 2.0* in 2024. ## Proposal - Onboard Polkadot + Kusama Data on Dune (12 months, 3/15/24-3/14/25) Colorful Notion proposes to onboard the Polkadot + Kusama relay chains by mid-March 2024, available for the community for at least _2 years_. This proposal funds the _first_ year after a PoC is completed in early 2024, for 3/15/24 - 3/14/25. Onboarding Polkadot will be Dune’s second non-EVM ecosystem integration, after Solana. Separate from this OpenGov proposal, Colorful Notion has negotiated pricing for all parachains with Dune, beyond the Polkadot and Kusama Relay Chain. This will enable Polkadot + Kusama Substrate data to be represented in Dune for both relay chains (using "polkadot" and "kusama") and all Polkadot parachains in an official integration with hourly indexing. # Demonstration: Polkadot Staking Dashboard In December 2023, Colorful Notion developed a proof of concept with key Dune engineers for the Polkadot Relay Chain with a few representative datasets. Here is one of the first Polkadot dashboards built in Dune: [https://dev.dune.com/mkchungs/polkadot-staking-validator\](https://dev.dune.com/mkchungs/polkadot-staking-validator) 
Once data is ingested from Colorful Notion’s substrate-etl into Dune, this dashboard can be rapidly constructed within a couple hours from SQL knowledgeable engineers such as Colorful Notion cofounder [@mkchungs](https://polkadot.polkassembly.io/user/mkchungs). Here is one query: 
Any on-chain dataset from Polkadot or Kusama involves writing a query such as the against some substrate-etl 2.0 dataset. Colorful Notion’s commercial terms have been developed to allow for a certain amount of storage ingesting a certain number of chains so that “Dune wizards” can query these tables, views, and materialized views and produce shareable charts like the above. Everyone with a Dune account can query and build these dashboards in Dune. While Google BigQuery can be connected to open source Apache Superset (and we have, see our W3F grant work on [Deep Account Analytics](https://github.com/w3f/Grants-Program/pull/1286) and inkubator-supported [ChainIDE + Polkaholic.io WASM Contract Explorer Integration](https://github.com/use-inkubator/Grant-Milestone-Delivery/blob/master/deliveries/chainide-polkaholic-integrated-wasm-contract-explorer-integration-milestone\_1.md)), Dune is more usable and more shareable. Whereas much of 2022 involved developing Polkaholic.io and 2023 involved substrate-etl 1.0 bringing data into Google BigQuery, Colorful Notion will have 2024 involve robust ingestion of Polkadot and Kusama data into Dune and ensuring the community can use it to support blockchain analytics and marketing. # Proposal Details [Full Proposal](https://docs.google.com/document/d/1UMpD152JmvHm6O9Q76TlF8GR1O\_FRJydjf7bndClj-k/edit) [Cost Breakdown](https://docs.google.com/spreadsheets/d/1c7-3kdfAjWdtWTkx2anEQgYpToUiWo3vav1WrapFuEI/edit#gid=1479719106) [Slides](https://docs.google.com/presentation/d/1wulreQof9V7QwCTuJZ9e64eDHhzAtqTS/edit#slide=id.g2a79f725924\_0\_8) **Requested Amount:** 325,000 DOT [$1,968,200] # December 2023 Decision For DOT Tokenholders: Dune Now or late 2024? Colorful Notion has been advocating a Polkadot Data Alliance Bounty along with Parity Data throughout much of 2023 (see [this from CN](https://docs.google.com/document/d/1ryC6dxcd9tiQsB7KiCc2BY\_TwBJ5jKloGfCyVAGCkKo/edit#heading=h.gjdgxs) and [this from Parity Data](https://docs.google.com/document/d/1fA5ARHy-frzgZC66rniKZ5o7CSbDvCTkS--kWaMzuMs/edit#heading=h.foftxwdr3st2)). In November, we finally decided to submit #248 and reach out to Dune as an outstanding goal of both teams. From the DOT Tokenholder point of view, once it is accepted that an ecosystem wide Dune integration is desirable and both teams are competent (as we do), we believe the decision is largely a tradeoff between Dune NOW vs Dune LATER: 
A simplified comparison is as follows: 
Which boils down to the following choice: * If you believe you want a Dune Integration NOW (early 2024), vote Aye on #366. * If you believe you want a Dune Integration LATER (or don’t want it at all), vote Nay on #366. We believe the answer is to integrate Dune NOW to support marketing and general ecosystem growth. # About Colorful Notion Colorful Notion is led by [Sourabh Niyogi](https://forum.polkadot.network/u/sourabhniyogi) and [Michael Chung](https://forum.polkadot.network/u/mkchungs). We have been active in the Polkadot ecosystem over the last 2 years and look forward to building with others in Polkadot 2.0’s “map reduce” ubiquitous computing future. ## Updates * 12/21/2023 - [AAG #93 - et-l brute?](https://www.youtube.com/watch?v=3jlA51DLz2o) * 12/22/2023 - PoC of Polkadot Relay Chain completed -- see [Polkadot Proof-of-Concept Dashboard](https://dune.com/sourabhniyogi/polkadot-proof-of-concept) * 1/3/2024 - Happy New Year 2024! Basic CN-Dune terms are being developed by Dune General Counsel; Partially redacted summary of Terms used for MSA draft may be posted here, _with permission_. * 1/4/2024 - CN received MSA from Dune. Publicly sharable CN-Dune terms are included below for transparency:  * 1/5/2024 - 3 Order forms developed for MSA. * 1/9/2024 - [Relay Chain/Parachain tables](https://docs.google.com/spreadsheets/d/1eG14AFNpJBx8Ng46XIy19hM1R\_POcT94t5mvfWaQoFE/edit#gid=1994869400) (shared with Dune permission) * 1/11/2024 - MSA finalized * 3/19/2024 - Dune public announcement: https://twitter.com/DuneAnalytics/status/1770087920994603325 # Ongoing Dashboard development in Q2 2024: [Polkadot Wiki PR](https://github.com/w3f/polkadot-wiki/pull/5933) ## Polkadot Relay Chain Polkadot Governance: * General: https://dune.com/substrate/polkadot-gov * Refs: https://dune.com/substrate/polkadot-referendum Polkadot Staking: https://dune.com/substrate/polkadot-staking ## Polkadot Parachains: AssetHub: https://dune.com/substrate/assethub Interlay: https://dune.com/substrate/interlay Astar: https://dune.com/substrate/astar-dapp-staking Bifrost: https://dune.com/substrate/bifrost HydraDX: https://dune.com/substrate/hydradx Centrifuge: https://dune.com/substrate/centrifuge Moonbeam: https://dune.com/substrate/moonbeam-governance Stellaswap: https://dune.com/substrate/stellaswap Acala: https://dune.com/substrate/acala Unique: https://dune.com/substrate/unique Phala: https://dune.com/substrate/phala Neuroweb: https://dune.com/substrate/neuroweb Manta: https://dune.com/substrate/manta Mythos: https://dune.com/substrate/mythos Collectives: https://dune.com/substrate/collectives Nodle: https://dune.com/substrate/nodle Invarch: https://dune.com/substrate/invarch ## Kusama Relay Chain Kusama Staking: https://dune.com/substrate/kusama-staking Kusama Governance: * General: https://dune.com/substrate/kusama-gov * Refs: https://dune.com/substrate/kusama-referendum ## Kusama System Chains / Other Coretime: https://dune.com/substrate/coretime\_kusama DED: https://dune.com/substrate/dotisded For the latest of ongoing ingestion status and dashboard catalyst, see this [public sheet](https://docs.google.com/spreadsheets/d/1eG14AFNpJBx8Ng46XIy19hM1R\_POcT94t5mvfWaQoFE/edit#gid=2015037)
Comments (10)
Requested
Proposal Passed
Summary
0%
Aye
0%
Nay
Aye (76)0.0 DOT
Support0.0 DOT
Nay (117)0.0 DOT
Requester address: 121Rs6fKm8nguHnvPfG1Cq3ctFuNAVZGRmghwkJwHpKxKjbx
Verified identity: Yes
When registered/verified: 2 Oct 2022 / 2 Oct 2022 (updated 15 Dec 2023)
Email: [email protected]
Twitter: @colorfulnotion
Previous OpenGov TPs from this account:
W3F Grants:
https://github.com/w3f/Grants-Program/pull/1039 (Cancelled at grantee's request)
https://github.com/w3f/Grants-Program/pull/1286 (Milestone 2 delivered, currently under review)
@BILL
Accurate! Thanks for the audit.
If any parachain wishes to chat, click here, yes, I'm a real person =)
I commend the goal, we need to be integrated with Dune - however your proposed costs are far too high for the work required and not transparent enough. In addition, 50% of the allocated funds going to your team and then charging other parachains to be a part of the Dune dataset doesn't sit well with me. Voting NAY
@birdo
Thanks for supporting the goal. We added a section above "December 2023 Decision For DOT Tokenholders: Dune Now or late 2024?" to characterize the choice, and I know you are a very thoughtful person so I hope I can change your mind a bit. If you believe Parity employees should do it later at a $400K savings at the end of 2024 (or later!) instead of Colorful Notion enterprenuers doing it at the beginning of 2024, you might not be estimating the value of a Dune integration in terms of marketcap increases.
The math is:
$400K / $9.32B = 0.00429%
We think a Dune integration NOW is worth far more than a 0.00429% change in marketcap. Do you agree with this math? If not, what did I misunderstand?
In fact, just a little bit of attention on marketing this fall caused a CMC gain of like 3-4 positions. I believe this proposal, along with a few other marketing initiatives, can increase awareness of Polkadot and its top parachains.
This integration will pay for itself if done NOW and there is little value in delaying to the end of 2024 with a bounty
The cost of Dune+GCP+CN are in the cost breakdown. That is all the precision we can provide at this time and its unrealistic to have further transparency, sorry. We do have bulk rates for Parachains with marketcap above > $100MM and those in the $15MM - $100MM range that we believe are reasonable to pay, not that dissimilar from what Etherscan and Subscan costs respectively. If parachains don't get the value of spending a proportional amount to increase their own position, they probably won't survive relative to those who do. Any Dune integration should be seen as a marketing expense and not solely an infrastructure expense. We will be able to share this after the relaychains are integrated (assuming this passes) in early Q1 but not before. Parachains who wish to integrate can reach out here.
Summary: expensive with a sense of urgency is worth FAR more than being cheap and slow.. Do the math and help others to do the same!