Blockchain Deployment Tools: CI/CD for Smart Contracts and Nodes

Decision-makers summary

Hey there! This guide is here to help you build a reliable CI/CD setup for your smart contracts and blockchain nodes by 2025. Let's dive in and get you set up! We've got some great tool recommendations for you, along with useful workflow tips and rollout strategies. These will help you reduce risks and quickly get you up and running on the mainnet. It covers a bunch of helpful insights on both EVM and non-EVM stacks. You'll find details on verification and simulation checkpoints, tips for creating reproducible builds, and info on supply-chain attestations. Plus, there’s also guidance on managing nodes with Kubernetes.

Why CI/CD for blockchain is different in 2025

In the realm of traditional software, you often have the upper hand when it comes to managing the servers where your applications run. But with web3, everything shifts a bit--you’re working with chains that aren’t exactly yours. When something goes sideways during a deployment, fixing the fallout can be pretty tough. It’s not usually a quick fix, and it can really throw a wrench in the works. On top of that, your node infrastructure needs to stay adaptable to all the fast-paced changes happening around you, whether it's updates from clients or upgrades to the network.

To keep things running like a well-oiled machine, your CI/CD process needs to focus on a few important areas. First off, it's essential to ensure everything is predictable. You also want to perform formal safety checks, like verifying storage layouts to avoid any surprises down the line. Don’t forget about those chain-specific verifications, either! Plus, running pre-flight simulations against the live state can really help catch any issues before they become a problem. Lastly, maintaining a clear audit trail is crucial for keeping regulators and partners in the loop.

Here’s a quick reference you can tweak to fit your stack perfectly at the moment.

The 2025 tooling landscape at a glance

Smart Contracts (EVM):
Build/Test: You can kick things off using Foundry, which comes with tools like forge, cast, and anvil, or you can dive in with Hardhat 3. Either way, you’ll be in good shape to get started!
Upgrades/Validation: Don't forget to take a look at the OpenZeppelin Upgrades plugins and the ERC-7201 for those namespaced storage checks. They’re pretty handy! (docs.openzeppelin.com).
Static/Dynamic Analysis: If you're looking to keep your code in shape, you can't go wrong with tools like Slither, Echidna, Mythril, and Halmos. They really help you maintain quality and catch any issues early on! (github.com).
Verification: To make sure your contracts are legit, go ahead and use Hardhat Verify. It plays nice with a bunch of different explorers. Plus, don’t forget to check out Sourcify for an extra layer of confirmation! (hardhat.org).
Simulation/Staging: When it comes to testing, Tenderly's Virtual TestNets and Simulation RPC are your best bets. Just so you know, Forks will be retired on March 31, 2025, so keep that in mind! (docs.tenderly.co).
Supply Chain: Make sure your artifacts are safe and sound! You can use GitHub artifact attestations (that’s SLSA provenance for you) paired with Sigstore cosign to add that extra layer of security. (docs.github.com).
Node Infrastructure (Ethereum):
Clients: You've got some pretty solid choices here! Geth is great if you're looking to use snap and checkpoint/BL sync features. Then there's Erigon 3, which has some handy prune modes. Don’t forget about Nethermind and Besu - both are trustworthy options too! (geth.ethereum.org).
K8s: Take a look at the Hyperledger Besu Helm patterns! The community has put together some awesome charts through ethPandaOps that cover EL/CL, metrics, and explorers. It's definitely worth exploring! (besu.hyperledger.org).
Observability: If you're diving into Geth, Nethermind, or Besu, you'll be pleased to know that they all come with built-in Prometheus metrics endpoints. (geth.ethereum.org).
Key/Secrets:
When it comes to cloud access, go ahead and use GitHub OIDC--it's super handy since you won’t need any static keys. For managing your secrets, Vault OIDC does the trick nicely. Plus, if you need to do some remote signing, Web3Signer has got you covered. And don’t forget about the Safe CLI; it’s a great tool for keeping those deployment approvals smooth and ensuring there’s a human involved in the loop! (github.com).
Quick Note about Defender: Just wanted to let you know that as of June 30, 2025, new sign-ups are no longer available. Hey there! Just a quick heads up: if you're working with Defender Relayers or Workflows, don’t forget to map out your migration plans! You can find all the info you need over at docs.openzeppelin.com. Happy migrating!

Part 1 -- CI for smart contracts (EVM)

Repository and build profiles

Let's stick with a mono-repo that includes: Alright, so here’s the deal: you’ll want to create a folder called contracts/ to store all your Solidity files. Then, set up a test/ folder where you’ll handle your testing--whether you’re using Foundry or Hardhat, that’s the spot. Don't forget about the scripts/ folder for your deployment scripts and the deploy/ folder to keep all your network configurations organized. If you’re getting audits done, stash those reports in the audits/ folder. For all your workflows, make sure you’ve got a ci/ folder. And last but not least, you’ll need a docs/ folder for all your documentation. Happy organizing! Hey everyone! Exciting news--Hardhat 3 has just launched build profiles! 🎉 This means you can now compile and verify your code using the same profile, like the production profile. It really helps to avoid any mismatches during the verification process. How cool is that? Take a look at the details right here: hardhat.org. You’ll find all the info you need!

Example (Hardhat 3 verification with production profile):

# Build and verify with the same profile
npx hardhat build --build-profile production
npx hardhat verify --network mainnet DEPLOYED_ADDRESS "arg1"

Deterministic compilation matrix

Make sure to specify your solc versions in either foundry.toml or hardhat.config. This way, you can test with the exact same compiler and optimizer settings that you plan to use when you deploy. It’s a smart move to keep everything consistent! Hey, just a quick reminder! Make sure to include a matrix for the L2s--like the chain IDs and the EIPs that are enabled. Oh, and don’t forget about the bytecode linkage and those library addresses too!

Static analysis gates

Dive into CI for quick detectors and tailored inspections:

python3 -m pip install slither-analyzer
slither . --solc-remaps @openzeppelin=node_modules/@openzeppelin

Slither is awesome for working with Foundry and Hardhat setups! It also lets you whip up your own custom detectors, like those upgradeability checks. Pretty cool, right? Feel free to take a look at it on GitHub!

Symbolic and Property Testing:

So, let's dive into symbolic and property testing! It’s all about testing programs or systems to make sure they’re doing what they’re supposed to do. In symbolic testing, you’re basically using symbols instead of specific values to create a sort of abstract representation of the program. It’s like taking a step back and looking at the big picture. On the other hand, property testing focuses on checking if certain properties hold true for the system. Think of it as verifying that the program behaves correctly under various conditions. Both methods are super useful in catching bugs early on and ensuring a smoother experience down the line!

Echidna: This is a handy tool for checking invariants! You can easily link it up using the official GitHub Action, and don't forget to save your corpus for some regression testing down the line. Check it out here.
Mythril: This tool is fantastic for diving deep into bytecode-level symbolic execution on your important artifacts. Dive into it here.
Halmos: This tool lets you really take advantage of the Foundry tests you already have for some cool symbolic exploration. Learn more here.

Upgrade safety and storage layout

It’s important to keep the storage layout consistent in all releases using OpenZeppelin Upgrades. Make sure to use validateUpgrade and prepareUpgrade in your CI process. Also, it’s a good idea not to allow any merges if there’s a layout difference. This way, you can keep everything running smoothly!
Check out ERC-7201 namespaced storage, which you can find in OpenZeppelin Contracts version 5. You’ve got some upgradeable versions available, which really helps to reduce the risk when you’re making changes to inheritance. (docs.openzeppelin.com).

Example (Foundry) in CI:

If you’re diving into Continuous Integration (CI), you really can’t go wrong with Foundry. It’s a fantastic option for making your workflow smoother and more efficient! Let’s take a quick look at how it all works and see if it could be a good match for your projects!

What’s Foundry All About?

Foundry is a super flexible platform that helps you streamline and automate your build, testing, and deployment processes. It's designed to make your life easier! It's built to make team collaboration a breeze by fitting right in with the tools you're already using.

Automation: Foundry takes care of those boring, repetitive tasks, so you can spend your time on what really matters--like the core development work you love.
Integration: It works smoothly with a bunch of tools like GitHub, Jenkins, and Docker.
Scalability: Foundry is really flexible! Whether you’re just starting out with a small team or you’ve got a big organization in the mix, it can easily grow alongside your project.

Key Features

Version Control: Stay on top of changes and work together effortlessly.
Testing Framework: This helps you run tests automatically, so you can spot any problems before they become bigger headaches.
Deployment: Get your code out the door quicker with our smooth deployment processes!

Sample Setup

Let me give you a quick example of how you can set up Foundry in your project:

# foundry.yml
version: 1.0
build:
  commands:
    - npm install
    - npm run build
test:
  commands:
    - npm test
deploy:
  commands:
    - npm run deploy

This YAML snippet is a simple and effective way to set up your build, test, and deployment processes.

Why Choose Foundry?

If you’re on the hunt for a game-changing tool to amp up your CI pipeline, Foundry might be exactly what you're looking for! With its mix of automation, seamless integration, and the ability to scale, it’s definitely a top choice for any development team out there.

Want to learn more about Foundry? Just click here to dive in!

# After forge build, run OZ upgrade validations
npx ts-node ./ci/check-upgrade.ts  # calls @openzeppelin/upgrades-core validate

OpenZeppelin Upgrades Core is equipped with a super useful CLI and API that allows you to run validations without any fuss, all within your CI setup. It's a great way to keep things smooth and efficient! Take a look at this link: docs.openzeppelin.com. It’s got some great info for you!

Gas, fuzz, and invariants as quality budgets

Think of gas snapshots in Foundry and invariant test counts as your spending limits. If the gas usage exceeds the set limits for key functions, or if we see a dip in the invariant coverage, we should definitely fail the CI.

Pre-flight simulations against live state

It’s about time we look past just testnets and dive into actually “simulating against the mainnet state.” This way, we can really get a grip on what’s happening with the mempool and MEV scenarios. Hey! Before you go ahead and push that deployment, make sure to give the Tenderly Simulation RPC a try. You can use tenderly_simulateTransaction and tenderly_simulateBundle in your CI. It’s a great way to catch any issues beforehand! If you’re looking for more info, check out this super useful guide on Tenderly's blog. It’s got everything you need to get started with the Web3 Gateway! You can find it right here. Happy reading! Hey, just a quick reminder to make sure you switch over any old “fork” processes to Virtual TestNets. They’re planning to phase out forks by March 31, 2025, so it’s a good idea to get that done sooner rather than later! Feel free to check out the details in the Tenderly documentation. It’s a great resource for getting all the info you need!

Multi-explorer verification (Etherscan, Blockscout, Sourcify)

Hardhat Verify makes it super easy to verify your contracts. You can do it with a bunch of different providers, and guess what? You can even automate the whole process programmatically! Just remember to stick to your build profile like we talked about earlier! (hardhat.org).

import hre from "hardhat";
import { verifyContract } from "@nomicfoundation/hardhat-verify/verify";
await verifyContract(
  { address, constructorArgs, provider: "sourcify" }, hre
);

Whenever you get the chance, try to opt for Sourcify’s “full metadata” verification. It really helps with long-term reproducibility! Take a look at this link: (docs.sourcify.dev). It's got some great info! Hey, if you're diving into ZK chains like zkSync Era, make sure to check out their Hardhat verification plugin when it's relevant! It'll really make your life easier. More info here: (docs.zksync.io).

Supply-chain provenance and attestations

Want to ensure your artifacts like bytecode, ABI, and addresses.json have solid build provenance? You can easily do this with GitHub’s attest-build-provenance action. It’s a part of that SLSA in-toto thing, and it’s super handy! Hey, just a quick reminder to make sure you include Sigstore verification in your admissions/gates! It's definitely worth a look. You can check it out here: github.com. Happy coding!

permissions:
  id-token: write
  contents: read
  attestations: write

- uses: actions/attest-build-provenance@v3
  with:
    subject-path: 'artifacts/**/*.json'

Hey there! Just a quick tip: if you’re working with containerized tools like builders or deployers, don’t forget to sign your images with cosign in your CI pipeline. It’s a great way to keep things secure! Take a look at this: github.com. It’s definitely worth checking out!

Part 2 -- CD for smart contracts (gating and execution)

Environment gates and secrets

Use GitHub Environments to manage your promotions better. Involve your reviewers for the mainnet, and don't forget to store your environment-specific secrets there. (docs.github.com). Forget about those old static cloud keys! You can make life easier by using GitHub OIDC (id-token: write) to take on cloud roles. This way, you can easily access deployer secrets or connect to private RPCs without the hassle.
(github.com).
Grab secrets from Vault using OIDC JWT authentication (let's ditch those pesky long-lived tokens in CI for good). (github.com).

Key custody and approvals

How about we separate the "build signer" from the "on-chain signer"? "Whenever we need to make any changes on-chain, let's suggest them through CI and get them done using multisig."

We can definitely take advantage of OpenZeppelin Upgrades' prepareUpgrade feature to deploy new implementations. Just remember, you'll need a Safe to handle the upgrade transaction. The Safe CLI is awesome for scripting proposals and executions! We can really amp it up by adding some manual review checkpoints in the Environments, too. (pypi.org).

Note on Defender

Hey there! If you’re working with Defender Relayers or Workflows, heads up! You’ll want to start prepping for a migration coming up in 2025-2026. Hey there! Just wanted to give you a quick heads-up--if you’re thinking about signing up, keep in mind that new registrations will close after June 30, 2025. And then, everything’s set to shut down on July 1, 2026. So, make sure you don’t miss out! Don’t forget to take a peek at OpenZeppelin’s migration tips! Also, if you’re feeling adventurous, you might want to explore self-hosted options like Web3Signer paired up with some custom relayers. (docs.openzeppelin.com).

Part 3 -- CI/CD for blockchain nodes

When you’re running your own nodes, whether they're RPCs, indexers, or validators, it’s super important to stay on top of a few key things. You’ve got to keep an eye on your release pipelines, make sure rollouts are safe, and ensure you have good observability in place. It's a lot to juggle, but it’s worth it to keep everything running smoothly!

Client choices and sync modes (Ethereum)

Geth: So, by default, it goes with Snap sync, which is pretty standard. But if you want to kick things up a notch and get the consensus layer up and running faster, you can try out checkpoint or beacon-light sync. Those options are definitely worth considering! Oh, and don’t forget, you can also show Prometheus metrics by using the --metrics flag! It's pretty handy! If you're curious about the details, you can take a look here. It's worth a peek!
Erigon 3: This update comes with some pretty neat new prune modes! You’ve got options like minimal, full, and archive. Plus, there are also some choices for keeping your receipts around. There's this cool "all-in-one" mode that comes with a built-in consensus client. Just a quick heads-up: the flags here aren’t exactly the same as what you’d find in Geth. Get the scoop here.
Nethermind: If you really love keeping an eye on things, you’ll appreciate that Nethermind offers detailed Prometheus metrics. They’ve got you covered on syncing and trie internals, making it a great choice for heavy monitoring! If you're looking for something that really nails detailed monitoring, this is definitely a great pick! If you want to dive deeper, you can find more info here.
Besu: Don’t forget to take a look at their official Docker images and the Kubernetes guides they have. They really have some useful stuff there! Besu also shares Prometheus metrics, which makes it a solid choice for enterprise networks. If you want to get into the nitty-gritty, check it out here. You’ll find all the juicy details waiting for you!

Kubernetes deployment patterns

Hey, don't forget to take a look at the community Helm charts from ethPandaOps! They’ve got everything you need to get your EL/CL set up, along with the sentry topology, RPC proxies, explorers, and metrics exporters. It's a great resource to help you get things rolling smoothly! You’ll come across some really handy tools like the ethereum-node umbrella, checkpointz, ethereum-metrics-exporter, and blockscout. They're all pretty cool and make working with Ethereum a lot easier! Check out this link: github.com. It’s worth a look!

helm repo add ethereum-helm-charts https://ethpandaops.github.io/ethereum-helm-charts
helm search repo ethereum-helm-charts
helm install execution ethereum-helm-charts/geth \
  --values values/geth-prod.yaml

ethPandaOps charts come packed with some super useful tools! For instance, there's checkpointz, which helps you sync your CL checkpoints seamlessly. Plus, they've got different exporters that make it easy to monitor EIP-4844 blobs. It's all about keeping things running smoothly! These tools really come into their own when you're tackling network upgrades. Feel free to take a look at them here!

Toughening up your production K8s setup: Hey there! Just a quick reminder to set runAsNonRoot, turn on readOnlyRootFilesystem, and don't forget to define your resource requests and limits. It’s super important! Make sure to set up anti-affinity rules for your EL/CL pods, and don’t forget to create Pod Disruption Budgets (PDBs) to keep things running smoothly and ensure high availability! Go ahead and choose PVCs for those fast NVMe drives, and don’t forget to create separate storage classes for your archive pods. It’ll make things a lot more organized! Make sure to set up some network policies to keep incoming RPC traffic restricted to only your API layer.

Observability

Scraping native metrics:
Geth: Take a look at /debug/metrics/prometheus by using the --metrics flag. It's pretty handy! If you want to dive deeper, check out more info here. It’s got some great details!
Nethermind: So, this one comes packed with a ton of metric namespaces, like sync, trie, AuRa, shutter, and a whole lot more. It's pretty neat! Check out all the juicy details right here.
Besu: Check out the official Kubernetes tutorial where you'll find some handy Prometheus scrape annotations and Grafana dashboards. They're really useful! Check it out here.

Rollout strategy

So, here’s the plan: You’ll want to create blue/green nodes for every client and network you’re working with, whether it’s the mainnet or those L2s. Once you’ve got them set up, you can manage the switch over by using your RPC proxy or load balancer. If you’re familiar with dshackle or erpc, you’ll find those handy charts pretty useful too! Check it out here.

When you're dealing with big forks, it’s smart to set up new client versions ahead of time and do a warm sync using snapshots or checkpoint syncs. It makes the whole process a lot smoother! And hey, make sure you don’t skip out on practicing in a private devnet! It’s super helpful. Kurtosis' ethereum-package is a really useful tool for setting up short-lived multi-client testnets, perfect for those practice sessions. If you want to dive deeper, just check it out here. There’s plenty of info waiting for you!

Key management for validators and bots

Just a quick reminder: Keep your private keys safe and stay away from those nodes! It's super important to protect your keys. Instead, you might want to check out Web3Signer for signing on EL/CL, which combines BLS and secp256k1. Just make sure to back it up using disk storage or an external vault for extra security! If you want to dive deeper into the details, just head over to the Web3Signer docs. There’s a ton of useful info waiting for you there!

Part 4 -- Non‑EVM notes (Solana and Cosmos)

Solana

For local program testing, give solana-test-validator a try. It's a handy tool! It sets you up with a complete single-node cluster, allows you to clone accounts, and lets you load BPF programs. It’s just right for integrating into your CI. (docs.solanalabs.com).
If you really want to grasp how validator operations work and get comfortable with the CLI, the official docs are your best bet. Don't forget to team that up with some community Prometheus exporters for monitoring, like the SolanaExporter. It really helps keep everything in check! (docs.solanalabs.com).

Hey program teams, have you considered using the EVM playbook? It’s a great way to go about things! Start off by running a pre-flight simulation on a local validator. Once you're comfortable there, take it to the testnet. Finally, wrap things up with a canary on the mainnet. Just a quick reminder: make sure you keep those upgrade authority windows nice and tight!

Cosmos/CosmWasm

Consistent Contract Builds: If you want to make sure you’re getting the same results every time, check out the cosmwasm/optimizer Docker or give cw-optimizoor a whirl. It really helps keep things steady! It makes your CI process super smooth by generating consistent, minimized WASM. Oh, and make sure you remember to publish those checksums! You can take a closer look here.
Chain CI: So, if you’re diving into app-chains, you’ll want to get familiar with Ignite CLI--it’s like your best friend for making the whole process of setting things up, testing, and launching a lot faster and easier! Oh, and by the way, the docs have been refreshed to include all the new info on CometBFT, which you might know better as Tendermint. Check out all the details here!

Reference CI example (GitHub Actions, EVM)

Check out this smooth, ready-to-go workflow that you can easily customize to fit your needs! It’s all about having a strong defense-in-depth approach. This means looking at everything--like analyzing what you’ve got, making sure any upgrades are legit, running simulation gates, verifying the results, and keeping track of signed provenance.

name: contracts-ci

on:
  push:
    branches: [ main ]
  pull_request:

permissions:
  contents: read
  id-token: write  # for OIDC to cloud/Vault and attestations
  attestations: write

jobs:
  test-analyze:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Setup toolchains (Node + Foundry)
      - uses: actions/setup-node@v4
        with: { node-version: '20' }
      - name: Install Foundry
        run: curl -L https://foundry.paradigm.xyz | bash && ~/.foundry/bin/foundryup

      # Install deps
      - run: npm ci

      # Compile & unit tests (Foundry)
      - run: |
          forge --version
          forge test -vvv --ffi --gas-report

      # Static analysis
      - name: Slither
        run: |
          python3 -m pip install slither-analyzer
          slither . --solc-remaps @openzeppelin=node_modules/@openzeppelin

      # Fuzz invariants (Echidna) -- keep short on PRs, longer on nightly
      - name: Echidna (smoke)
        uses: crytic/echidna-action@v2
        with:
          files: src
          test-mode: property
          format: text
          test-limit: 2000

      # Upgrade/storage layout validation
      - name: OZ Upgrades validate
        run: node ci/validate-upgrades.js

      # Pre-flight simulation (Tenderly Simulation RPC)
      - name: Simulate deployment script
        run: node ci/simulate.js  # calls tenderly_simulateTransaction
        env:
          TENDERLY_ACCESS_KEY: ${{ secrets.TENDERLY_ACCESS_KEY }}

Slither and Echidna play really well with GitHub Actions! Take a look at their stuff on GitHub! You won’t be disappointed. Hey, have you checked out Tenderly's Simulation RPC? It's pretty cool! You can actually run pre-deploy simulations using the latest chain state directly from your CI. It's a great way to make sure everything's working smoothly before you launch! Want to dive deeper? Check out the Tenderly blog! It's a great resource to get started with the Tenderly Web3 gateway and learn a ton of useful stuff.

Promotion to Mainnet with Environmental Protection, OIDC, Verification, and Attestations:

Hey everyone! We’re getting ready to launch on the mainnet, and I can't wait to share some really cool features that we’ll be rolling out with it!

Environmental Protection

We're really passionate about sustainability. We've got some pretty solid eco-friendly initiatives in place to help reduce our carbon footprint with this promotion. It really comes down to finding that sweet spot between being creative and staying responsible.

OIDC

Now, we're diving into OpenID Connect (OIDC)! Basically, this means that logging in will be easier and a lot safer for you. Get ready for a smoother login experience--it's definitely a nice perk!

Verification

Verification is super important, and we're stepping it up to make sure that everyone and every transaction is the real deal. This gives you that extra sense of trust and security, so you can feel really good about your interactions on the network!

Attestations

Finally, we're excited to introduce attestations!
This feature lets users verify specific facts about themselves while keeping their personal information private. At the end of the day, it's really about keeping your privacy intact and having control over your own data.

With these updates, we're not just keeping up; we're really raising the bar for what a mainnet can be. Keep an eye out for the launch! It’s coming soon!

  deploy-mainnet:
    needs: [ test-analyze ]
    runs-on: ubuntu-latest
    environment: mainnet  # requires approval & grants env secrets
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20' }

      # Assume cloud role via OIDC (no static keys)
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/ci-deployer
          aws-region: us-east-1

      # Fetch deploy secrets from Vault via OIDC JWT
      - uses: hashicorp/vault-action@v2
        with:
          method: jwt
          url: ${{ secrets.VAULT_URL }}
          role: ci-deployer
          secrets: |
            kv/data/contracts DEPLOYER_KEY | DEPLOYER_KEY

      - run: npm ci

      # Hardhat build/verify with production profile
      - run: npx hardhat build --build-profile production
      - run: node scripts/deploy.js --network mainnet
      - run: npx hardhat verify --network mainnet $(cat deployments/mainnet/Address.json | jq -r .MyContract)

      # SLSA provenance attestation for artifacts
      - uses: actions/attest-build-provenance@v3
        with:
          subject-path: 'artifacts/**'

If you’re looking to dive into cloud services like AWS, GCP, or Azure in 2025, using OIDC for short-term credentials is definitely the way to go! Check it out here.

Using Vault OIDC lets you store long-lived secrets safely, away from GitHub. If you're looking for more details, just check this out here. It's got all the info you need!
Getting your Hardhat Verify build profile in sync is a great way to avoid those frustrating verification mismatches.
Learn more here. Hey there! Just a quick reminder that attestations are super handy for giving you that signed proof of provenance. Make sure you're enforcing this at your K8s admission or release gates! If you want to dive deeper into the details, check out this link here. Happy reading!

For upgrades, why not let CI recommend what needs to be updated? After that, just add a Safe approval step to get everything rolling. It’s a smooth way to make sure things go according to plan! Take a look at it right here: (pypi.org). You'll find all the info you need!

Reference CD example (nodes on Kubernetes)

Helm-based Approach with Prometheus/Grafana Out of the Box and Safe Rollouts

If you're looking to deploy applications on Kubernetes, using Helm alongside Prometheus and Grafana can make your life a whole lot easier. Not only does it streamline the process, but it also gives you better visibility into what's going on. Plus, you can make sure that your rollouts are both safe and efficient. Let’s take a little dive into how this all works!

What You Get

Easy Deployment: Helm charts really take the hassle out of deploying your applications. It's just so straightforward! You can pretty much set everything up in one YAML file and then just run a couple of commands. It's that simple!
Monitoring and Visualization: Once you have Prometheus and Grafana set up together, you can easily keep tabs on your apps from the very start. It’s super handy! You’ll love the real-time metrics and those awesome dashboards! They really simplify tracking how things are performing.
Smooth Rollouts: Helm lets you do gradual rollouts, which means you can upgrade your apps without any fuss. This feature really helps cut down on downtime and lowers the chances of adding any pesky bugs to your production environment.

Key Components

Here's the usual stuff you'll need to set up:

1. Helm: Think of it as your go-to package manager for Kubernetes. It makes it way easier to define and manage all the different parts of your application.

2. Prometheus: This is a really cool monitoring tool that keeps an eye on your applications by gathering metrics. Plus, it’s got some solid alerting features to help you stay on top of things.

3. Grafana: This awesome tool helps you turn those metrics into beautiful dashboards, giving you a really clear snapshot of how your app is doing.

How to Get Started

1. Install Helm: Before you get started, just make sure you've got Helm installed on your computer! For all the nitty-gritty details on installing Helm, you can check out the official installation guide here. It's got everything you need!

2. Getting Started with Prometheus and Grafana: If you're looking to set up Prometheus and Grafana, the easiest way to get their Helm charts is by checking out their official charts repository. It’s all there waiting for you!

3. Deploy Your Application: Once you've got your monitoring stack all set up, go ahead and use Helm to get your application deployed! Let me give you a quick example of a command you can use:

helm install my-app ./my-app-chart

4. Set Up Alerts: Make sure you take a moment to configure those alerts in Prometheus! It's super helpful for staying on top of things. That way, you'll get a heads-up if anything goes wrong with your application.

5. Create Dashboards: Jump into Grafana and whip up some dashboards tailored to your needs. This way, you'll have everything you need right at your fingertips for easy monitoring!

Conclusion

If you go with a Helm-based setup along with Prometheus and Grafana, you'll really simplify your deployment process and still be able to keep tabs on how your application is performing. It’s a great way to make everything run smoothly! And hey, with safe rollouts, you can update your apps without a worry in the world! Happy deploying!.

# Add repos
helm repo add ethereum-helm-charts https://ethpandaops.github.io/ethereum-helm-charts
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Observability
helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace eth --create-namespace

# Execution client: Geth or Erigon
helm install geth ethereum-helm-charts/geth -n eth -f values/geth-prod.yaml
# or
helm install erigon ethereum-helm-charts/erigon -n eth -f values/erigon-prod.yaml

# Consensus client and checkpoint sync provider
helm install checkpointz ethereum-helm-charts/checkpointz -n eth

Hey there! If you're getting into Besu, you're in for a treat. There’s a complete Kubernetes (K8s) guide that walks you through everything you need, including how to set up Prometheus scrape annotations and Grafana dashboards. It's super helpful! Also, feel free to make adjustments to this method for your EL/CL stacks! Check it out here.

Hey Geth users! Just a heads up--there's a super useful metrics endpoint that you can check out at /debug/metrics/prometheus. It gives you all the data in Prometheus format, but don’t forget to make sure it's activated first! If you're looking for more info, you can check it out here.

Oh, and let's not overlook Erigon! It’s got three different prune modes that can really help you strike that perfect balance between disk space and performance. Whether you’re prioritizing RPC or diving into archive needs, it’s got you covered! If you want to dive deeper into it, check it out here. You’ll find some great info waiting for you!

Operational tips:

Hey there! Just a quick tip: to keep your public RPC safe, it's a good idea to use an RPC proxy. Make sure it has things like method allow-lists, rate limits, and budgets for each IP address. This way, you can protect your setup and keep everything running smoothly! Take a look at some really useful tools, like the dshackle and erpc charts. You’ll find them super helpful! You can check them out right here.

Hey, when you're working with forks or upgrades, such as Pectra, it's smart to start staging client versions ahead of time across your blue and green nodes. It'll save you some headaches later on! Go ahead and use checkpoint sync, then switch the traffic over once everything seems to be running smoothly. If you're looking for more info, check it out here. There’s plenty to discover!

Emerging practices to adopt now

Hey there! So, have you heard about namespaced storage (ERC-7201) in upgradeable contracts? It’s pretty neat! This approach really helps cut down on inheritance issues and slot hazards when it comes time for those future upgrades. Give it a look for more info! You can check it out here: docs.openzeppelin.com.

Hardhat 3 build profiles: Let’s just go with “production” for our building and verification process. It keeps things simple! This should really help us reduce some of that annoying CI flakiness we’ve been dealing with. More info here! (hardhat.org).
Tenderly: Hey folks, it’s about time we say goodbye to those outdated fork-based tests and make the leap to Virtual TestNets! Oh, and don’t forget, we can totally use the Simulation RPC for our PR gates! Check out their docs for all the details! You can find everything you need over at docs.tenderly.co. Happy reading!
Erigon 3 has this awesome feature for “minimal/full/archive” pruning that really comes in handy for keeping those RPC fleets running on a budget. Hey, just a quick reminder! Whenever you change modes, make sure to do a full reset and resync. It'll help everything run smoothly! Check it out! (docs.erigon.tech).
GitHub artifact attestations are super important if we want to achieve SLSA L3 build provenance using our reusable workflows. Let’s definitely make it a point to set this up in our cluster using admission policies! You can find all the details here. (docs.github.com).

Hey team,

Let’s get ahead of the game and start planning for the OpenZeppelin Defender migrations before we hit mid-2026. It’s a good time to take a closer look at how our Web3Signer, Safe, and custom relayer setup are doing together. Looking for more info? Check this out! (docs.openzeppelin.com). You’ll find loads of insights there!

What 7Block Labs would implement for you (pattern you can copy)

Contract CI: For our contract continuous integration, we’re all in on Foundry, Slither, Echidna, and Halmos. They’ve really got our backs! We've got OZ storage checks set up, along with a Tenderly sim gate. Plus, we're double-checking everything across different explorers to make sure everything's on point. On top of that, we'll take care of our addresses.json, the ABI, and the provenance attestations.
CD: When it comes to Continuous Delivery, we're good to go with GitHub Environments and have our required reviewers lined up. We’re using OIDC for cloud access, and we’re grabbing secrets straight from Vault. For any special actions we need to take, we’re sticking with Safe-based execution.
Nodes: We’re excited to announce that we’re rolling out some K8s Helm charts customized for each client! This means we’re also using blue/green rollouts to make updates a lot smoother. Plus, we’re keeping everything in sync with checkpoint sync to ensure a seamless experience. Great news! Our RPC proxy is all set up and ready to roll. Plus, we’ve got Prometheus and Grafana in the mix for monitoring. It’s shaping up nicely! Hey, just a quick reminder about those Web3Signer-backed keys we discussed! Also, don’t overlook the disaster-recovery runbooks we’ve created. They’ll really come in handy if anything goes wrong!

Brief, in-depth details: edge cases we actually see

Hey there! If you’re running into verification issues on Etherscan after updating to Hardhat 3, it’s usually because the build profiles don’t match up. Don't forget to pin "production" for both the building and verifying stages! For more info, just head over to hardhat.org. It's got all the details you need!

Hey there! Just a quick heads up - when you're adding those base contracts, keep an eye on any potential storage layout issues. It’s easy to miss, but it can trip you up! By using ERC-7201 alongside OpenZeppelin plugins, you can spot these issues before they become bigger problems. Just a quick reminder to make sure you have “validateUpgrade” marked as a required status check. You don’t want to miss that! If you want to dive deeper into it, check out the details at docs.openzeppelin.com. It's got everything you need to know!

Tenderly's "simulate bundle" feature is an absolute game-changer when it comes to catching those tricky upgrade initializer ordering issues that unit tests sometimes overlook. It's especially handy because, you know, live proxies already have their own state going on. Dive deeper at (docs.tenderly.co).

Hey there! If you're running into sync issues with Geth on new clusters, I highly recommend trying out the CL checkpoint sync. It can seriously speed things up--think just seconds to a few minutes to get everything finalized--before you transition to steady-state. It's a game changer! Make sure to take a look at the info over at (geth.ethereum.org). It’s got some great details you won’t want to miss!

If you're using Erigon, you definitely want to check out the prune.mode=minimal setting. It's a game changer, especially if you're running a wallet or bot setup and need to handle historical queries without keeping all that data yourself. Trust me, it’ll make your life a lot easier! Hey, just a quick heads up! If you're changing prune modes, you'll have to re-sync everything, so make sure to schedule your maintenance window accordingly. If you need more details, check out the info here: docs.erigon.tech.

Hey there! If you’re diving into Cosmos or CosmWasm, you should definitely think about adding cosmwasm/optimizer to your continuous integration setup. It’ll help you whip up reproducible WASM files and create a handy checksums.txt file, which is super useful for keeping things organized. If you’re looking for that solid audit-grade traceability, don’t forget to publish both of those when you roll out your deployment. Take a look at it over on GitHub! It's worth a peek!

Final checklist

Contracts Hey, why not kick things off by running some static analysis with Slither? After that, you could dive into property fuzzing with Echidna. And if you’re feeling adventurous, give symbolic analysis a whirl using Mythril or Halmos. It could be pretty interesting! (github.com). Don't forget to run those upgrade and storage layout checks (big thanks to OZ for the reminder!) and take a peek at ERC-7201 whenever you get a chance. (docs.openzeppelin.com). Check out those live-state simulations using Tenderly's Sim RPC, or why not take the Virtual TestNets for a spin? They're pretty cool! (blog.tenderly.co). Hey, just a quick reminder--make sure you’re using multi-explorer verification! It’s a good idea to pair up Hardhat Verify with Sourcify for the best results. Don't let it slip your mind! (hardhat.org). Make sure to keep things safe by using signed provenance (that's the attest-build-provenance) and sign your containers with cosign. It's a smart way to protect your work! (github.com).
Nodes Make sure to jot down which client you’re going with and how you want to sync things up. For example, you could use Geth’s snap or checkpoint modes, or maybe go for Erigon's prune option. Whatever fits your needs best! (geth.ethereum.org). Hey, don’t forget to get those Helm-based rollouts set up! Make sure everything’s linked up with the Prometheus metrics too. And a quick reminder: you’ll want to have that RPC proxy in front. Just a little nudge to keep it all running smoothly! (github.com).
Consider using Web3Signer or signing keys backed by HSM for extra security. It's also a good idea to manage your secrets with Vault OIDC. Plus, for giving your CI cloud access, using GitHub OIDC is a smart move. (docs.web3signer.consensys.net). Hey, before you dive into mainnet, it’s definitely a smart move to run a fork rehearsal or have a Virtual TestNet dress rehearsal. It’ll help you smooth out any bumps in the process! (docs.tenderly.co).

If you can adopt just a few of these practices, your team will ramp up their speed, increase security, and build a solid audit trail that future partners and regulators will really value.