Running a validator
Learn how to run your own Tableland validator node.
Tableland is a permissionless network where anyone can run and operate their own node. This page walks through how to run your own node, including hardware requirements, installation steps, and common questions.
Background
go-tableland
is a Go language implementation of a Tableland node, enabling developers and service providers to run nodes on the Tableland network and host databases for web3 users and applications. Note that the Tableland protocol is currently in open beta, so node operators have the opportunity to be one of the early network adopters while the responsibilities of the validator will continue to change as the Tableland protocol evolves.
What is a validator?
Validators are the execution unit/actors of the protocol.
They have the following responsibilities:
- Listen to on-chain events to materialize Tableland-compliant SQL queries in a database engine (currently, SQLite by default).
- Serve read-queries (e.g.,
SELECT * FROM foo_69_1
) to the external world.
In the future, validators will have more responsibilities in the network.
Validator and network relationship
The following diagram describes a high level interaction between the validator, EVM chains, and the external world:
To better understand the usual mechanics of the validator, let’s go through a typical use case where a user mints a table, adds data to the table, and reads from it:
- The user will mint a table (ERC721) from the Tableland
Registry
smart contract on a supported EVM chain. - The
Registry
contract will emit aCreateTable
event containing theCREATE TABLE
statement as extra data. - Validators will detect the new event and execute the
CREATE TABLE
statement. - The user will call the
mutate
method in theRegistry
smart contract, with mutating statements such asINSERT INTO ...
,UPDATE ...
,DELETE FROM ...
, etc. - The
Registry
contract, as a result of that call, will emit aRunSQL
event that contains the mutating SQL statement as extra data. - The validators will detect the new event and execute the mutating query in the corresponding table, assuming the user has the right permissions (e.g., table ownership and/or smart contract defined access controls).
- The user can query the
/query?statement=...
REST endpoint of the validator to execute read-queries (e.g.,SELECT * FROM ...
), to see the materialized result of its interaction with the smart contract.
The description above is optimized to understand the general mechanics of the validator. Minting tables and executing mutating statements also imply more work both at the smart contract and validator levels (e.g., ACL enforcing), which are being omitted here for simplicity sake.
The validator detects the smart contract events using an EVM node API (e.g., geth
node), which can be self-hosted or served by providers (e.g., Alchemy, Infura, etc).
If you're curious about Tableland network growth, eager to contribute, or interested in experimenting, we encourage you to try running a validator. To get started, follow the step-by-step instructions provided below. We appreciate your interest and welcome any questions or feedback you may have during the process; stay tuned for updates and developments in our Discord and Twitter.
For projects that want to use the validator API, Tableland maintains a public gateway that can be used to query the network.
Running a validator
Running a validator only involves running a single process. Since we use SQLite as the default database engine, it is embedded and has many advantages:
- There’s no separate process for the database.
- There’s no inter-process communication between the validator and the database.
- There’s no separate configuration or monitoring needed for the database.
We provide everything you need to run a validator with a single command using a docker-compose setup. This will automatically build everything from the source code, making it platform-independent since most OSes support docker. The build process is also dockerized, so node operators don’t need to worry about installing compilers or similar.
If you like creating your own setup (e.g., run raw binaries, use systemd, k8, etc.), we’re also planning to automate versioned Docker images or compiled executables. If there are other setups you're interested in, feel free to let us know or even share your own setup.
The Docker Compose setup section below describes how to run a validator in more detail, including:
- Folder structure.
- Configuration files.
- Where the state of the validator lives.
- Baked in observability stack (i.e., Prometheus + Grafana with dashboard).
- Optional
healthbot
process to have an end-to-end (e2e) healthiness check of the validator.
Reviewing this section is strongly recommended but not strictly necessary.
Usage
System requirements
Currently, we recommend running the validator on a machine that has at least:
- 4 vCPUs.
- 8GiB of RAM.
- SSD disk with 10GiB of free space.
- Reliable and fast internet connection.
- Static IP.
Hardware requirements might change with time, but this setup is probably over provisioned in the current state. We’re planning to do a stress testing benchmark suite to understand and predict the behavior of the validator under different loads to have more data about potential future recommended system requirements.
Firewall configuration
If you’re behind a firewall, you should open ports :8080
or :443
, depending on if you run with TLS certificates. By default, TLS is not required, thus, expecting :8080
to be open to the external world.
System prerequisites
There are two prerequisites for running a validator:
- Install host-level dependencies.
- Get EVM node API keys.
Tableland has two separate networks:
mainnet
: this network syncs mainnet EVM chains (e.g., Ethereum mainnet, Arbitrum mainnet, etc.).testnet
: this network is syncing testnet EVM chains (e.g., Ethereum Sepolia, Arbitrum Sepolia, etc.).
This guide will focus on running the validator in the mainnet
network.
We do this for two reasons:
- The
mainnet
network is the most stable one and is also where we want the most number of validators. - We can provide concrete file paths related to
mainnet
and avoid being abstract.
We’ll also explain how to run a validator using Alchemy as a provider for the EVM node API the validator will use. The configuration will be analogous if you use self-hosted nodes or other providers. Note that if you do want to support testnets, you can, generally, replace this documentation's mainnet
reference with testnet
(e.g., an environment variable with MAINNET
would be TESTNET
; docker/deployed/mainnet
would shift to docker/deployed/testnet
).
Install host-level dependencies
To run the provided docker-compose setup, you’ll need to have installed:
git
: Installation guide.docker
with the Compose plugin: The default Docker engine installation already includes the Compose plugin (i.e.,docker compose
command).
Note that there’s no need for a particular Go
installation since binaries are compiled within a docker container containing the correct Go compiler versions. Despite not being strictly necessary, creating a separate user in the host is usually recommended to run the validator.
Create EVM node API keys
The current setup needs one API key per supported chain. The default setup expects Alchemy keys for the following: Ethereum, Optimism, Arbitrum One, and Polygon; QuickNode for Arbitrum Nova. But, you are free to use a self-hosted node or another provider that supports the targeted chains.
To get your Alchemy keys, create an Alchemy account, log in, and follow these steps:
- Create one app for each chain using the
+ Create App
button. - You’ll see one row per chain—click the
View Key
button and copy/save theAPI KEY
.
To get your QuickNode Arbitrum Nova key, create a QuickNode account, log in, and follow these steps:
- Create an endpoint.
- Select Arbitrum Nova Mainnet.
- When you finish the wizard, you should be able to have access to your API key.
Note: For Filecoin, we recommend Glif.io RPC support, which does not require authentication; the .env
variable's value (shown below) can be left empty.
Run the validator
Now that you have installed the host-level dependencies, have one wallet per chain, and provider (Alchemy, QuickNode, etc.) API keys, you’re ready to configure the validator and run it.
1. Clone the go-tableland
repository
Navigate to the folder where you want to clone the repository and run:
git clone https://github.com/tablelandnetwork/go-tableland.git
Running the main
branch should always be safe since it’s the exact code that the public validator is running. We recommend this approach since we’re moving quickly with features and improvements but expect soon to be better guided by official releases.
2. Configure your secrets in .env
files
You must configure each EVM account's private keys and EVM node provider API keys into the validator secrets:
-
Create a
.env_validator
file indocker/deployed/mainnet/api
folder—an example is provided with.env_validator.example
. -
Add the following to
.env_validator
(as noted, this focuses on mainnet configurations but could be generally replicated for testnet support):VALIDATOR_ALCHEMY_ETHEREUM_MAINNET_API_KEY=<your ethereum mainnet alchemy key>
VALIDATOR_ALCHEMY_OPTIMISM_MAINNET_API_KEY=<your optimism mainnet alchemy key>
VALIDATOR_ALCHEMY_ARBITRUM_MAINNET_API_KEY=<your arbitrum mainnet alchemy key>
VALIDATOR_ALCHEMY_POLYGON_MAINNET_API_KEY=<your polygon mainnet alchemy key>
VALIDATOR_QUICKNODE_ARBITRUM_NOVA_MAINNET_API_KEY=<your arbitrum nova mainnet quicknode key>
VALIDATOR_GLIF_FILECOIN_MAINNET_API_KEY=Note: there is also an optional
METRICS_HUB_API_KEY
variable; this can be left empty. It's a service (cmd/metricshub
) that aggregates metrics likegit summary
and pushes them to centralized infrastructure (GCP Cloud Run) managed by the core team. If you'd like to have your validator push metrics to this hub, please reach out to the Tableland team, and we may make it available to you. However, this process will further be decentralized in the future and remove this dependency entirely. -
Tune the
docker/deployed/mainnet/api/config.json
:-
Change the
ExternalURIPrefix
configuration attribute into the DNS (or IP) where your validator will be serving external requests. -
In the
Chains
section, only leave the chains you’ll be running; remove any chain entries you do not wish to support.Reference: example entry
{
"Name": "Ethereum Mainnet",
"ChainID": 1,
"Registry": {
"EthEndpoint": "wss://eth-mainnet.g.alchemy.com/v2/${VALIDATOR_ALCHEMY_ETHEREUM_MAINNET_API_KEY}",
"ContractAddress": "0x012969f7e3439a9B04025b5a049EB9BAD82A8C12"
},
"EventFeed": {
"ChainAPIBackoff": "15s",
"NewBlockPollFreq": "10s",
"MinBlockDepth": 1,
"PersistEvents": true
},
"EventProcessor": {
"BlockFailedExecutionBackoff": "10s",
"DedupExecutedTxns": true,
"WebhookURL": "https://discord.com/api/webhooks/${VALIDATOR_DISCORD_WEBHOOK_ID}/${VALIDATOR_DISCORD_WEBHOOK_TOKEN}"
},
"HashCalculationStep": 150
}
-
-
Create a
.env_grafana
file in thedocker/deployed/mainnet/grafana
folder—an example is provided with.env_grafana.example
. -
Add the following to
.env_grafana
:
GF_SECURITY_ADMIN_USER=<user name you'd like to login intro grafana>
GF_SECURITY_ADMIN_PASSWORD=<password of the user>
Note: the GF_SERVER_ROOT_URL
variable is optional and can be left empty. By default, Grafana is hosted locally at http://localhost:3000
.
That’s it...your validator is now configured!
It's worthwhile to review the config.json
file to see how the environment variables configured in the .env
files inject these secrets into the validator configuration. Also, note how supporting more chains only requires adding an extra entry in the Chains
, so it's straightforward to add support for any of the supported testnets
of each mainnet
chain. Note that adding a new mainnet
chain that's not yet supported by the network is not possible as this requires the core Tableland protocol to separately deploy a Registry
smart contract in order to enable new chain support. This is performed on a case-by-case basis, so please reach out to the Tableland team if you'd like support for a new mainnet
chain.
3. Run the validator
To run the validator, move to the docker
folder and run the following:
make mainnet-up
Some general comments and tips:
- The first time you run this, it can take some time since you’ll have a cold cache regarding images and dependencies in Docker; subsequent runs will be quite fast.
- You can inspect the general health of containers with
docker ps
. - You can tail the logs with
docker logs docker-api-1 -f
. - You can tear down the stack with
make mainnet-down
.
The default docker-compose setup has a baked-in observability substack with Prometheus and Grafana. You can learn more about this in the next section.
While the validator is syncing, you might see the logs are generated rather quickly. In the docker/deployed/mainnet/api/database.db
, you should expect that the SQLite database will start to grow in size.
Docker Compose setup
The docker-compose setup can feel a bit magical, so in this section, we’ll explain the setup's folder structure and important considerations. Remember that you don’t need to understand this section to run a validator, but knowing how things work is highly recommended.
Architecture and port bindings
When you run make mainnet-up
, you’re running the following stack:
If you’re running the validator, you’ll see these four containers running with docker ps
.
There’re two main port binding groups:
:8080
and:443
to theapi
container (the validator), depending if you have configured TLS in the validator.:3000
to thegrafana
container to access the Grafana dashboard. Remember that if you want to access to Grafana from the external world, you’ll have to configure your firewall.
Regarding the containers:
api
is the container running the validator.healthbot
is an optional container to have an e2e daemon checking the healthiness of the full write-query transaction and events execution. More about this in the Healthbot section.grafana
andprometheus
are part of the observability stack, allowing a fully-featured Grafana dashboard that provides useful live information about the validator. There's more information about this in the Observability section.
Folder structure
The docker/deployed/mainnet
folder contains one folder per process that it’s running:
api
folder: contains all the relevant secrets, configuration and state of the validator.config.json
file: the full configuration file of the validator..env_validator
file: contains secrets that are injected in theconfig.json
file.database.db*
files: when you run the validator, you’ll see these files, which are the SQLite database of the validator (running in WAL mode).
grafana
andprometheus
folders: contain any state from these daemons. For example, Grafana can include alerts or settings customizations, and Prometheus has the time-series database, so whenever you reset the container, it will keep historical data.healthbot
folder: contains secrets and configuration for the healthbot.
From an operational point of view, you usually don’t have to touch these folders apart from the api/config.json
or api/.env_validator
if you want to change something about the validator configuration or secrets. The Prometheus setup has a default 15 days retention time for the time series data, so the database size should be automatically bounded.
Configuration files
The validator configuration is done via a JSON file located at deployed/mainnet/api/config.json
.
This file contains general and chain-specific configuration, such as desired listening ports, gateway configuration, log level configuration, and chain-specific configuration, including name, chain ID, contract address, wallet private keys, and EVM node API endpoints.
The provided configurations in each deployed/<environment>
already have everything needed for the environment and other recommended values. The environment variable expansion parts of the config.json
file, such as secrets and other attributes in the .env_validator
file, were explained in the secret configuration section above. For example, the VALIDATOR_ALCHEMY_ETHEREUM_MAINNET_API_KEY
variable configured in .env_validator
expands a ${VALIDATOR_ALCHEMY_ETHEREUM_MAINNET_API_KEY}
present in the config.json
file. If you want to use a self-hosted Ethereum mainnet node API or another provider, you can edit the config.json
file in the EthEndpoint
endpoint. This same logic applies to every possible configuration in the validator.
Observability stack
As mentioned earlier, the default docker-compose setup provides a fully configured observability stack by running Prometheus and Grafana.
This setup configures the scrape endpoints in Prometheus to pull metrics from the validator and data sources dashboard for Grafana. These automatically bound configuration files are in docker/observability/(grafana|prometheus)
folders. They are not part of the state of the processes. This is intentional so that, for example, the dashboard is part of the go-tableland
repository, and you’ll get automatic dashboard upgrades while is being improved or extended.
After you spin up the validator, you can go to http://localhost:3000
and access the Grafana setup. Recall that you configured the credentials in the .env_grafana
file in docker/deployed/mainnet/grafana
.
If you browse the existing dashboards, you should see an existing Validator dashboard that should look like the following, which aggregates all metrics that the validator generates:
Healthbot (optional)
The healthbot
daemon is an optional feature of the docker-compose stack and is only needed if you support a testnet network; it's disabled by default.
The main goal of healthbot
is to test e2e in order to see if the validator is running correctly:
- For every configured chain, it executes a write statement to Tableland smart contract to increase a counter value in a pre-minted table that is owned by the validator.
- It waits to see if the increased counter in the target table was materialized in the table, thus, signaling that:
- The transaction with the
UPDATE
statement was correctly sent to the chain. - The transaction was correctly minted in the target blockchain.
- The event for that
UPDATE
was detected and processed by the validator - A
SELECT
statement reading that table should read the increased counter in the target table.
- The transaction with the
In short, it tests most of the processing healthiness of the validator. For each of the target chains, you should mint a table with the following statement:
CREATE TABLE healthbot_{chainID} (counter INTEGER);
This would result in having four tables—one per chain:
healthbot_11155111_{tableID}
(Ethereum Sepolia)healthbot_11155420_{tableID}
(Optimism Sepolia)healthbot_421614_{tableID}
(Arbitrum Sepolia)healthbot_314159_{tableID}
(Filecoin Calibration)
You should create a file .env_healthbot
in the docker/deployed/testnet/healthbot
folder with the following content (an example is provided with .env_healthbot.example
):
HEALTHBOT_ETHEREUM_SEPOLIA_TABLE=healthbot_11155111_{tableID}
HEALTHBOT_OPTIMISM_SEPOLIA_TABLE=healthbot_11155420_{tableID}
HEALTHBOT_ARBITRUM_SEPOLIA_TABLE=healthbot_421614_{tableID}
HEALTHBOT_FILECOIN_CALIBRATION_TABLE=healthbot_314159_{tableID}
Finally, edit the docker/deployed/testnet/healthbot/config.json
file Target
attribute with the public DNS where your validator is serving to the external world. This is the endpoint where the healthbot will be making the healthiness probes. Since running the healthbot
requires custom tables to be minted, it’s disabled by default.
To enable running the healthbot
, you should run the following make testnet-up
with the HEALTHBOT_ENABLED=true
environment value set:
HEALTHBOT_ENABLED=true make testnet-up
After a few minutes, you should see the HealthBot -e2e check
section of the Grafana dashboard populated:
Pruning docker images (optional)
Removing old docker images from time to time may be beneficial to avoid unnecessary disk usage. You can set up a cron
rule to do that automatically. For example, you could do the following:
- Run
crontab -e
. - Add the rule:
0 0 * * FRI /usr/bin/docker system prune --volumes -f >> /home/validator/cronrun 2>&1
Backups and other routines
All validators are equipped with a backup scheduler that runs a background routine that executes a backup process of the SQLite database file at a configurable regular frequency. Besides the main backup of the database, the Backuper
process executes a VACUUM
process in the backup file and compresses it with zstd
.
How the backup process works
The backup process called Backuper
takes a backup of SQLite
database file and stores it in a local directory relative to where the database is stored.
The process uses the SQLite Backup API provided by mattn/go-sqlite3. It is a full backup in a single step. Right now, the database is small enough not to worry about locking and how long it takes, but an incremental backup approach may be needed when as the database grows in the future.
How the scheduler works
The scheduler ticks at a regular interval defined by the Frequency
config. It is important to mention that the time it runs is relative to the epoch time. That means, as the validator becomes operational and healthy after a deployment, it will start a backup routine in the next timestamp multiple of Frequency
relative to epoch. That allows having backup files evenly distributed according to timestamp.
Vacuum
After the backup is finished, it executes the VACUUM
SQL statement in the backup database to remove any unused rows and reduce the database file. This process may take a while, but it's expected since there shouldn't be any other connections to the backup database at this point.
Compression
After vacuum, we shrink the database even further by compressing it using the zstd algorithm implemented by compress library.
Pruning
We don't keep all backup files around—at the end, we remove any files exceeding the backup's KeepFiles
config, located in cmd/api/config.go
. The default value is 5
.
Filename convention
The backup files follow the pattern: tbl_backup_{{TIMESTAMP}}.db.zst
. For example, it should resemble the following: tbl_backup_2022-08-25T20:00:00Z.db.zst
.
Decompressing the file
If you're on Linux or Mac, you should have unzstd
installed out of the box. For example, run unzstd tbl_backup_2022-08-25T20:00:00Z.db.zst
(replace with your file name) to decompress the compressed database file.
Metrics
We collect the following metrics from the process through logs:
Timestamp. time.Time
ElapsedTime time.Duration
VacuumElapsedTime time.Duration
CompressionElapsedTime time.Duration
Size int64
SizeAfterVacuum int64
SizeAfterCompression int64
Additionally, we collect the metric tableland.backup.last_execution
through Open Telemetry and Prometheus.
Configs
The backup configuration files are located in the docker/deployed/mainnet/api/config.json
file. The following is the default configuration:
"Backup" : {
"Enabled": true, // enables the backup scheduler to execute backups
"Dir": "backups", // where backup files are stored relative to db
"Frequency": 240, // backup frequency in minutes
"EnableVacuum": true,
"EnableCompression": true,
"Pruning" : {
"Enabled": true, // enables pruning
"KeepFiles": 5 // pruning keeps at most `KeepFiles` backup files
}
}
Development
Get started by following the validator setup steps described above. From there, you can make changes to the codebase and run the validator locally. For a validator stack against a local Hardhat network, you can run the following from the docker
folder:
make local-up
make local-down
For a validator stack against deployed staging environments, you can run:
make staging-up
make staging-down
Configuration
Note that for deployed environments, there are two relevant configuration files in each folder docker/deployed/<environment>
:
.env_validator
: allows you to configure environments to fill secrets for the validator, plus, expand variables present in the config file (see the.env_validator.example
example file).config.json
: the configuration file for the validator.
Besides that, you may want to configure Grafana's admin_user
and admin_password
. To do that, configure the .env_grafana
file with the values of the expected keys shown in .env_grafana.example
. This all should have been set up already but is worth noting.
Contributing
PRs accepted. Feel free to get in touch by:
- Opening an issue.
- Joining our Discord server.
Small note: If editing the README, please conform to the standard-readme specification.
License
MIT AND Apache-2.0, © 2021-2024 Tableland Network Contributors