Quickstart Guide: Python Version

Welcome to GolemDB! Use this guide to get started quickly.

Info

We're using Python here. Looking for the TypeScript version of this doc? Go here.

GolemDB is a database that stores your data inside a blockchain store, bringing the benefits of blockchain to your data, including Tamper proof, Proof of originality, and Immutability. In particular:

Immutability and tamper-evident: The data is practically impossible to alter without it being totally obvious it was changed. Data is added through blockchain transactions, and each block of transactions is crytographically linked to the one before it, forming a chain. If anyone tries to modify one of these transactions or blocks, the change could easily be detected. This is ideal where data integrity is required, such as original intellectual property and the timestamp it was created, or, for example, voting records.
Decentralized availability: Unlike data stored in a typical database server, which can be taken offline and accessed by a single person or organization, GolemDB data on Ethereum is replicated across dozens, even hundreds of nodes worldwide. This adds to the data resillience, making sure it cannot be censored or removed by any single party.
Cryptographically-verifiable provenance: Because the transactions used to add the data are signed by a private cryptographic key, this siganture acts as proof of the data's origin from a specific address. This allows you to prove the data is in its original form and that you were the one who sent it.
Unparalleled Transparency and Auditability: Because every transaction in the blockchain is public and verifiable by anyone, your data gets a permanent timestampe, and easily traceable audit trail. Using block explorers, you can easily see who submitted a particular piece of data. (This means, of course, that if you want your data private and not readable by everyone, you need to encrypt it yourself before sending it. But even though others can't read it, your or they can still verify it hasn't been altered.) This can eliminate disputes over when the data was stored and what its original state was.

Data Storage Details

Data is stored as Entities

Data is stored as what we call entities. Each entity consists of

The main data you wish to store (also called payload). This can be text or binary data, up to 128,000 bytes.
Annotations: These are key/value pairs you define, typically to describe the data. Annotation values can be strings, which together with the keys are called string annotations, or numbers, which together with the keys are called numeric annotations.

Note

Annotation keys must start with a letter or underscore, and can be followed by any combination of letters, numbers, or underscores.

Blocks to live (BTL)

When you create an entity, you specify how many blockchain blocks the entity should live; after that the entity is removed from storage. We call this the BTL, which stands for Blocks to Live.

Operations on entities

You can:

Create entities
Delete entities
Update entities
Extend the number of blocks to live on an entity
Read the entities, either just the data value, or the metadata, which consists of BTL and annotations.

Transactions

Entities are created through transactions. We have forked the popular op-geth application to allow for entities to be managed by sending the information to a specific address in the form of a transaction. When the node receives a transaction directed towards that address, our own code runs, inspecting the transaction data for the create, delete, update, and extension operations. Further, the data is given an owner based on the address that sent the transaction.

Note

You don't need to use transactions; our SDKs provide simple functions for creating, deleting, updating, and extending entities. Behind the scenes these function calls send transactions through JSON RPC calls.

Note

The data is RLP-encoded; again, our SDKs take care of that part for you.

Querying the data

We also provide a sophisticated query system that allows you to search for entities based on key/value annotation pairs, and by owner.

Environment set up

Note

You will need Python version 3.10 or greater.

Start with a new folder/directory and create a python environment as usual by typing either:

python3 -m venv venv

or

python -m venv venv

depending on how your environment is currently configured.

Activate the environment by typing:

source ./venv/bin/activate

Next, add the necessary packages:

pip install golem_base_sdk asyncio anyio eth-account xdg-base-dirs

Write Code that Creates a Wallet

Create a file called wallet.py and add the following to it:

import os
import json
import getpass
from pathlib import Path
from eth_account import Account
from xdg_base_dirs import xdg_config_home

# Disable the warning that occurs when creating a new account without a password
Account.enable_unaudited_hdwallet_features()

def create_new_wallet(password: str, file_path: Path):
    try:
        # Generate a new random account
        account = Account.create()
        account_address = account.address
        print(f"New account address created: {account_address}")

        # Encrypt the account's private key with the provided password
        encrypted_wallet = account.encrypt(password)

        # Ensure the directory exists before writing the file
        file_path.parent.mkdir(parents=True, exist_ok=True)

        # Write the encrypted JSON string to the specified file path
        with open(file_path, 'w') as f:
            json.dump(encrypted_wallet, f, indent=4)

        print(
            f"Successfully created encrypted wallet file at: {file_path}"
        )
    except Exception as e:
        print('Failed to create encrypted wallet file:', e)
        # Re-raise the exception to be caught by the main script's try/except block if needed
        raise e

if __name__ == '__main__':
    # Determine the cross-platform configuration directory
    wallet_dir = xdg_config_home() / 'golembase'
    wallet_file_path = wallet_dir / 'wallet.json'

    # Check if the wallet file already exists first
    if os.path.exists(wallet_file_path):
        print(f"Error: The file '{wallet_file_path}' already exists.")
        print("Please delete it manually if you want to create a new one.")
    else:
        # If the file doesn't exist, proceed to prompt for a password and create it
        try:
            password = getpass.getpass("Enter a password for the wallet (characters will be hidden): ")
            if not password:
                print("Password cannot be empty. Please try again.")
            else:
                create_new_wallet(password, wallet_file_path)
        except Exception as e:
            print(f"An error occurred while getting the password: {e}")

Since our purpose here is to build GolemDB apps, we won't go through this code snippet line-by-line; however, we encourage you to do so at a later time.

You should see output stating that the wallet was created, and it will also print out your address.

Copy the account address it prints out, including the starting 0x. You will need it in the next step.

Tip

If you ever need to locate the address again, you can simply open the wallet.json file found in ~/.config/golembase/wallet.json.

Fund your account

Next, fund your account by visiting the faucet for the testnet here, and pasting in your wallet number:

https://kaolin.holesky.golem-base.io/faucet/

Tip

Bookmark the above, and save your account address where you can easily find it so you can request funds periodically.

Using other wallets?

What if you want to use your own wallet, such as MetaMask? We advise against it for now. For these examples, we recommend using a "throwaway" account that you can easily recreate and start over with. The above instructions create just such a wallet account.

Build an app

Now open up your editor, create a new empty file called main.py and add the following code to it:

import asyncio
import os
from golem_base_sdk import (
    Annotation,
    GolemBaseClient,
    GolemBaseCreate,
    GolemBaseExtend,
    GolemBaseUpdate,
    decrypt_wallet,
)


async def main():
    #
    # Step 1: Read the wallet and from that the key
    #         Note: The Python SDK handles reading from the correct
    #               file and decrypting it with a single call.
    #

    try:
        private_key = await decrypt_wallet()
    except Exception as e:
        print(f"Error: {e}")
        return

if __name__ == "__main__":
    asyncio.run(main())

Go ahead and run it just to make sure you have everything correct so far by typing:

python main.py

You should see a message that the wallet was successfully decrypted.

(For the remainder of this tutorial, we recommend building and running after each section, but we won't mention it again.)

Connnecting to the TestNet

Now let's add code that connects to the testnet. Add the following to the end of your main function:

    #
    # Step 2: Create a client that connects to a GolemDB node
    #

    client = await GolemBaseClient.create_rw_client(
        rpc_url='https://kaolin.holesky.golem-base.io/rpc',
        ws_url='wss://kaolin.holesky.golem-base.io/rpc/ws',
        private_key=private_key,
    )

    #
    # Try connecting to the client.
    #     As an example, we'll request the current block number

    block = await client.http_client().eth.get_block("latest")
    print(block['number'])

Tip

Check the indentation of the first line after you paste the code in!

Creating an entity

For this next step, notice the pattern of our data structures. We'll be creating two entities and storing them with a single call to the node.

Again, add the following to the end of your main function:

    #
    # Step 3: Create two entities and store them
    #

    creates = [
        GolemBaseCreate(
            data=b'Imagine Dragons',
            btl=25,
            string_annotations=[
                Annotation('first_album', 'Night Visions'),
                Annotation('singer', 'Dan Reynolds'),
            ],
            numeric_annotations=[Annotation('year_formed', 2008)],
        ),
        GolemBaseCreate(
            data=b'Foo Fighters',
            btl=25,
            string_annotations=[
                Annotation('first_album', 'Foo Fighters'),
                Annotation('singer', 'Dave Grohl'),
            ],
            numeric_annotations=[Annotation('year_formed', 1994)],
        ),
    ]

    receipts = await client.create_entities(creates)
    print('Receipts from create (entity key and expiration block):')
    print(receipts)

Tip

The test net gets a lot of activity. Monitor it and see if 25 for BTL is really enough for your application's needs.

Deleting an entity

Now let's try out deleting an entity. Because this is a sample app, we'll just hardcode the indexes for the keys received from the receipts and save them into the variables first and second, and from there the entity keys in first_key and second_key.

Then we'll delete an entity. Then we'll check how many entities we have. (If you run this code multiple times, you might not get the number you're expecting, because entities eventually expire, possibly sooner than you expect.)

Again, add the following to the end of your main function:

    #
    # Step 4: Delete the second of the two entities
    #

    # Let's grab keys. This is just a sample so we're safe
    # hardcoding the indexes, 0 and 1.
    first = receipts[0]
    second = receipts[1]
    first_key = first.entity_key
    second_key = second.entity_key

    print('Have the first and second key:')
    print(first_key)
    print(second_key)

    # Notice we pass an array; we're allowed to delete multiple entities
    # Also notice that we're sending in an array of items of which each
    # has a member called entity_key. Thus we can simply use the receipts
    # from create_entities.
    await client.delete_entities([second])

    # Let's print out how many entities we now own.

    # Get the owner
    owner = client.get_account_address()

    # Get the count; watch closely if you this app multiple times,
    # and factor in blocks to live as well.
    entity_count = len(await client.get_entities_of_owner(owner))
    print(f"Number of entities after delete: {entity_count}")

Updating an entity

Updating an entity is a simple matter of sending over a revised entity with any changes you need, along with the key of the entity to change.

Tip

Behind the scenes, you're just replacing the entity's value and metadata with what you send to the update function. At present we don't support an "update" where you update only parts of the data, so be sure to include all the annotations, not just the ones you're changing.

Once again, add the following to the end of your main function:

    #
    # Step 5: Update an entity
    #

    # Let's update the Imagine Dragons entity by adding the second album.
    # First, we'll read the existing entity

    imagine_data = await client.get_storage_value(first_key)
    print(imagine_data.decode('utf-8'))

    imagine_metadata = await client.get_entity_metadata(first_key)
    print(imagine_metadata)

    updates = [
        GolemBaseUpdate(
            entity_key=first_key,
            btl=40,
            data=imagine_data,
            string_annotations=imagine_metadata.string_annotations + [
                Annotation('second_album', 'Smoke + Mirrors')
            ],
            numeric_annotations=[],
        ),
    ]

    await client.update_entities(updates)

    print('Entities updated!')

    # Let's verify

    imagine_metadata2 = await client.get_entity_metadata(first_key)
    print(imagine_metadata2)

Extending the blocks to live

To extend the number of blocks to live, you send over an array of items, each of which consists of a key and the number of blocks to extend by. Here's the code to add:

Add the following to the end of your main function:

    #
    # Step 6: Extend the blocks to live
    #

    # Let's extend the blocks for the first entity to live by 40

    extensions = [
        GolemBaseExtend(
            entity_key=first_key,
            number_of_blocks=40,
        ),
    ]

    await client.extend_entities(extensions)

    # Let's verify again

    imagine_metadata3 = await client.get_entity_metadata(first_key)
    print(imagine_metadata3.expires_at_block - imagine_metadata2.expires_at_block)

To verify, we're retreiving the metadata again for the same entity, and subtracting from its expiresAtBlock member the same from the previous run. It should print out 40.

A query

And finally, let's do a query. Although we might only have one block, we'll still do a query with an "and" operator in it so you can try out the query system. Here's the code to add to the end of your main function:

    #
    # Step 7: A simple query
    #

    # Let's do a quick query for demo purposes, even though we have only one entity
    result = await client.query_entities(
        'first_album="Night Visions" && singer="Dan Reynolds"'
    )
    print(result)

    await client.disconnect()

Tip

If you run this app and quickly run it again before the entities expire from the previous run, you might see more than one result in the query.

The Final Code

import asyncio
import os
from golem_base_sdk import (
    Annotation,
    GolemBaseClient,
    GolemBaseCreate,
    GolemBaseExtend,
    GolemBaseUpdate,
    decrypt_wallet,
)


async def main():
    #
    # Step 1: Read the wallet and from that the key
    #         Note: The Python SDK handles reading from the correct
    #               file and decrypting it with a single call.
    #

    try:
        private_key = await decrypt_wallet()
    except Exception as e:
        print(f"Error: {e}")
        return

    #
    # Step 2: Create a client that connects to a GolemDB node
    #

    client = await GolemBaseClient.create_rw_client(
        rpc_url='https://kaolin.holesky.golem-base.io/rpc',
        ws_url='wss://kaolin.holesky.golem-base.io/rpc/ws',
        private_key=private_key,
    )

    #
    # Try connecting to the client.
    #     As an example, we'll request the current block number

    block = await client.http_client().eth.get_block("latest")
    print(block['number'])

    #
    # Step 3: Create two entities and store them
    #

    creates = [
        GolemBaseCreate(
            data=b'Imagine Dragons',
            btl=25,
            string_annotations=[
                Annotation('first_album', 'Night Visions'),
                Annotation('singer', 'Dan Reynolds'),
            ],
            numeric_annotations=[Annotation('year_formed', 2008)],
        ),
        GolemBaseCreate(
            data=b'Foo Fighters',
            btl=25,
            string_annotations=[
                Annotation('first_album', 'Foo Fighters'),
                Annotation('singer', 'Dave Grohl'),
            ],
            numeric_annotations=[Annotation('year_formed', 1994)],
        ),
    ]

    receipts = await client.create_entities(creates)
    print('Receipts from create (entity key and expiration block):')
    print(receipts)

    #
    # Step 4: Delete the second of the two entities
    #

    # Let's grab keys. This is just a sample so we're safe
    # hardcoding the indexes, 0 and 1.
    first = receipts[0]
    second = receipts[1]
    first_key = first.entity_key
    second_key = second.entity_key

    print('Have the first and second key:')
    print(first_key)
    print(second_key)

    # Notice we pass an array; we're allowed to delete multiple entities
    # Also notice that we're sending in an array of items of which each
    # has a member called entity_key. Thus we can simply use the receipts
    # from create_entities.
    await client.delete_entities([second])

    # Let's print out how many entities we now own.

    # Get the owner
    owner = client.get_account_address()

    # Get the count; watch closely if you this app multiple times,
    # and factor in blocks to live as well.
    entity_count = len(await client.get_entities_of_owner(owner))
    print(f"Number of entities after delete: {entity_count}")

    #
    # Step 5: Update an entity
    #

    # Let's update the Imagine Dragons entity by adding the second album.
    # First, we'll read the existing entity

    imagine_data = await client.get_storage_value(first_key)
    print(imagine_data.decode('utf-8'))

    imagine_metadata = await client.get_entity_metadata(first_key)
    print(imagine_metadata)

    updates = [
        GolemBaseUpdate(
            entity_key=first_key,
            btl=40,
            data=imagine_data,
            string_annotations=imagine_metadata.string_annotations + [
                Annotation('second_album', 'Smoke + Mirrors')
            ],
            numeric_annotations=[],
        ),
    ]

    await client.update_entities(updates)

    print('Entities updated!')

    # Let's verify

    imagine_metadata2 = await client.get_entity_metadata(first_key)
    print(imagine_metadata2)

    #
    # Step 6: Extend the blocks to live
    #

    # Let's extend the blocks for the first entity to live by 40

    extensions = [
        GolemBaseExtend(
            entity_key=first_key,
            number_of_blocks=40,
        ),
    ]

    await client.extend_entities(extensions)

    # Let's verify again

    imagine_metadata3 = await client.get_entity_metadata(first_key)
    print(imagine_metadata3.expires_at_block - imagine_metadata2.expires_at_block)

    #
    # Step 7: A simple query
    #

    # Let's do a quick query for demo purposes, even though we have only one entity
    result = await client.query_entities(
        'first_album="Night Visions" && singer="Dan Reynolds"'
    )
    print(result)

    await client.disconnect()


if __name__ == "__main__":
    asyncio.run(main())

Final Links

Ready for the API? Find the Python API here