Media Library Step 2: Coding the Backend (Part 1)
Now move into the dist folder, and create a file called index.ts. We'll provide the complete code at the end of this file; here are the steps to create it.
Tip: We're assuming you're using VS Code, which can automatically add the import statements for you. You can either use that feature, or scroll to the bottom to the entire code and copy and paste them in.
Start by creating the express app, and configuring CORS and the json parser:
const app = express()
const port = 3000
const corsOptions = {
origin: 'http://localhost:4200',
}
app.use(cors(corsOptions))
app.use(express.json())
Next we need to configure multer so that it can process files in-memory:
const storage = multer.memoryStorage()
const upload = multer({ storage: storage })
Now comes the part where we read in the private.key file:
const keyBytes = readFileSync('./private.key')
const key: AccountData = new Tagged('privatekey', keyBytes)
export const client = await createClient(
1337,
key,
'http://localhost:8545',
'ws://localhost:8545'
)
This is making use of some types provided by the golem-base TypeScript SDK, including Tagged. Notice also that to call createClient, we use the await keyword since it's asynchronous.
Next is a little interesting; we wanted to make this backend somewhat useable without the Angular front end. As such, we supply some very simple static HTML if the user goes to the index of the app.
Note: Remember, the front end and back end both run simultaneously on two different ports. If you decide to skip the front end, the node.js backend will by default run on port 3000. If you want to see this initial form without the Angular front end, you can simply go to http://localhost:3000. (Note also that this is optional; if you don't want to use this part, you can excluse this entire route.)
app.get('/', (req, res) => {
res.send(`<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Image Uploader</title>
<style>
body { font-family: sans-serif; max-width: 600px; margin: 2em auto; }
form { display: flex; flex-direction: column; gap: 1em; }
input, button { padding: 0.5em; }
</style>
</head>
<body>
<h1>Upload an Image</h1>
<form action="http://localhost:3000/upload" method="POST" enctype="multipart/form-data">
<div>
<label for="imageFile">Choose image:</label>
<input type="file" id="imageFile" name="imageFile" accept="image/*" required />
</div>
<div>
<label for="filename">Filename (if you want it different from original):</label>
<input type="text" id="filename" name="filename" />
</div>
<div>
<label for="tags">Tags (comma-separated):</label>
<input type="text" id="tags" name="tags" value="landscape, nature, sunset" required />
</div>
<div for="custom_key1">Optional Custom Tags (Key, Value)</div>
<div>
<input type="text" id="custom_key1" name="custom_key1" value="" />
<input type="text" id="custom_value1" name="custom_value1" value="" />
</div>
<div>
<input type="text" id="custom_key2" name="custom_key2" value="" />
<input type="text" id="custom_value2" name="custom_value2" value="" />
</div>
<div>
<input type="text" id="custom_key3" name="custom_key3" value="" />
<input type="text" id="custom_value3" name="custom_value3" value="" />
</div>
<button type="submit">Upload</button>
</form>
</body>
</html>
`)
})
Now before we continue, let's build a little helper function. Image IDs are long hexadecimal numbers like so:
0x8803b6027dbc498e260e6e076d7ada2dcf4bc250be1fbd4d9cb80386d4f18e18
and they're always prepended with 0x. But sometimes people might leave the 0x off. So let's write a helper function that checks if it exists, and if not add it on:
const prepend0x = (id: string): Hex => {
// Prepend '0x' if it's missing
if (!id.startsWith('0x')) {
id = '0x' + id
}
return id as Hex
}
Notice the types we're using here. Inside the SDK we've created a TypeScript type called Hex. It's defined as 0x${string}. While not technically necessary (really it's just a string), it helps TypeScript do its job.
Now let's go in a different order from how we walked through everything in the overview. Let's finish up the helper functions and then do the express routes. There's another helper function we need: getFullImage, which reads all the chunks of the image from the database and combines them. First, let's create an interface that we can use here and elsewhere in the code:
interface ImageResult {
id: string | null
image_data: Buffer
filename: string
mimetype: string
}
When we read an image from the database, we'll return the image's ID (even though it will be passed into the function; this is just for completeness), the actual raw data in the form of a Buffer, the filename stored with the image, and the mimetype. We're not including any other annotations here, because for the app in its current state, the filename and mimetype are all we need, as you'll see in the GET /image/:id route.
Here's the code:
const getFullImage = async (id: Hex) => {
// For those not familiar with Partial, it's a great way to build up the object as we go
// without having to put a bunch of | null's at the end of each type in the Interface
// (because we don't want them to be null when we return the object.)
// Here's the ref: https://www.typescriptlang.org/docs/handbook/utility-types.html#partialtype
let result: Partial<ImageResult> = {
id: id,
mimetype: '',
filename: '',
}
// Grab the metadata
const metadata = await client.getEntityMetaData(id as Hex)
console.log(metadata)
// Grab the filename and mime type
let filename = 'image'
let partof = 1
for (let annot of metadata.stringAnnotations) {
if (annot.key == 'filename') {
filename = annot.value
} else if (annot.key == 'mime-type') {
result.mimetype = annot.value
}
}
for (let annot of metadata.numericAnnotations) {
if (annot.key == 'part-of') {
partof = annot.value
}
}
result.filename = filename
console.log(filename)
console.log(result.mimetype)
console.log(partof)
result.image_data = Buffer.from(await client.getStorageValue(id as Hex))
// See if there are more parts.
if (partof > 1) {
const chunks = [result.image_data]
// The query only gives us the payload and not the metadata, so we'll query them each individually
// (Note that we saved the values 1-based not 0-based, so the second has index 2 now)
for (let i = 2; i <= partof; i++) {
const chunk_info = await client.queryEntities(
`parent="${id}" && type="image_chunk" && app="golem-images-0.1" && part=${i}`
)
console.log(`CHUNKS ${i}:`)
console.log(chunk_info)
chunks.push(chunk_info[0].storageValue as Buffer)
}
console.log(`SENDING ${chunks.length} chunks`)
result.image_data = Buffer.concat(chunks)
}
return result as ImageResult
}
Notice how we're reading in the initial image file's metadata only, which doesn't include the actual image data. This in turn tells us how many chunks we need to read in, along with the filename and the mime type.
Notice also that the call like other calls into Golem Base is asynchronous, and hence we use the await keyword:
const metadata = await client.getEntityMetaData(id)
That in turn means our entire function needs to be given the async keyword.
Note: Because of the way annotations are stored as an array of key-value pairs, we loop through them searching for the ones we need.
Notice that at this point we're ready to read in the first chunk (or the entire image if there's only one chunk):
result.image_data = Buffer.from(await client.getStorageValue(id as Hex))
It's important that we wrap the result inside Buffer.from call. The data is actually stored as Uint8Array. Buffer is technically a superclass of Uint8Array, and node's standard libraries work well with either type. The sharp library, however, needs a Buffer instance. And this function creates a Buffer instance from the Uint8Array data.
And finally, if there are more than one chunk, we loop through the correct number, pulling in each additional chunk, adding it to an array, starting with the original one loaded. We then call Buffer.concat to combine them into a single Buffer, which gets saved in the ImageResult object.
Tip: You might take a look at the comment on Partial. That's a TypeScript helper that allows us to initialize an object with only some of the required data in our ImageResult object. Without it, TypeScript will complain that we haven't filled everything in. Then at the very end we cast the result to a full ImageResult type.
Express routes
Now let's move on to the express routes. Since we just wrote the getFullImage function, let's go ahead and write the route that gets an image by ID:
app.get('/image/:id', async (req, res) => {
let id: Hex = prepend0x(req.params.id)
let result: ImageResult = await getFullImage(id)
res.set('Content-Disposition', `inline; filename="${result.filename}"`)
res.type(result.mimetype)
res.send(result.image_data)
})
This function is pretty simple in that it just calls the two helper functions. Then notice how we use the members of the ImageResult type returned by getFullImage: We set the filename in the Content-Disposition header (the browser will then use that if you do a file-save-as), and then we set the mimetype, which came from the image when it was originally uploaded (as you'll see when we write that function soon). Then we send the image data out to the user.
Now let's look at the upload route. Here we didn't break out the code into a helper function, because at this point in this short demonstration we're only doing it once. (We considered including a "resize" route that saves resized images; in that case we would want to break out the image saving code into its own function. But we decided we wanted to keep this demonstration relatively simple, and as such left out the resizing feature. Feel free to add it!)
Here's the entire code for the image upload; remember we're assuming this is being called in response to form submission:
app.post('/upload', upload.single('imageFile'), async (req, res) => {
try {
let entity_key = ''
// Check if a file was uploaded
if (!req.file) {
return res.status(400).send('No image file was uploaded.')
}
console.log('Filename:')
console.log(req.body.filename || req.file.originalname)
// Check for the tags field
console.log(req.body)
const { tags } = req.body
if (!tags || typeof tags !== 'string') {
return res.status(400).send('Tags string is required.')
}
console.log(`Received upload with tags: "${tags}"`)
let stringAnnotations = []
let numericAnnotations = []
// Add each tag as an annotation.
const tag_list = tags
.split(',') // split by commas
.map((tag) => tag.trim()) // remove leading/trailing space
.filter((tag) => tag.length > 0) // remove empty strings resulting from multiple commas
for (let tag of tag_list) {
stringAnnotations.push(new Annotation('tag', tag))
}
for (let i = 1; i <= 3; i++) {
const key = req.body[`custom_key${i}`]
const value = req.body[`custom_value${i}`]
if (key && value) {
console.log(`Found custom key/value ${i}:`)
console.log(key, value)
if (typeof value === 'number' && !isNaN(value)) {
numericAnnotations.push(new Annotation(key, value))
} else {
stringAnnotations.push(new Annotation(key, String(value)))
}
}
}
// The original image is already in memory as a Buffer
const originalImageBuffer = req.file.buffer
console.log(`Original image size: ${originalImageBuffer.length} bytes`)
// Create a resized thumbnail using Sharp
// sharp takes the buffer, resizes it, and outputs a new buffer
console.log('Resizing image to 60px width...')
const resizedImageBuffer = await sharp(originalImageBuffer)
.resize({
width: 100,
height: 100,
fit: 'inside', // This ensures the image is resized to fit within a 100x100 box
})
.jpeg({ quality: 70 })
.toBuffer()
console.log(`Resized image size: ${resizedImageBuffer.length} bytes`)
// Break into chunks if it's too big
const chunks: Buffer[] = []
const chunkSize = 100000
for (let i = 0; i < originalImageBuffer.length; i += chunkSize) {
const chunk = Buffer.from(
originalImageBuffer.subarray(i, i + chunkSize)
)
chunks.push(chunk)
}
console.log(`Number of chunks: ${chunks.length}`)
for (let chunk of chunks) {
console.log(chunk.length)
}
try {
// We have to do these creates sequentially, as we need the returned hash to be used in the thumbnail (and additional parts if needed).
let creates_main: GolemBaseCreate[] = [
{
data: chunks[0],
btl: 25,
stringAnnotations: [
new Annotation('type', 'image'),
new Annotation('app', 'golem-images-0.1'),
new Annotation(
'filename',
req.body.filename || req.file.originalname
),
new Annotation('mime-type', req.file.mimetype),
...stringAnnotations,
],
numericAnnotations: [
new Annotation('part', 1),
new Annotation('part-of', chunks.length),
...numericAnnotations,
],
},
]
console.log('Sending main:')
console.log(inspect(creates_main, { depth: 10 }))
const receipts_main = await client.createEntities(creates_main)
let hash = receipts_main[0].entityKey
console.log('Receipts for main:')
console.log(receipts_main)
entity_key = receipts_main[0].entityKey
// Now if there are more chunks for the larger files, build creates for them.
// Start at index [1] here, since we already saved index [0]
for (let i = 1; i < chunks.length; i++) {
const next_create: GolemBaseCreate[] = [
{
data: chunks[i],
btl: 25,
stringAnnotations: [
new Annotation(
'parent',
receipts_main[0].entityKey
),
new Annotation('type', 'image_chunk'),
new Annotation('app', 'golem-images-0.1'),
new Annotation(
'filename',
req.body.filename || req.file.originalname
),
new Annotation('mime-type', req.file.mimetype),
...stringAnnotations,
],
numericAnnotations: [
new Annotation('part', i + 1),
new Annotation('part-of', chunks.length),
...numericAnnotations,
],
},
]
const next_receipt = await client.createEntities(next_create)
console.log(`Next receipt: (part ${i + 1})`)
console.log(next_receipt)
}
console.log('Sending thumbs and chunks:')
let create_thumb: GolemBaseCreate[] = [
{
data: resizedImageBuffer,
btl: 25,
stringAnnotations: [
new Annotation('parent', receipts_main[0].entityKey),
new Annotation('type', 'thumbnail'),
new Annotation('app', 'golem-images-0.1'),
new Annotation('resize', '100x100'),
new Annotation(
'filename',
`thumb_${req.body.filename || req.file.originalname}`
),
new Annotation('mime-type', 'image/jpeg'), // Our thumbnail is jpg
...stringAnnotations,
],
numericAnnotations: [],
},
]
const receipts_thumb = await client.createEntities(create_thumb)
console.log('Receipts for thumb:')
console.log(receipts_thumb)
} catch (e) {
console.log('ERROR')
if (e instanceof Error) {
if ((e as any)?.cause?.details) {
throw (e as any).cause.details
}
} else {
throw e
}
}
res.status(200).json({
message: 'File processed successfully!',
originalSize: originalImageBuffer.length,
resizedSize: resizedImageBuffer.length,
tags: tags,
entity_key: entity_key,
})
} catch (error) {
console.error('Error processing image:', error)
res.status(500).send(
`An error occurred while processing the image: ${error}`
)
}
})
At this point you should have a pretty good feel for how we're chunking out the images, and as such we won't go through this code line-by-line. But let's call out a few important parts.
First, notice how we build up the string annotations from the data passed in. We split up the single tag line into individual strings, and push them into an array of Annotations called stringAnnotations. We then go through any custom entries include as well, doing the same.
Tip: Golem Base lets you have multiple annotations with the same key. That's incredibly handy here for our tags. We create multiple tag annotations. Then we can query on any of them as in tag="pets".
Notice that we grab the uploaded file and save it into a variable as is:
const originalImageBuffer = req.file.buffer
Ultimately, if there's only one chunk, this is the same buffer we'll be pushing out to Golem-base.
Next we go through a loop that checks if the image is greater than 100,000 bytes, and if so, starts chunking it out. We push each chunk into an array.
We then build the first instance of GolemBaseCreate, which holds all the data for either the first chunk or the entire image if it's smaller. Here you'll see where we're adding in the app annotation, a type annotation of image, the filename annotation, the mime-type, and the part and part-of annotations as numbers.
Then we send that one out via:
const receipts_main = await client.createEntities(creates_main)
Next if there are additional chunks, we build up new instances of GolemBaseCreate similar to the original, except the type is image_chunk, the part number is incremented, and we're also including the id of the first chunk.
Note: Each chunk will get its own ID inside Golem-Base. But we're using the first chunk as the main ID for the image.
And finally, we build up a GolemBaseCreate instance for the thumbnail. For that we're using code from earlier that we haven't looked at yet, the code that uses Sharp to resize the image. Notice in that code we're specifying a width of 100, a height of 100, and a fit of "inside" which means if one side is longer than the other, the longer side will become size 100, and the shorter side will be shortened by the aspect ratio, padded with extra space. That way the original image will be visible inside the little thumbnail.
And now we build up the GolemBaseCreate; notice that it's using similar information as earlier. However, notice the mime-type is jpeg, as that's what we used for the thumbnail creation.
And finally we send back a little bit of JSON noting that the image was saved successfully, along with the ID of the saved image. (And again, that's the first chunk if there are more than one.)
Let's continue with the remainder of the express routes in the next step.
Head to Step 3.