The Stage

What is a side project, even the most interesting and challenging one without a lot of fun 🤸? We don’t want to spend our free time with boring stuff, do we? What brings us joy is what pushes good side projects forward. The fun aspect is what I’d like to focus on today and drift away a bit from the technicalities behind Cinode 🙃. There will be a bit of a source code at the end so you may want to scroll down if you’re not interested in the fun part 😉. But if you’re still curious, try guessing what I come up with this time without scrolling down 🤫. Let me slowly reveal the puzzle then…

Tools

First hint ahead, here are some of the tools that I’ve used while preparing for post:

• Some wires - I took them out of some old box with computer cables
• Glue
• Sharp knife
• Dremel
• Some dirt cheap soldering iron
• Blender

Well, most of those tools are definitely used to build something physical, not another piece of code but something that one can touch and sense. But one tool stands out - Blender - why is it needed? That’s a 3D modeling software, it is used to build virtual things, not physical ones. Maybe some 3D movies or computer games, right? Well, if you’re curious, keep reading…

Actors on the stage

Today we’ll also meet again our old friend from my previous blog post - RaspberryPI Zero W. We know that this small SoC can run some Cinode code - but so far it was a proof-of-concept rather than something really useful. Today we’ll get to a bit more advanced usage down below, in the code part.

But before we assign our hero some tough work to do, let’s figure out where to put him - I mean physically, where to place him so that he can do its job in a calm and peaceful environment.

I have an access to a small-ish server rack that I could use here (and it’s not a coincidence, it was planned for a long time 😏). And because Cinode is a network-related project, a rack like this is a perfect place to put some Cinode actors in there. Let’s build up the stage then…

Back to the mysterious 3D modeling software that I mentioned before. Any guess what I need it for?

3D models

We’ve got the actor - our little Raspberry Pi board. And we’ve got the place where our stage will be located - the rack (now I can’t resist calling Raspberry Pi a Bard-Board and the scene a Rack-Stage 🤣). Now let’s think about what the stage itself should look like. Server racks have slots for putting things inside and there are standard shapes that match the rack interior. That’s why we’ll need some 3D modeling software. Those shapes have to be somehow created - and my plan was to build the 3D model in Blender and print it using a 3D printer. Blender is an open-source free software so no issue with that. And my son got a 3D printer as a gift few months ago (a rather low-spec one, but ideal for our current use-case) thus, with his permission of course 🙄, I could print some stage components in there. Sounds like a perfect plan to fill-in that Cinode rack slot.

First important piece for the whole setup is the power supply. Our boards are usually powered up through some microUSB or usb-C inputs. That’s a 5V power source. But in order to avoid putting many different USB chargers why not have a single Power Supply Unit (PSU)? I decided to buy a 5V PSU with enough wattage to power up at least a dozen boards like the one we’ll use today. That’s plenty of spare power that we could use in the future.

Next to the PSU we will need some way to mount the board. It is very convenient to have some flexible attachment mechanism - that way devices can be swapped out easily. My design consists of a two-level approach. The PSU will be directly connected to larger and universal plates that can be easily attached to each other - that way we can quickly add more overall space for boards. I also created a simple latch mechanism on each plate that will hold the board during normal operation. When needed, the board will be released by simply pressing that latch. How it is done - that’s something for a separate post.

Here’s a 3D model of the PSU box and a mount point for a single board:

I have to mention that before this project I did not use Blender a lot (only learned some basics a few years ago) so it took me some time to get enough experience for my 3D modeling needs. In the end I think the result is really nice and I found out that we have a ton of quality online learning resources for Blender nowadays.

Now a little bit about the design. The PSU box is the base of the scene - it creates a separate box for the PSU to isolate it from the rest of the system. I wanted to have such separation to avoid electrical incidents. The PSU takes 220V as input so better be safe than sorry. It also needs some air to disperse the heat. I don’t think it will be loaded a lot but I’ve added some ventilation holes on the exterior sides to get some fresh air in there:

Plates with a latch system are then attached to the PSU box. I’ve made it such that segments can be also attached to each other - which means that the whole setup can be easily extended for new boards if necessary. I also made sure that 5V lines are embedded into the plate in such a way that one doesn’t have to manually connect or solder any wires to attach the power source to the next plate. Mount points on those plates already have all necessary sockets and connectors built in so everything just works if you connect all the pieces together.

Of course such a design took some time to adjust and find the correct shape. I made it iteratively and with a lot of prototyped parts. For example the latch needed 3 prototypes before I found the correct gap sizes so that things could slide in smoothly. But here it is - Cinode PSU mounted to the rack with two mount points attached:

You may have noticed that there are 6 screws on the PSU box to keep such a small setup attached to the rack case. That way it will be a rock-solid mount 😉. It’s definitely more than needed. I have a ton of rack bolts laying around though so why not just use them? But I think that even with 2 it would be strong enough.

Let’s bring Actors onto the stage

Now is the time to reveal the best part.

How do we attach boards to plates? The shape of a Raspberry Pi does not match the latch on the plate so I had to come up with an additional holder for it - something maybe inspired by blade servers? But when I was thinking about the shape of the holder I realized that current server racks, even though well organized and pretty, are a bit boring to me. While I was thinking about different ways to mount our little boards to the plate I imagined how those will be used.:

• each such board will have its own small role in the whole cluster…
• and I plan to have a lot of them
• just like small enjoyable minions in their tribe 😏

Thus let me introduce….. 🥁🥁🥁🥁

… printing …

… still printing, almost there … 🥁🥁🥁🥁

🥁

🥁

🥁

🥁

🥁

🥁

🥁

🥁

Old dude with a new look

CinoDave (spelled See-No-Dave 😉, but he prefers the shorter form - just Dave), even tough looks a bit crappy without his natural colors, is a diligent and joyful pioneer in our Cinode cluster. He was born out of our well-known Raspberry Pi Zero W board proudly showing it to the world in the front of his belly. Dave, besides testing the arm32 builds of Cinode, has a dedicated role to play on the stage - he’s The Builder.

This role replaces some of our previous work where a legacy CI/CD pipeline was used to take the source code of this blog, compile it and deploy into the Cinode world. Now it is Dave’s job.

Let’s take a look at his screenplay:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144

#!/bin/bash

# 
# Prelude
# =======
#

# Bash strict mode on
set -euo pipefail

# Load secret and not-so-secret knowledge
test -f ./env && . ./env

# Setup defaults
DEFAULT_GIT_REMOTE_NAME="origin"
DEFAULT_GIT_REMOTE_BRANCH="main"
DEFAULT_NO_CHANGE_SLEEP=60

: "${GITREMOTE_NAME:=${DEFAULT_GIT_REMOTE_NAME}}"
: "${GIT_REMOTE_BRANCH:=${DEFAULT_GIT_REMOTE_BRANCH}}"
: "${NO_CHANGE_SLEEP:=${DEFAULT_NO_CHANGE_SLEEP}}"

# Test if we're set up correctly
if [[ "${GIT_REPO_LOCAL_PATH:-}" == "" \
   || "${CINODE_DESTINATION:-}" == "" \
   || "${CINODE_WRITER_INFO:-}" == "" \
]]; then
    cat 1>&2 <<EOF
Please setup the following environment variables:

    GIT_REPO_LOCAL_PATH - the path to a directory where blog's
        source repository is checked out
    CINODE_DESTINATION - destination where to upload the set
        of compiled blobs (URL)
    CINODE_WRITER_INFO - writer info for the root link

Following optional environment variables can be configured too:

    GIT_REMOTE_NAME - name of the remote,
        default: ${DEFAULT_GIT_REMOTE_NAME}
    GIT_REMOTE_BRANCH - name of the branch on remote
        to track, default: ${DEFAULT_GIT_REMOTE_BRANCH}
    NO_CHANGE_SLEEP - number of seconds to wait between
        consecutive checks for new changes in
        git repository, default: ${DEFAULT_NO_CHANGE_SLEEP}

EOF
    exit 1
fi

while true; do  # The main loop

    #
    # Chapter 1 - The watcher
    # =======================
    #

    cd "${GIT_REPO_LOCAL_PATH}"

    while true; do
        git fetch
        
        LOCAL_HASH="$( git rev-parse HEAD )"
        REMOTE_HASH="$( git rev-parse "${GIT_REMOTE_NAME}/${GIT_REMOTE_BRANCH}" )"

        if [[ "${LOCAL_HASH}" != "${REMOTE_HASH}" ]]; then
            break
        fi

        echo "No change detected, git hash: ${LOCAL_HASH}"
        sleep "${NO_CHANGE_SLEEP}"
    done

    #
    # Chapter 2 - Hugo's hat
    # ======================
    #

    echo "New change detected"
    echo "  local hash: ${LOCAL_HASH}"
    echo " remote hash: ${REMOTE_HASH}"

    # Ensure we've got a completely clean git repository
    git reset --hard
    git clean -fdx

    # Copy files from remote - this does not change the HEAD commit yet
    # because we want to move HEAD once everything succeeds
    git checkout "${GIT_REMOTE_NAME}/${GIT_REMOTE_BRANCH}" -- .
    git submodule update --recursive

    # Impersonate as hugo and build static html files
    hugo

    #
    # Chapter 3 - Data protector
    # ==========================
    #

    # Compile and upload Cinode datastore files
    mkdir datastore

    RUN_AS_UID="$(id -u)"
    RUN_AS_GID="$(id -g)"

    docker                                                          \
        run                                                         \
        --rm                                                        \
        --volume "$(pwd)/public:/cinode/source:ro"                  \
        --volume "$(pwd)/datastore:/cinode/destination"             \
        --read-only                                                 \
        --mount type=tmpfs,destination=/tmp                         \
        --user="${RUN_AS_UID}:${RUN_AS_GID}"                        \
        ghcr.io/cinode/static_datastore_builder:latest              \
            compile                                                 \
            --source /cinode/source                                 \
            --destination /cinode/destination                       \
            --raw-filesystem                                        \
            --writer-info "${CINODE_WRITER_INFO}"

    #
    # Chapter 4 - Announcement
    # ========================
    #

    # Upload results
    find datastore -type f -printf "%P\n" | while read fl; do
        echo -n "Uploading $fl ... "
        curl                                                        \
            --silent                                                \
            --fail                                                  \
            --show-error                                            \
            --request PUT                                           \
            --header "Content-Type: application/octet-stream"       \
            --data-binary "@datastore/$fl"                          \
            --url "${CINODE_DESTINATION}${fl}"
    done
    echo "Uploading finished !!!"

    # Finally move the local HEAD confirming that this version of blog
    # was compile and announced correctly
    git reset --hard "${GIT_REMOTE_NAME}/${GIT_REMOTE_BRANCH}"

done    # And back to Chapter 1

What Dave is doing most of the time is watching the blog’s source code git repository. If nothing changes, he’ll just keep watching and will be very good at this task. But whenever something new appears on the git horizon, he won’t hesitate to act and move to the chapter 2 of his play.

In chapter 2 he brings back hugo from his memory and by impersonating as hugo in his play he assembles the blog into static html form. Once this is done, hugo’s hat is put back into Dave’s preserved memory banks and Dave moves to chapter 3. In this chapter he fires off docker with the instruction to encrypt the dataset and protect its precious content to finally move to chapter 4 where the page is sent out to the world… to a distant server beyond seven mountains and seven seas… or wherever the Cinode server currently runs (which is currently just few centimeters away in another network node 😉).

It’s Dave’s job to announce new blog posts to the world!

So… how’s it going?

Everything went smoothly at the beginning. But then something bad happened. Dave stopped talking - did not want to respond to pings, not to mention some deeper conversations through SSH. He was frequently absent from the WiFi network. It looked like he felt offended by something.

But the truth was much more serious - Dave was sick.

First I thought that it was because of the old damage that destroyed some of Raspberry Pi’s IO ports (which I briefly mentioned in the previous post). But it turned out that while mounting him to the holder I had damaged his WiFi antenna.

So he was transported to the “hospital”, went through some serious “surgery” and is currently recovering and getting into shape after WiFi antenna transplantation. He can’t wait to do his job now.

The whole story is way too long to put all the details into this post. But let me assure you that Dave is back in his cluster spot now. He even wanted me to share his picture, so here it is, CinoDave ready to rock-and-roll:

Screenplay

Is this the end of Dave’s adventures? Definitely not. Let’s take a look at his screenplay - that shell script can be improved a lot.

First major upgrade that will come in some day: instead of compiling the datastore to local files we could just upload it directly to the destination Cinode server. There’s no need to do this in two steps. That will be especially important in environments with constrained disk space.

But there’s much more. The script is just a file - a blob of data. Currently it’s just copied into Dave’s SD card. But since it’s a blob of data, why not put it into the Cinode dataset and let Dave fetch it from there? We can reason in the same way about the blog’s source code, hugo binary, even the Cinode data compiler. All those things are just data blobs forming a filesystem that can be stored in Cinode:

With this approach Dave would be working only within his workspace filesystem and everything, including the output folder would be located as sub-entries of the root folder. Once such pipeline is implemented, all Dave would need is:

• Entrypoint to the workspace filesystem - most of the things here are only read by Dave and never written - someone else would be responsible for preparing this dataset and Dave would only watch for any changes in the workspace filesystem (such as updated hugo binary or new blog post content) that would trigger execution of the builder script
• Writer info for the output folder - the output folder would be kept behind a dynamic link that Dave knows how to update, such update (because of dynamic link) can be done independently from the root filesystem. Dave would simply put the final html files to the output folder and that would (with enough tooling of course) update the link and all the data behind it.

Notice that I didn’t have to mention where the data comes from or how it will be uploaded to the network. That’s one of the goals I’d like to reach in this project - location erasure. Dave doesn’t have to know where he takes the input dataset from, for him it is important that the data is located behind a dynamic link that he trusts. If anything is published there, he will trust it because dynamic link is equivalent to digital data signature.

Same with the output. Dave has to know writer info - equivalent to a private key - and once the dataset is compiled and published, the dynamic link of that output data will be updated which would be equivalent to signing the data with the private key. Some other node in the network will know the dynamic link of the output and will trust that it was generated by Dave or any other builder out there that knows the corresponding writer info.

Speaking of the blog readers - do they have to know where Dave is located? It’s not necessary as long as they can access binary blobs with the content. This is in fact the location erasure property for Dave. He could be literally anywhere - even on the Moon. As long as the data Dave produces can reach the reader, everything would work.

Let’s extend it even more - to blog readers it also does not matter who created the data as long as it’s behind dynamic link - that’s another property - identity erasure. We could exploit this by having more than one builder - Dave and maybe Bob. Both could be doing the same work - take source information, compile the final html files and publish the result. Those would share the writer info and would rush to publish the data. Of course they will overwrite their work but assuming that they would be executing the same script on the same input, the output should be consistent. Forward progress rules would ensure the conflict is always properly resolved. For the blog reader it doesn’t matter who did the compilation. But if Dave was sick again, Bob would keep working, achieving failure tolerance of the whole process.

Whoa, a lot of work ahead to get to such a state… but it’s doable 😉.