Friends

He never felt guilty for how he looked. He was always like that. Nobody really knows what happened, maybe it was a wrong scale factor on the horizontal axis that made his belly so large, or maybe the unrestrained thirst for the information… Now since he joined the small but cozy rack shelf he’s been called The Big Guy over and over again. Yes, that’s how Cinodave likes to call him, right before exploding with a spasm of a contagious laugh.

Now they are two best friends that do not waste the time and chit-chat whenever possible.

  • Hey yo, Big Guy!
  • Huh?
  • Why your backpack has this big cable?
  • Dunno, I don’t hear anything from you if I plug it off, guess it’s for my connection… ?
  • Uuuuuu, a Bigga Cablaa for the Biggaa Guy, hahahaha.
  • Hahahaha, and you? You don’t have a Bigga Cablaa.
  • Naaa, but I can hear you without a Bigga Cablaa, no need one for me.
  • You poor Dave, maybe I will give you my Cablaa so you hear better?
  • No no no, no need for that Cablaa for me. I’m perfect without Cablaa. But hear this: you take the Cablaa, do bungee jumpaaa and go outside the Rack?
  • Oooo, go outside, maybe we’ll chase some boar… and eat it, I want to eat some boar.
  • Uuuu, like rodeo? Me too, me too.
  • So you need my Bigga Cablaa to go outside?
  • Aaaaaa, ^%#!@#%&$#$.
  • Hahahaha, calm down my friend, calm down, we will find a way to visit other places…

And they blink their LEDs like that all day long.

Insatiable hunger

The other day he was traveling through the vast oceans of the internet giggling every few minutes:

  • Buhahaha, these Redditors are crazy! Just like Romans!

Nobody really knows what he was reading back then, we could only guess as all access logs were cleared a long time ago. But for sure he was better and better over time. It was indeed his daily hobby. Over time he became so good at this task that he became the Archiver. That way he could fulfill, or at least try to fulfill, his thirst for data.

But there was a single thing that he was really interested in…

Cartographer

Cinodave once asked me:

  • BYO, can we talk?
  • Sure, what’s up Dave?
  • It’s the Big Guy, the new one…
  • Something wrong with him?
  • He is not happy, looks like he can’t find a way…
  • Hmm, what’s the issue?
  • He wants to know the world, travel, find the boar!
  • Then he will need some maps!

And that’s how I started working on a prototype to bring high quality maps to the world of Cinode.

I first did some research starting with a few failed attempts. But I finally got an idea on how this should look like.

Open maps - Cinode premiere

If we’re talking about good quality maps then the choice is obvious - OpenStreetMap. It’s a great source of data, a collaborative effort of thousands of people. That’s one of those creations where we, as a community, can create something of the best possible quality out there.

But how to get those into Cinode?

I didn’t want to abuse OpenStreetMap servers and just download data from them directly. That’s why I started looking for a way to generate map data locally.

In OpenStreetMap we can store data in two ways. First one is a vector format - where the server stores data as shapes - points, lines, polygons and their properties. The client library - e.g. a javascript code in the browser is then responsible for displaying such data in a visually pleasing form. Alternative is to use pre-rendered tiles in some image format such as png. Those tiles are then displayed in the browser and the javascript code manages them and ensures we don’t see any gaps, everything is aligned nicely, zooms in smoothly etc.

Initially I wanted to go with the vector format - not only the quality of the result is much better, but also the dataset is much smaller. But I couldn’t quickly find a solution that would not require some license or that would be trivial to implement in the amount of time I could spend on the prototype. Going with vectors is definitely possible - that thing I’m sure. But I had so many failed attempts that I decided to go for simple png files. Vectors will come later. Let’s not delay our map prototype any further.

Switch2OSM

Switch2OSM is a great page explaining how to create OpenStreetMap-based maps on one’s web page. It has some interesting tutorials on how to create the tile server too.

For the source of my map I’ve chosen a Poland country map from GeoFabrik. Why Poland? Well, that’s currently where the majority of Cinode and all things related are being created. That dataset is also still relatively small - I can start small and expand later. Of course the plan is to have a very detailed maps server covering the whole globe to a great detail but that task will require some significant resources… so maybe one day.

Following the tutorial on how to setup a docker-based tile server I did set up a local docker container with tile server and grabbed all the needed tiles from it storing them as local png files:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

# Create a docker volume for tile server data
docker volume create osm-data

# Import map data from the downloaded pbf file, this took some time
docker run                                                    \
    -v "$(pwd)/poland-latest.osm.pbf:/data/region.osm.pbf:ro" \
    -v osm-data:/data/database/                               \
    overv/openstreetmap-tile-server                           \
    import

# Run a local tile server,
# Note: the --shm-size option was needed since postgresql complained
#       about the disk space
docker run                              \
    -p 8080:80                          \
    -v osm-data:/data/database/         \
    -d                                  \
    --shm-size  512MB                   \
    overv/openstreetmap-tile-server     \
    run

# Download all the tiles up to given depth level, this really takes time
BASE_URL="http://localhost:8080/tile"
for zoom in {0..9}; do
    for x in $(seq 0 $((2 ** $zoom - 1)) ); do
        mkdir -p "page/tiles/${zoom}/$x"
        for y in $(seq 0 $((2 ** $zoom - 1)) ); do
            echo "tiles/${zoom}/${x}/${y}.png"
            curl -s "$BASE_URL/${zoom}/${x}/${y}.png" -o "page/tiles/${zoom}/${x}/${y}.png"
        done
    done
done

Viewer

Now since I had the tiles there was a question of how to display those. The tileserver that I used to grab png files from was very helpful here. It came with a built-in viewer page that uses the Leaflet library to display the map in HTML browsers. I looked up the built-in index page and made few adjustments:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

<!DOCTYPE html>
<html>

<head>
    <title>Cinode maps</title>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link rel="stylesheet" href="leaflet/leaflet.css" />
    <script src="leaflet/leaflet.js"></script>
    <style>
        html,
        body,
        #map {
            width: 100%;
            height: 100%;
            margin: 0;
            padding: 0;
        }
    </style>
</head>

<body>
    <div id="map"></div>
    <script>
        var map = L.map('map').setView([0, 0], 3);
        L.tileLayer('/tiles/{z}/{x}/{y}.png', {
            maxZoom: 9,
            attribution: 'Map data &copy; <a href="https://www.openstreetmap.org/">OpenStreetMap</a> contributors, <a href="https://creativecommons.org/licenses/by-sa/2.0/">CC-BY-SA</a>',
            id: 'base'
        }).addTo(map);
    </script>
</body>

</html>

With such a loader code, I was able to assemble everything into a main page directory, all using static files:

  • tiles folder - contains pre-rendered png tiles
  • leaflet folder - contains the leaflet library and all necessary resources
  • index.html file - the launcher for the map

Compilation

After checking that everything works locally (I love the python3 -m http.server one-liner for such tests) the map was ready to be compiled into Cinode datastore:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

mkdir datastore

RUN_AS_UID="$(id -u)"
RUN_AS_GID="$(id -g)"

docker                                                          \
    run                                                         \
    --rm                                                        \
    --volume "$(pwd)/page:/cinode/source:ro"                    \
    --volume "$(pwd)/datastore:/cinode/destination"             \
    --read-only                                                 \
    --mount type=tmpfs,destination=/tmp                         \
    --user="${RUN_AS_UID}:${RUN_AS_GID}"                        \
    ghcr.io/cinode/static_datastore_builder:latest              \
        compile                                                 \
        --source /cinode/source                                 \
        --destination /cinode/destination                       \
        --raw-filesystem                                        \
        --writer-info "${CINODE_WRITER_INFO}"

I was surprised to see how quickly the compiled dataset was created, even up to a point that I thought that maybe there’s some bug and not all data was compiled. But the docker command finished without an error with a message indicating successful compilation:

1
2
3
4

{
  "entrypoint": "9g4L3oUEuF3ozdfnLGKLNBXNoMKrZyzfZwwHVrEeu6WNRYp873cZDs2jYwUh1VtLmz49dyjfWE21U9ZBAKgECnnWcMuEMGUW5M",
  "result": "OK"
}

Roughly few minutes compared to around 2 hours to build png tiles - not that bad. Let’s see how it works then:

1
2
3
4
5
6
7
8

docker run \
    --rm \
    --interactive \
    --publish 8080:8080 \
    --volume "$(pwd)/datastore:/encrypted" \
    --env CINODE_ENTRYPOINT=9g4L3oUEuF3ozdfnLGKLNBXNoMKrZyzfZwwHVrEeu6WNRYp873cZDs2jYwUh1VtLmz49dyjfWE21U9ZBAKgECnnWcMuEMGUW5M \
    --env CINODE_MAIN_DATASTORE="file-raw:///encrypted" \
    ghcr.io/cinode/web_proxy

And now! For the first time ever! Here are Cinode-powered maps:

Cinode maps

Release

Now that I can run the map locally, let’s upload it to our data node .

I’ve reused the bash script that was used by Cinodave, but had to add a few tweaks. The dataset consists now of over 30000 files, uploading them one-by-one would take a long time if we were to run curl commands sequentially. I’ve added a really hacky way (seriously, don’t try this at home, or work, or wherever) to parallelize uploads:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

find datastore -type f -printf "%P\n" | while read fl; do
    # The while loop will retry uploading blob until we succeed to overcome
    # network hiccups
    while ! curl                                                \
        --silent                                                \
        --fail                                                  \
        --show-error                                            \
        --request PUT                                           \
        --header "Content-Type: application/octet-stream"       \
        --data-binary "@datastore/$fl"                          \
        --url "${CINODE_DESTINATION}${fl}"; do
            echo "Retrying $fl"
            sleep 1
    done &
    # Note the `&` above - that will spawn the `while` loop as a separate subprocess
    # in the background so we won't wait for it to complete. The sleep below will do
    # a bit of delay before we fire off the curl instance for the next blob.
    sleep 0.02
done

The main hack is to spawn curl commands in parallel with a given frequency assuming that those will finish in reasonable time. It could easily explode the machine if too many curl processes would accumulate. To see how it works I just spawned a background counter of curl processes:

1

watch 'pgrep curl | wc -l'

The whole thing worked really well, even in a bit unreliable WiFi environment.

And since the dataset is uploaded, you can see how it works yourself by running a local Cinode Web Proxy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# A dynamic link entrypoint that will hold the most recent version of the map viewer
ENTRYPOINT="9g4L3oUEuF3ozdfnLGKLNBXNoMKrZyzfZwwHVrEeu6WNRYp873cZDs2jYwUh1VtLmz49dyjfWE21U9ZBAKgECnnWcMuEMGUW5M"

# Fetch data from the global datastore
DATASTORE="https://datastore.cinodenet.org/"

# Let's finally run the proxy
docker run \
    --rm     \
    --interactive     \
    --publish 8080:8080     \
    --env CINODE_ENTRYPOINT="${ENTRYPOINT}" \
    --env CINODE_ADDITIONAL_DATASTORE_1="${DATASTORE}" \
    ghcr.io/cinode/web_proxy

Well, that prototype revealed some issues with my proxy code, most likely related to HTTP request cancellation. Make sure to restart it when tiles stop showing up 😉.

Let’s meet our Big Guy

Our prototype can now go to our Big Guy. I will still need to do some work to adjust it to his abilities, but at least we’ve proven that it’s possible 😁.

But I didn’t introduce the Big Guy yet, did I? So everybody, this is CinObelix:

CinObelix

That’s a picture of our main character in his early days while still being worked on including some temporary cables attached.

If you take a closer look, you may deduce that what he’s wearing on his back is actually our well-known LaFrite board. That board does not have any built-in WiFi - that’s why CinObelix needs a normal ethernet cable plugged in - this is the Bigga Cabla that CinoDave was talking about. CinObelix needs it plugged in to be able to connect to the external world, including talking to CinoDave.

With that kind of connection, CinObelix will be responsible for periodic refresh of important datasets in Cinode, including maps that I referred to today. I still have to figure out how to do this on the LaFrite scale - I’m not even sure if that board would be able to handle the tile server so most likely he will be connecting to an already existing server and uploading tiles from it to the datastore. Also let’s make it clear - CinObelix won’t be able to handle all the essential data out there that I would love to have in Cinode, that’s a task for a small cluster. But we have a good starting point boosting the power of the current tiny Cinode network.

Cinode deduplication

Let me quickly show you one very interesting thing:

1
2
3

$ du -sh page/ datastore/
1,4G    page/
179M    datastore/

Here I’m showing the on-disk size of data for our maps. The page folder consists of raw png tiles and a small html viewer page. The datastore contains the same data compiled into Cinode encrypted format.

Why is there almost a 10x difference in size? Did we skip some data while compiling the Cinode dataset? Similar disproportion can be seen even with the number of files in those folders:

1
2
3
4

$ find page/ -type f | wc -l
349533
$ find datastore/ -type f | wc -l
33733

Turns out that Cinode automatically deduplicated identical png tiles into a single Cinode blob. Well, not only tiles, also completely identical directories too. But how did it happen?

That’s because Cinode relies on the Content-Addressable-Storage. Names of blobs are directly generated from the blob content - and nothing else - so identical tiles will just produce a single blob with the same content and Blob Name.

Same happens with identical directory deduplication - directory in Cinode is just a serialized list of references to other blobs. If we have identical directories, those will have identical byte representation after serialization. Deduplication jumps in here too.

To confirm that everything is OK I also analyzed the source png files with some bash magic😉:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

$ find page/tiles/ -type f   \
| xargs sha256sum            \
| awk '{print $1}'           \
| sort                       \
| uniq -c                    \
| sort -n -k1                \
| tail -n 10
      1 fffa855b32a24108e839fd25d99336d865858ca495713bfe1724c50185247167
      1 fffb155442913ddd5cf53a6e74cc3eb70259480b9f69a68e94aca297546caa33
      1 fffb16e206cf22be60717a716d44c2f586e15dcaa0523589f119240da2ecb868
      1 fffb36dbf51569d78250fa2d69a9842a0ff6d9473082e0e24e27bf4f6663d91d
      1 fffd4694f339215fb044bb0936d817def2fee723cb4b7f3693e9049ba8d570c5
      1 fffed8cb46ac263002db2a3774f8c72b233365cd81dc71dee1eefb6b0578bb31
      1 ffff0958a650ab97ceaf7f06f9fe1521b484ea2a4b5352f451eb7c856294aabe
  57989 47583fc23a668456dca858659bfb168a6aaa4b802372c7bc112a0b56f473c376
  66787 e4a91278c8a59cbb6737268bea08c0155148035cd15bb14258f70a08c0ea223f
 192065 1c77b348765c66299f86929a49254e3e6d7893d3930322ff7879dda6d9071899

What we can see here is that some three images appear in a total of 57989+66787+192065=316841 tiles. And here they are:

Tile 4758 - Ice

Tile e4a9 - Land

Tile 1c77 - Water

Those tiles are fully filled with water, ice or land colors covering large areas of the whole map.

And how many png files in total do we have?

1
2

 find page/tiles/ -type f | wc -l
349525

So roughly 90.6% of all png files share the same byte-byte identical content 🤯! Of course if we were to use complete maps with the majority of the land filled with details, there would be less opportunity for deduplication. But still, oceans cover around 70% of earth’s surface - and that will be just blue. We should be fine 😉.

Friends on the stage

Let me finish this post with a picture of two close friends living on a small shelf in a rack. They tend to talk all night sharing their passions and future visions on how Cinode could look…

Friends