Today, I’d like to focus on Cinode’s filesystem layer. This layer transforms binary data blobs into a manageable format, greatly simplifying data interaction from an application perspective. Taking this blog as an example: it is served through a dedicated proxy exposing a complex blob-based dataset in a nice structure of directories and files that every browser understands.

Until now, the toolbox for building filesystem data in Cinode has been somewhat limited. A prime example is the Static Datastore Builder project, designed to upload the contents of a local directory directly into Cinode. My experience in developing Cinode maps revealed certain limitations of this tool. Firstly, it does not offer any way to handle dynamic links. Secondly, it requires the uploaded dataset to be stored on a local filesystem, which is impractical for huge datasets that could be computed on the fly like pre-rendered map tiles. Let’s delve into the Cinode maps case to understand this better.

CinodeFS abstract by ChatGPT4

Cinode Maps Directory Structure

Utilizing CinodeFS, Cinode maps organize data in a simple directory structure. You can see a visual representation of how the directories and files are arranged in the image below.

Cinode Maps Directory Structure

The structure includes:

  • Root Directory: Here, the index.html file resides, acting as the entry point to the Cinode maps application.
  • Leaflet Directory: Contains the Leaflet library files essential for map functionality, referenced via a dynamic link for ease of updates.
  • Tiles Directory: Also dynamically linked, this directory holds the pre-rendered map tiles.

Instead of directly referencing Leaflet and map tiles, these two sub-folders are attached to Cinode maps through dynamic links. These links offer several operational advantages:

  • Independent Updates: Updates to the Leaflet library or map tiles can occur independently from the main Cinode maps content, allowing for more streamlined maintenance. These updates can be executed through a dedicated and isolated process, enhancing security by limiting access to the application’s more sensitive components.
  • Efficient Resource Sharing: Multiple projects can share the same Leaflet instance and link to the same map tiles library. As a result, any updates are automatically propagate to all connected projects.
  • Permission Delegation: The authority to update a given dataset can be delegated. For instance, updates to the Leaflet library are currently managed by an updater within my small cluster. In the future, this responsibility could transition to the library’s own maintainers.

Enhancing the Uploader for CinodeFS

The quest for a better uploader for Cinode maps began with enhancements to the toolkit used for creating CinodeFS structures. Before my recent changes, two methods were available to upload content to Cinode: using the Static Datastore Builder or manually writing custom code to set up protobuf structures representing CinodeFS data.

The first version of CinObelix's script simply created a single large directory containing everything needed: a launcher script, the Leaflet library, and map tiles. It uploaded these using the Static Datastore Builder tool. Later, I refined this script with hand-crafted Go code that converted the leaflet and tiles folders into dynamic links by operating directly on the protobuf data. Currently, the CinObelix script only updates the tiles folder, leaving the rest untouched. Now, it’s time to make the CinObelix logic more robust.

Acknowledging the limitations of both methods, my initial focus was on developing a more user-friendly library for interacting with CinodeFS. The development process was neither quick nor straightforward. However, after several iterations, we now have a functional and reliable library. This library is still evolving, with important features under development, so stay tuned for further enhancements.

The CinodeFS Toolkit

The core toolkit for working with CinodeFS is accessible through the CinodeFS package on GitHub. The centerpiece is the FS interface, comprising various methods for files and directories management within Cinode:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

type FS interface {
    SetEntryFile(ctx context.Context, path []string, data io.Reader, opts ...EntrypointOption) (*Entrypoint, error)
    CreateFileEntrypoint(ctx context.Context, data io.Reader, opts ...EntrypointOption) (*Entrypoint, error)
    SetEntry(ctx context.Context, path []string, ep *Entrypoint) error
    ResetDir(ctx context.Context, path []string) error
    Flush(ctx context.Context) error
    FindEntry(ctx context.Context, path []string) (*Entrypoint, error)
    DeleteEntry(ctx context.Context, path []string) error
    InjectDynamicLink(ctx context.Context, path []string) (*WriterInfo, error)
    OpenEntryData(ctx context.Context, path []string) (io.ReadCloser, error)
    OpenEntrypointData(ctx context.Context, ep *Entrypoint) (io.ReadCloser, error)
    RootEntrypoint() (*Entrypoint, error)
    EntrypointWriterInfo(ctx context.Context, ep *Entrypoint) (*WriterInfo, error)
    RootWriterInfo(ctx context.Context) (*WriterInfo, error)
}

The FS interface optimizes updates to the filesystem. It temporarily stores changes to the directory structure in memory, avoiding immediate uploads to the datastore. This approach is beneficial as it prevents excessive and potentially unnecessary write operations to the datastore.

Once all modifications are finalized, the Flush method must be called to execute the updates. This method ensures that all the changes made are efficiently committed to the datastore.

Static files, due to their often large size, are uploaded directly to the datastore without interim memory storage. While this method might result in some orphaned data in the datastore if the update process is interrupted, it’s a reasonable trade-off to maintain efficient memory usage. Upon executing a Flush, these files are seamlessly integrated into the updated CinodeFS structure, ensuring data consistency.

To effectively reuse existing datasets such as the Leaflet library or map tiles, we need to identify the entrypoints for their dynamic links. Currently, Cinode is a cozy little project with a select user base - and by select, I mean it’s pretty much just yours truly for now! 😉 As such, we don’t have a massive central library repository like PyPI or npm, and all the links and entrypoints are managed by me.

In the spirit of simplifying things and streamlining the onboarding process, I’ve added a small sub-page on this blog: curated list of globally available Cinode datasets that contains all this information:

Cinode links page screenshot

This is very much an early version, so please bear with me and pardon the ‘work-in-progress’ feel of the experience 😅.

Dig into CinodeFS Structures

Good engineering goes hand in hand with good tools. To delve into the data structures of CinodeFS, I’ve developed the CinodeFS Analyzer, a straightforward web application that allows for the exploration of a CinodeFS filesystem’s raw structure. You can conveniently run it locally using Docker to examine your own datasets:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

$ docker run -p 8080:8080 ghcr.io/cinode/cinodefs-analyzer:latest --help
Web server to analyze cinodefs entries.

Usage:
  web_analyzer [flags]

Flags:
  -d, --datastore string    Datastore address (default "https://datastore.cinodenet.org/")
  -e, --entrypoint string   Starting entrypoint (default "9g1R5xUqhAxfHnVBPAyBY9NXNe1dzKK949czZtT9THSMesRk3tKSRTWh2bsaKp4ivFVYyZX3vXMdE74XiAw9ckEWQoKLRouJnn")
  -h, --help                help for web_analyzer
  -p, --port int            Http listen port (default 8080)

This command spins up a local instance of the CinodeFS Analyzer, ready for you to dissect and understand your filesystems directly from your browser.

Alternatively, you can use my public CinodeFS viewer instance which is available for the data I’ve published on Cinode.

Here’s a short preview of how it looks:

CinodeFS viewer

Be sure to visit the curated list of globally available Cinode datasets on this blog. You’ll notice that each link includes a reference to this CinodeFS viewer, providing an even simpler way to examine the structure of a specific CinodeFS link. Enjoy the discovery process! 🎉