I'll go this way, Hugo that way

Ok ok, terrible pun. But it is what first came to mind. So here we are.

TL;DR:

#

This is a long post. So settle in.

The “Why”

#

So much has changed since I started my original blog site using Grav.

Obsidian

#

There is “life before Obsidian”, and then there is “life after Obsidian.” This is the single biggest change.

I have now been using Obsidian as my “second brain” for awhile now. It is where I jot down all my notes/etc. and has become my own “single source of truth” so to speak. In fact, I moved all the various notes/info I kept into Obsidian that I had in places like Joplin, Apple Notes, various text files and other places on my computer.

Obsidian, like Grav, works with simple text files in Markdown format. An Obsidian vault is nothing more than a directory of subdirectories and Markdown formatted text files (with filenames ending in .md). This makes it eminently searchable with macOS Spotlight or grep, easy to backup/sync, and easy to manipulate. The Obsidian software manages all this, providing a nice editing environment and handling such things as links between files, and if you change the filename of, say, an image, Obsidian makes sure to update any and all links to that file in any other files within the vault. It also has an enormous ecosystem of plugins that let you extend what all you can do.

Even more important, should Obsidian itself ever go away, I won’t be locked out of my content (as I was with MS OneNote v1). I can always fall back on the simplest of tools like vim, TextEdit, or Notepad. And for me this is key.

No feedback/comment system

#

I never added feedback/comments to my original blog site, since Grav has no good options of its own for this. This meant I would have to hook in another, 3rd party piece of software to do so.¹

This made the transition easier since I did not have to worry about bringing over any comments/etc.

“Friction level”

#

Over time, my original workflow for writing blog posts was not really working for me. The “friction level” was still too high. More on this below.

Online vs. Local Editing

#

In my original post on creating this blog site (see that for more history), I wrote that “I wanted a solution that was available on the Web, both for viewing and editing.” This came from my time using things like DokuWiki. I thought that having something at my fingertips, where all I needed was a web browser to get out my thoughts, was the best approach. This meant I could be at almost any computer, and as long as it was online and had a browser, I was good to go.

But since then, I have shifted to dumping my thoughts into Obsidian. And the reality is that I only really do my writing when I am at one of my own computers. Since setting up the blog, I have never leveraged this “online” aspect.

In fact, if anything, this online aspect of Grav hampered me more than anything. If I got an idea for something I wanted to write about, but I wasn’t at a computer or I was somewhere without Internet access (e.g., in a car, on a plane), I had to wait until I was. Or I had to write it down somewhere else and then later remember to transfer that to a draft page in Grav. This hampered my flow.

Also, I have my Obsidian vault synchronized between my systems using iCloud and Syncthing. So now I can get out my thoughts quickly at whatever computer I am sitting at, or even on my iPhone. I simply create a note and go. If I get an idea while out and about, I simply pull out my phone, open Obsidian, and write it down quickly. Then it is there when I get back to a computer to flesh things out. So that workflow has changed a bit.

FYI: All the other original points for what I wanted in a blog setup still stand, however. And my new setup supports them as well. I wanted a solution

• where I could write/document things as quickly/simply as possible
• where I would always have control/ownership over the content
• that was in a format I could, if need be, migrate elsewhere should the need arise ²

Go (Golang)

#

Over the past several years, I have been working more with Golang. One key aspect of that language which I really like is the ability to generate tools that are single binaries with no external dependencies. I cannot overstate how beneficial this is vs. other solutions written in languages like PHP, Python, Perl, etc.. Those languages all bring with them a dependency chain that can be quite frustrating if you use multiple systems and/or distribute your code.

This, in turn, has had me looking at all things written in Go. That includes static site generators like Hugo and GitHub alternatives like Gitea.

Hugo

#

Speaking of Hugo, I have noticed a serious uptick in sites using Hugo among more technical folks in recent years. That made me look at it more closely.

Static site generators such as

• Hugo (written in Go)
• Jekyll (written in Ruby)
• Hyde (written in Python)
• Next.js (written in Javascript)

and others typically take some simple formatted content (e.g., Markdown) and from that, combined with some theme, generate the necessary HTML, CSS, and JavaScript for a website.

The upside of this is that all the “heavy lifting” is done up front during the creation process. This means the workload on the actual server is minimal. Static sites are just that: static files that can be served quickly by any common webserver. It does not get simpler than this.

Compare this to sites using something like Grav, which requires a webserver that supports PHP. Every page load requires going through the PHP interpreter, putting more CPU load on the server. While creating the content in Markdown format is easy enough, and software like Grav will try to cache the conversion of pages from Markdown to HTML on the first web request so that future requests can be served without having to re-convert the content, it still places more strain on the server than any static site would. (Other software like WordPress require even more in the form of an RDBMS (Relational Database Management System) like MySQL/MariaDB. This adds to the CPU/RAM/storage requirements of any such website.)

The downside to sites using tools like Hugo is that the sites are “static”, lacking interactivity that any server-side scripting language such as PHP or Python can provide. This means no ability to provide a “back and forth” between the client and the server. It’s purely “Here’s the content. Have a nice day.” The only interactivity possible is what can be done using Javascript within the client-side browser.

Cloud Server Limitations

#

My blog site is hosted on a very small, single vCore/1GB RAM cloud server. Yeah, less than most Raspberry Pis. So I have always been mindful of the system requirements of any software I run there.

And over time I have noticed the responsiveness of my Grav site slowing down a little. I suspect this is more due to the OS overhead with each successive Ubuntu LTS release, as that is the base OS on the box. But still.

This had me contemplating whether I should move to a more static site format to keep system requirements low. Or so that I could possibly even host the site on GitHub or Cloudflare Pages. My website does not really leverage the server-side scripting much anyway. If I built my site with something like Hugo, I could host it just about anywhere. This would give me more freedom/flexibility should that need arise.

The one server-side feature, if any, that I used a few times within Grav was to set a blog post to publish at a later time. So I will lose that ability. But it is a small price to pay really. And considering that this feature then required Grav to continually run some code on the server to check if it was time to publish a post, I would say, if anything, that this will ease resource usage even more on the server.

(UPDATE: Ok, after playing a bit with my new setup, I did realize a few other bits that I don’t have now. I did not really notice, though, as I haven’t spent much time with them, so they aren’t really a loss to me per se.

For example, since Grav is, after all, a server-side based CMS, it has a dashboard that not only shows you whether your copy of Grav and/or any plugins and themes are up-to-date or require updating, but also provides tools like automatic backups and reporting, not to mention–with plugins like Page Stats–that you can get insight into who all is visiting your site by country/browser/platform, etc. I had other plugins for doing things like syncing my site content over to GitHub, or reindexing the search info for my pages.

Mind you, the raw data to mine for page stat info/etc. is all right in the NGINX logs. So if I really wanted that, I’m sure I could add a separate tool to do so such as GoAccess. But still, I wanted to be clear that each solution does bring with it both benefits and tradeoffs. The tradeoff here is that if I want page stats, I will have to find another tool to do so now.)

So my end goal was to find a way to simplify things even further than I had come up with using Grav. Ideally I wanted to be able to write my posts as Obsidian notes and then somehow automagically push them to my blog site. This way they were published but still completely at my fingertips for any searching/editing/etc.

Old Workflow

#

In the end, after using it for awhile, my workflow with Grav was not quite what I had hoped for. It was not bad. It just was not as good as I would like it to be. It took too many steps. It had, as I tend to say, “too much friction” to it.

It went something like this:

1. Get idea for something to write

2. Check “Am I online?” I had to be online in order to do the rest of this. If not, either write this down somewhere else or try to remember it for later (yeah as if THAT ever worked). But basically stop at this point if not online.

3. Open browser if not open

4. Open a tab/window, point to the admin panel of my blog site, and log in

   This would be the tab/window where I created or selected a blog post and edited the content. Note that the Grav online editor does not show you the final version of your page. It tends to show the raw Markdown format, meaning images are just links, etc. Thus this requires…

5. Open another tab/window and point it to the public site

6. Switch back to the edit tab/window

7. Write/Edit page

8. For each page edited, make note of and copy the part of the URL to the page, then switch to the public site tab/window and manually copy/paste the relevant part of that URL into the address bar and reload. This lets you see what the page will actually look like when published.

   NOTE: Unlike WordPress, Grav does not offer you the ability to view your unpublished/draft pages as they would look once published. This means you have to actually publish your page just to see it in its final form. But if you do not want a page public on your site until you are ready, you have to do something like what I did.

   For example, create a draft folder within Grav in which you store your unfinished posts. Mark the posts as published (so Grav will route to them), but because there is no menu option on your public site to reach them in this draft folder, you would have to know the exact URL/slug (e.g., https://mysite/draft/slug-to-unfinished-post) in order to see the page. This lets you see the final rendered version before making it truly public.

9. Rinse/repeat the edit-page / page-reload steps, switching back/forth between tabs/windows until you are done

10. Once a post is done, move it to the public portion of the blog site using the admin panel. (The post was already marked “published”. It just was not in a location that Grav would then automagically build links to in your blog list, etc.)

This worked ok. And it may still be what I revert to if my current experiment fails. And in my original post I already explained my reasonings for choosing Grav when I did. But it is not as clean a workflow as I would like.³

New Workflow

#

So what has changed? Alright, so it took a bit of time to first learn the various bits and then to get the pieces in place. But over the break I made the switch. I will likely write up another post on my actual setup.

First, here is the new workflow:

1. Get idea for something to write
2. (optional) Open Terminal and run the command

blog

leaving this session running in the background. This is simply a shell script that

• rebuilds my Hugo site locally
• fires up hugo server in the root directory of my local Hugo site, and
• opens a browser tab pointing to http://localhost:1313

This allows me to see what my site looks like right on my computer. I don’t even have to be online!

3. Open Obsidian–if it’s not already–and navigate to the Blog site/blog folder at the root of my vault, and create/edit a blog post by doing the following: ⁴

   • create a folder with the slug name (e.g., ill-go-this-way-hugo-that-way) of the blog post I am creating
   • create a file inside this folder called index where the actual content will go
   • hit a hot-key I have setup to apply an Obsidian template that I created for blog posts which adds in the appropriate Hugo frontmatter (e.g., title, description, date, draft, tags)
   • drag/drop any images to be used in this post inside the created folder
   • edit the content of index, dragging images in as needed

4. Click a button in the Obsidian sidebar to update the local site.

   That is, while Obsidian itself can show me what the page will mostly look like, since it renders the Markdown as you type, if I really want to see what a blog post actually will look like on the website with the theme applied/etc.–and assuming that I did step #2 above and in my browser navigated to the post in my local site–I simply click a button I set up on my Obsidian sidebar. (More on this in a bit.) If I am on the blog post in my browser, it reloads the site and page automatically.

5. Rinse/repeat editing the page and (optionally) clicking the button to update the browser page

6. Finally, when I am ready to make my changes truly public (keep in mind steps 1-5 are purely local to the machine I am editing on), in the Terminal I then simply run

blogupload

and the entire site is pushed to my public server. blogupload is just another shell script that runs an rsync command to sync the local copy of my Hugo blog site to my actual cloud server. (If I ever change where I host my website, I simply modify this one script to point to the new host.)

But that is it. That is the entire workflow. And note some of those steps are optional. I could simply open Obsidian, even on my phone, and simply write up whatever is on my mind as it comes to me. Like the old Ragu commercials would say, “It’s in there.”

I could work on multiple blog posts, taking however much time I like for each. If I have several posts in progress, I can simply set the frontmatter value of draft to true (Obsidian makes this as simple as checking a box) for any posts that are not yet ready. And only when I do the final step does any of it go public.

And as mentioned, now my entire site is stored within my Obsidian vault, giving me full access to search it along with everything else that I write.

One ring to rule them all,
   one ring to find them,
One ring to bring them all
   and in the darkness bind them.
– J.R.R. Tolkien’s “The Hobbit”

Getting From Here to There

#

NOTE: Originally I had written up the “how” of putting together my new setup. But this post is already quite long. So I am only putting more general information here. Expect another post with the technical details.

There are many blog posts, tutorials, and YouTube videos on how to use each of Obsidian and Hugo. That is beyond the scope of this. And there are also posts, tutorials, and videos on how you can leverage Obsidian with Hugo. Here I just give a general overview.

Regarding Hugo itself, it is written in Go. So it comes down to a single binary file that you download to your computer and use via a terminal. You use this executable to do just about anything/everything Hugo-related. If not hugo, then likely you will be using git.

For example, hugo new site <sitename> will create the scaffolding for a new website. Then in many tutorials they will show how to edit the site configuration and how to find/download some Hugo theme, often using git. The key thing to note here is that you use hugo to build a local directory structure that holds everything needed to generate your website contents. There is a content folder that holds all the Markdown files which make up your site. In the end there is a folder called public at the root of this directory structure that holds the final rendered version of your website. All you need to do is transfer this content to whatever webserver hosts your site.

Now since Hugo is really a CLI tool, and its focus is on taking Markdown text files and rendering them as HTML files styled with CSS and Javascript using whatever theme you configured, it does not offer the “ease of use” that most non-technical folks would prefer. For those who want that, WordPress or even Grav are better choices by far.

But it is not surprising that the focus of most technical folks (e.g., programmers) who use Hugo tends to be on leveraging the very same tools they already use. This includes things like VS Code to edit the Hugo config files and Markdown pages, and using things like GitHub to store their Hugo site directories. They use GitHub repos not only to track the history/revisions of their site, but also use things like GitHub Actions to auto-trigger regenerating their sites with hugo any time they push a new commit.

While I could go this route as I do use VSCode, GitHub/GitLab (and even run my own Gitea server), this is NOT the kind of workflow I wanted. This kind of arrangement would mean the I had my information in two (2) places–Obsidian and wherever this Hugo-based content would be stored–which in turn would mean my information was “split”. This then would require me to search not one (1) but at least TWO (2) places any time I was looking for something. I wanted something simpler/cleaner. Ideally I wanted everything stored in one place, so it was ALL available to me with a single search/etc. And I wanted that place to be my Obsidian vault.

Now regarding the use of Obsidian with Hugo, some make use of Obsidian plugins, where different plugins work different ways. For example, Sync to Hugo lets you select specific notes to sync to your Hugo site. And Hugo Publish, which was the closest to what I was after, will automagically export ALL of the Obsidian notes that are set with some given tag located in your entire vault, converting Obsidian wikilinks to standard Markdown links and copying over referenced image files to your Hugo site’s static folder.

Other articles/tutorials/videos have you storing your blog site in its own, separate Obsidian vault, or having Obsidian open your Hugo site’s content directory as a vault.

Unfortunately, I simply found them all a bit wanting for my particular needs. Again, YMMV. So I kept tinkering until I found a workflow that suited me.

So how did I do this? For that, you will have to wait for a future post.

Benchmarking

#

Oh my. In the past several years, some things have clearly changed in my hosting environment since I did the ab⁵ testing as noted in my original post. All the request/second numbers are lower now, with numbers indicating more latency in general. But still, the difference is palpable.

Again, I attribute this to each update of Ubuntu LTS, likely combined with changes at my hosting provider that are outside my control. The point here is simply to say “Don’t compare my testing numbers from my original post with the numbers here, as that is not a fair comparison.” Instead, I ran my tests again on both my old Grav setup and my new Hugo setup so that the comparison would be fair.

When I did my ab testing from home to my cloud server during this migration,

Grav averaged 15 requests/second, taking ~6 seconds/request.
Hugo averaged 50-88 requests/second, taking 1-2 seconds/request!

These numbers get even more impressive if I run ab locally on the cloud server itself where the website is hosted, where the Grav numbers stay about the same, but the Hugo numbers go to ~245 requests/second, taking ~.4 seconds/request!!

This shows the kind of overhead that can be induced by server-side scripting based offerings. (And frankly, Grav only needs PHP itself. Imagine the overhead using an RDBMS-based solution like WordPress.)

Final Thoughts (as this is already too long)

#

So once I had all the pieces working, I flipped my production site from Grav to using Obsidian/Hugo. And I am typing this post in Obsidian, hitting the button in my left ribbon bar every so often to see the updated page in my browser. (To make the button stand out, I set the icon to a pinkish hexagon that looks a bit like the “H” in the Hugo logo). And this post is my second and (to date) longest post using this. So far, so good.

What I am already noticing is that it is much easier for me to start writing something, as I mostly live in Obsidian already. So it really is as simple as just creating a new Obsidian note. If I feel like it, I will bring up a Terminal window and run blog so that I can see what the final, formatted version will look like. But even without that, I can see for the most part how my post will look, since Obsidian lets you see the Markdown formatting as you type and even any images you drop into a post. This just “feels” much smoother/quicker.