Create an internal CLI

24 Aug, 2024

Assumed Audience: programmers who know their way around a terminal.

One useful way to capture and preserve institutional knowledge in a dev team is to grow a collection of useful snippets, scripts, or workflows. This is why almost every repo out there has some way to write and run tasks via Makefiles, bash scripts, or language-specific toolings like scripts in package.json for Javascript, or mix tasks for Elixir.

What about org-wide things, like installing useful tools, generating boilerplate code, or running complex AWS commands that no one remembers? Some companies like Slack and Shopify have their own internal CLI. The modern terminal Warp has a pretty sick feature for documenting and sharing workflows.

It's quite easy to get started with your very own CLI. Let's create one right now for our hypothetical company Acme Corporation.

Design

Here's a list of simple requirements:

Common entrypoint for all commands. All devs should trigger commands by doing acme <command> from anywhere, rather than having to navigate to a specific repo first.
Devs can easily contribute new commands. No need to learn a brand new language or complex syntax.
New versions are distributed easily. There should be a convenient way to contribute new commands. It should be easy to fetch new updates via acme update.
Commands should work across platforms. For example, acme download something command could use curl on Linux, and Invoke-WebRequest on Windows.
Commands should be discoverable. Calling acme list should enumerate the list of available commands and their short descriptions.

Just

Let's use just for this project. just is similar to make but designed to be a command runner. It is cross-platform and enables the possibility of running platform-specific commands. Lots of repos use it.

Some other possible tools we could use:

Slack's magic-cli: it's a fantastic starting point if your devs know Ruby.
make: Makefiles are simple, available out of the box on Linux and Mac machines (but can be installed on Windows machines too), and are very, very widely used.

Set up the project

Install just. Follow the instructions here.

Create a folder at ~/acme/cli and add the following justfile at the root:

default:
  just --list

# Show arch and os name
os-info:
  echo "Arch: {{arch()}}"
  echo "OS: {{os()}}"

The just documentation calls commands "recipes", so let's use that word from here on.

When just is invoked without a recipe, it runs the first recipe in the justfile. It is a common pattern to name the first recipe "default".

$ just
just --list
Available recipes:
    default
    os-info # Show arch and os name

Our default recipe runs just list, which is the in-built mechanism to enumerate all available recipes, alongside what we documented as comments.

Let's hide the default recipe in the list, since there's no point showing it.

Notice that running the recipe prints each command before it is executed. Let's suppress this output using a @ prefix. This is quite similar to how Makefiles work. We can add this prefix on each line we want to suppress, or add it to the recipe name to suppress the output for all lines.

[private]
@default:
  just --list

# Show arch and os name
@os-info:
  echo "Arch: {{arch()}}"
  echo "OS: {{os()}}"

Create the `acme` alias

We want to run these recipes using acme <command> instead of just <command>. Let's now add the acme alias to our .bashrc (or whatever rc file you use):

alias acme='just --justfile ~/acme/cli/justfile'

In the Forwarding Alias section of the README, just's creator said "I'm pretty sure that nobody actually uses this feature, but it's there". Hah, in your face, Casey!

Load our new alias using source ~/.bashrc or refresh the shell using exec bash.

Now, let's try it out:

$ acme
Available recipes:
    os-info # Show arch and os name

Alright, now we have our very own acme CLI. Roll credits!

Bonus exercise: try creating a setup recipe to do this automatically. Remember to cater for different shells and operating systems. Good luck!

Write new recipes

Simple recipes

Here's a simple but useful recipe for retrieving the current AWS IAM identity using awscli:

# AWS: retrieve the identity of the current user/role
@aws-id:
  aws sts get-caller-identity

Simplifying commands that no one remembers is probably the top use case of internal CLIs.

Note that we are making the assumption that awscli is reasonably cross-platform, so this recipe should work regardless of where we call it from.

Bonus exercise: create a ensure-aws recipe that detects the presence of awscli and install it automatically if it doesn't exist. Use it as a dependency for any recipes that use aws.

Platform-specific recipes

Snippets that involve tools like systemd are only relevant for Linux users, should they be exposed only if the dev is on a Linux machine.

# List systemd services
[linux]
@list-systemd-services:
  systemctl list-units --type=service

By tagging the recipe with a [linux] attribute, we can selectively enable a recipe only on Linux. I'm using a macbook now, let's see if it works:

$ acme os-info
Arch: aarch64
OS: macos

$ acme
Available recipes:
    aws-id  # AWS: retrieve the identity of the current user/role
    os-info # Show arch and os name
    update  # Update the Acme CLI

It's not in the list! But what if we try to run to run the recipe anyway?

$ acme list-systemd-services
error: Justfile does not contain recipe `list-systemd-services`.

So it's not just hidden, the recipe straight up does not exist. This is a great guardrail against devs who might not be aware that the recipe is not available on the respective platform. The error message can be better though..

Let's verify that it works on a Linux machine:

$ acme
Arch: x86_64
OS: linux

$ acme
Available recipes:
    aws-id                # AWS: retrieve the identity of the current user/role
    list-systemd-services # List systemd services
    os-info               # Show arch and os name
    update                # Update the Acme CLI

$ acme list-systemd-services
  UNIT                     LOAD   ACTIVE SUB     DESCRIPTION
  accounts-daemon.service  loaded active running Accounts Service
...

Cross-platform recipes

No one ever remembers how to get the size of a folder, so let's implement a acme get-folder-size <path>.

# Get the size of a folder
[linux]
[no-cd]
get-folder-size path:
  du -sh {{path}}

We add a [no-cd] attribute to the command for this recipe to run relatively to where the command is invoked. By default, acme will run with the working directory set to the directory that contains the justfile.

Windows doesn't have du, so we have to find another way to do it. We should also make it explicit the different shells we use for Linux and Windows:

set shell := ["bash", "-c"]
set windows-shell := ["powershell.exe", "-c"]

[private]
@default:
  just --list

...

# Get the size of a folder in MB
[windows]
[no-cd]
get-folder-size path:
  (Get-ChildItem "{{path}}" -Recurse -Force | Measure-Object -Property Length -Sum).Sum / 1MB

Now we have a acme get-folder-size for both Windows and Linux!

Scripts

We can embed entire scripts into our recipes! Simply start a recipe with a shebang (the #! part) and under the hood, just will save the content as a file and execute it.

This is useful if our workflow requires slightly more complex logic, like using control flows (if-else, loops), storing and manipulating variables, etc.

# Say hello world in sh
hello-world-sh:
  #!/usr/bin/env sh
  hello='Yo'
  echo "$hello from a shell script!"

This also means we can utilise programming languages with strong scripting abilities. Some things are easier to do in Python than in Bash.

# scale jpg image by 50%
[no-cd]
scale-jpg path:
  #!/usr/bin/env python3

  import PIL.Image
  image = PIL.Image.open("{{path}}")
  factor = 0.5
  image = image.resize((round(image.width * factor), round(image.height * factor)))
  image.save("{{path}}.s50.jpg")

Not all devs have Python on their machines, and even if they do, they may not have pillow installed. We can use nix to run scripts with dependencies included:

# scale jpg image by 50%
[no-cd]
scale-jpg path:
  #! /usr/bin/env nix-shell
  #! nix-shell -i python3 -p python3Packages.pillow

  import PIL.Image
  ...

Yes, I use nix btw

Rather than rolling our own distribution mechanism, let's just use git.

Acme Corp uses GitHub. Create a remote repository first, then turn what we have right now into a git repo and push it up.

$ git init
$ git commit -m "first commit"
$ git branch -M main
$ git remote add origin git@github.com:acme/cli.git
$ git push -u origin main

Now, anyone with access to this repo can contribute their changes by making a PR. Updating acme is also a simple matter of doing a git pull. Let's create a recipe to do that:

# Update the Acme CLI
@update:
  git fetch
  git checkout main

Note that acme runs from the working directory of the justfile by default, so this update recipe can be run anywhere.

Bonus exercise: create a mechanism for running acme update periodically. Use systemd or whatever you believe in. Bonus points for creating a acme setup-auto-update recipe!

Documentation

Adoption is critical to the success of an internal tool, and a good README that guides a new user to install and explore the tool is a major prerequisite.

# Acme CLI

## Prerequisites

`just`: Install just [here](https://github.com/casey/just/blob/master/README.md#installation)

## Installation

Clone this repo:
...

Set up the `acme` alias:
...

## Usage

List all available recipes:
...

acme is now ready to be used by all Acme Corp devs! We can now post a Slack message to encourage everyone to try it, and contribute their own snippets.

Further exploration

Use `nushell` for cross-platform scripts

This introduces yet another dependency, but nu is truly delightful to use. It offers an intuitive way for manipulating the results of commands, replacing tools like jq, awk, or grep. And it's cross-platform!

Completions

Completion is the mechanism that allows you to autocomplete subcommands, file paths, options, etc. when you hit the TAB key. Most shells offer this feature, most major CLI tools offer a way to install completions, and most major CLI frameworks - like click for Python, cobra for Golang, and clap for Rust - can generate them automatically.

Just can generate completions by running just --completion <shell>. However, the generated completion works for the just command, so we'll need to change the relevant parts from just to acme. I'll leave this as a bonus exercise for the reader since this is pretty easy, but I encourage checking the modified completion script into the git repo so devs won't have to do it themselves.

#cli #dev