A while back, I watched the video where Dennis Ritchie was showing Unix command line. And I though to myself.

How hard would it be to replicate the unix pipe in Go?

Given the fact I began experimenting 4 years ago, it might be tempting to say yes. It is hard! However, implementing a Unix command and a pipe itself is trivial in Go. All the necessary ingredients are already present in a standard library.

  • pipe is io.Pipe
  • the producer and consumer runs concurrently and Go has goroutines

TL;DR;

Let me introduce the project quicky. https://codeberg.org/gonix provides a Unix utilities and scripts as a (Go) programming library. Since the utilities and shell interpreter itself themselves are Go - there’s no fork+execve, no extra syscalls, it is all standard programming in userspace. And thanks to people working on reimplementation of GNU core utilities to Rust GNU compatible coreutils are available right now.

Consider this trivial example of the script.

> gonix/cat /etc/passwd | rust/wc -l
42

While it does look like one and behave like a script, it is all pure Go all the way down.

  1. https://github.com/mvdan/sh implements the shell itself.
  2. cat is implemented by a gonix project itself. https://codeberg.org/gonix/gonix/src/branch/main/cat
  3. wc uses WebAssembly build of https://github.com/uutils/coreutils project.

The names gonix/cat and rust/wc are arbitrary - scripts can be called by any name and these names has been chosen in order to showcase the fact this is not a script in a traditional sense.

Motivation

Despite using Linux for almost two decades, I have never liked the strict separation between scripting and a programming. There is no libgrep or libsed or libawk that combines the higher level tools with a proper programming environment. Projects like libcurl/curl do exist though. Even then - if a script solves the problem by calling a curl, the solution must be written again using interfaces tailored to C or other real programming language.

So while command line tools and scripting languages themselves are written in C (or Go or Rust or Python or Node). Consuming those back as a software package is awkward (see gpgme as an infamous example). This project is an attempt to merge both worlds together and keep scripting useful part and integrated part of a programming.

The cat | wc example above can be implemented using a few lines of Go and all runs in userspace inside the same process without a single additional fork+exec in sight.

// register commands
var cmds = map[string]unix.FilterBuilderFunc{
  "gonix/cat": cat.FromArgs,
  "rust/wc":  coreutils.Wc(),
}
// parse the script
file, _ := syntax.NewParser().
  Parse(strings.NewReader(script), "gonix/cat ${FILE} | rust/wc")

runner, _ := interp.New(
  interp.Env(expand.ListEnviron("FILE=test.me")),
  interp.StdIO(nil, os.Stdout, os.Stdout),
  interp.ExecHandlers(c.ExecHandler),
)
) = runner.Run(context.Background(), file)

Shoulders of giants

The most fascinating aspect of this project is how little code I needed to write. It is also impressive how much I can rely on work of others. The whole entire project contains less 10,500 lines of a code. This is a cool metric, especially in this AI agentic era.

This project would not be possible without the creators of Unix and Go and

The unix architecture

Consider this very simple Unix shell pipeline.

> cat /etc/passwd | wc -l
42

In a traditional unix architecture there are many components, which must be there for this to work.

  1. unix kernel
  2. sh, cat and wc binaries installed
  3. kernel starts the sh process
  4. sh must open(2) and read(2) the script
  5. sh must call pipe(2) to create a pipe
  6. sh must call fork(2) for each sub-process
  7. sh must call dup(2) connecting the stdout of cat with stdin of wc
  8. sh must call execve(2) for both subprocesses
  9. sh calls waitpid(2) to get a report from kernel when and how tools ended
  10. sh can finally exit(2) and report the status

Do not forget this is simplified version of all the work computer does. It ignores stuff like signal handling and all other stuff, dynamic linking, locales, memory calls like brk and mmap, opening locale and other files and so on. The GNU userspace executed as sh test.sh ran 567 syscalls.

Additionally, a script will only work if these are installed on the target system. This can become a problem if

  • You cannot modify the system.
  • The required tool is not easily available.
  • Your system don’t have latest GNU utilities and all it has are outdated unix tools.
  • your system is alien enough, that even if a Unix environment has been ported, this will never feel native.

With gonix shell scripting becomes 100% portable.

Gonix architecture

The core of unix command is the standard io. The C abstraction is called a file descriptor and is represented by a type int.

/* #include<unistd.h> */

/* File number of stdin; 0. */
#define STDIN_FILENO 0
/* File number of stdout; 1. */
#define STDOUT_FILENO 1
/* File number of stderr; 2. */
#define STDERR_FILENO 2

This concept can be easily expressed in Go.

type StandardIO interface {
  Stdin() io.Reader
  Stdout() io.Writer
  Stderr() io.Writer
}

Which is exactly what gio/unix does.

And while each C unix tool starts with a main function

#include <unistd.h>

int main() {
  write(STDOUT_FILENO, "stdout\n", strlen("stdout\n"));
  write(STDERR_FILENO, "stderr\n", strlen("stderr\n"));
}

For our purpose its better to define an interface

type Filter interface {
  Run(context.Context, StandardIO) error
}

and implement it

type Demo struct{}
func(Demo) Run(_ context.Context, stdio unix.StandardIO) error {
  fmt.Fprintf(stdio.Stdout(), "stdout\n")
  fmt.Fprintf(stdio.Stderr(), "stderr\n")
  return nil
}

Example: cat

The cat without any arguments is literary an one-liner with io.Copy doing all the work.

// import codeberg.org/gonix/gio/unix
func (Cat) Run(_ context.Context, stdio unix.StandardIO) error {
  _, err := io.Copy(stdio.Stdout(), stdio.Stdin())
  return err
}

Example: wc -l

The wc -l is a bit more lines, but still not very complicated.

// import codeberg.org/gonix/gio/unix
func (Lines) Run(_ context.Context, stdio unix.StandardIO) error {
  count := 0
  scanner := bufio.NewScanner(stdio.Stdin())
  for scanner.Scan() {
      count++
  }
  err := scanner.Err()
  if err != nil {
    return err
  }
  fmt.Fprintf(stdio.Stdout(), "%d\n", count)
  return nil
}

Command line arguments

Run and unix.Filter interface can satisfy the easiest tools without a command line arguments or any notion of an outer context like working directory. The typical Unix tool expects and receives command line arguments when executed. This is solved by filter builder pattern

func(command string, args []string, opts ...ShellContextOption) (Filter, error)

What way implementation can get more context than context and standard io only.

type CatWithArgs struct {
  command string
  args []string
}

// import codeberg.org/gonix/gio/unix
func (c CatWithArgs) Run(_ context.Context, stdio unix.StandardIO) error {
  if err := c.parse(c.args); err != nil {
    return fmt.Errorf("%s: %w", c.command, err)
  }
  _, err := io.Copy(stdio.Stdout(), stdio.Stdin())
  return err
}

The shell context

Let’s consider this example

FOO=42 printenv

Unix tools expects environment variables too. This is achieved by opts ...ShellContextOption of a builder. Every tool can then get proper functions which can be used to get an access. The key design concern here is to decouple the interface for such functionality with an actual implementation

  1. interface is defined in gio/unix
  2. injection is a part of unix.FilterBuilderFunc
  3. actual implementation if done in sh package for mvdan.cc/sh/v3.

Tools can register those helpers to obtain enough from the context as seen in a following example.

var b uroot.Builder
var builders = unix.FilterLookupMap{
  "mktemp": b.Mktemp(),
}
filters := sh.NewFilters(
  builders.Lookup,
  unix.WithLookupEnvFunc(sh.LookupEnvFunc),
  )

const source = "TMPDIR=/tmp42 mktemp -d t.XXXXXX"
parsed, err := syntax.NewParser().
  Parse(strings.NewReader(source), "")
require.NoError(t, err)

var got bytes.Buffer

runner, err := interp.New(
  interp.StdIO(nil, &got, nil),
  interp.ExecHandlers(filters.ExecHandler),
)
// expect error as /tmp42 does not exists

Available options are

  1. get working directory via unix.GetDirFunc
  2. lookup environment variable via unix.LookupEnvFunc
  3. get all environment via unix.GetEnvFunc

What did the gonix did for us?

Nothing at the moment. However I think this can improve an existing container tooling for example.

FROM scratch

# I need to create a directory
# but can't as mkdir is not there
RUN mkdir /app

or the debugging sessions

# go away: cannot execute /bin/ls: file not found
docker exec scratch ls -lh /app

At this moment all the tools are supposed to be installed inside the system itself. And consider a hypothetical improved tooling

FROM scratch

# mkdir is available as a part of Go runtime
# no need to install a thing
GONIX rust/mkdir /app
# ls is a part of runtime, so rust/ls will work
better-docker exec --gonix 'rust/ls -lh /app' scratch

The other possible use cases are

  • And monitoring probes
  • Tools which setups systems using scripts like distrobox
  • Experimenting with more exotic tools like those written for Plan9
  • AI companions writing and executing scripts
  • MCP server
  • And many more

Roadmap?

There is any as this is a project of love. However the interesting areas to consider are

  1. context aware io - at this moment individual filters much implement context checks individually - idea here is to wrap context and io together, so Read/Write methods will stop working once context is canceled
  2. busybox-like tool which packs as much other tools as possible
  3. interactive shell - being in Go it can have nice things like native fuzzy search

Limitations

There are more aspects to a real Unix system than just pipes and environment variables. Some are fundamental limits of how the kernel works, while others may be implemented later. These limitations include things like

Kernel properties

The project is an emulation of a real Unix system completely bypassing kernel and all its features. So process isolation, process tracking, signals, control groups, security context, SELinux labels, user and groups, extended attributes and so on - nothing does work inside gonix.

Unless the fork+exec is involved - https://codeberg.org/gonix/gio/src/branch/main/unix/exec.go#L30 is available.

In other words gonix is not a security layer. Additionally running inside the same OS process, you will never get SUID binaries like sudo working in userspace.

Can’t execve by default

Other part are unit tools calling fork+exec themselves like xargs. There is no good solution for a reverse wrapper which will hijack exec.Command and covert it to unix.Filter. So there is GNU compatible implementation of https://codeberg.org/gonix/gonix/tree/main/xargs/ which gets the unix.FilterBuilderFunc as a parameter and can execute other userspace tools.

This design allows setups like the main shell having an access to different filters than xargs itself. This is how it is unit tested by the way.

Signals and process tracking

While this one can be emulated, at the moment individual utilities don’t respect context.Context by default unless specified elsewhere. The design of Go context will most likely allow emulating the signals, but this is not yet done. The relevant part is a process tracking, this is not implemented - if something get stuck, you must kill a whole process.

Interactivity

While interesting, neither mvdan.cc/sh/v3 neither the utilities are developed and tested with interactivity in mind.

Are there any other similar projects?

I haven’t searched thoroughly and those are two examples I am aware of.

https://github.com/ewhauser/gbash seems to be the closest. However it seems to be more AI driven and I expect that most of code there is generated.

The gbash project itself references https://github.com/vercel-labs/just-bash which seems to be the same, only in Typescript.

AI

While code in the project was written by me, I must admit I would not be able to progress so quickly without a help of LLMs.

  1. Proof of concept code including wazero integration has been provided by various AI tools as well as a few test cases.
  2. The ideas behind unix.ShellContextOption has been investigated by Claude. It was nice to be able to see and reject all the wrong solutions it produced, which helped me to figure out the proper design.
  3. Experimental sbase web assembly port has been vibe-coded by Claude.
  4. deepl.com/write reviewed this text as usual.
  5. Google AI explained me how to use Inkscape to convert the png logo into svg.

Gopher by Rene French CC license https://upload.wikimedia.org/wikipedia/commons/2/2d/Go_gopher_favicon.svg and Unix Terminal Logo Dollar Akshay