In gonix: unix from userspace and gonix: the sixth element, I introduced the full user space Unix scripting system implemented in Go.

However, shell scripting is not only about running compiled binaries. It is also about writing new functions that can be called from the scripts. Writing code in a shell itself is clunky as it was never supposed to be a real programming language. Gonix addresses this by the fact all builtins are (or can be) trivially implementable in Go.

However if everything needs to be compiled, is is still a scripting?

Writing script commands in Go

I was thrilled, surprised and amazed when I discovered mvm.sh - a fast virtual machine for Go and beyond. Why? Well it allows one to write scripts in Go. The real scripts and not a go run that needs to be cimpiled first. This is a% scripting, yet powered by Go and its standard library.

So https://codeberg.org/gonix/mvm integrates with it. As a simple example is worth lines of text. So here is one.

// "codeberg.org/gonix/mvm/core"
// "codeberg.org/gonix/gio/unix"
const code = `
package main

import "fmt"
import "os"

func main() {
  fmt.Printf("hello %s\n", os.Args[1])
}
`
b := core.NewBuilder(code)
f, _ := b.Build("test", []string{"mvm"})

var stdout bytes.Buffer
err = f.Run(ctx, unix.NewStdio(nil, &stdout, nil))
// stdout.String == "hello mvm\n"

The Go script itself if simple. It prints the first argument. The fmt.Printf is a normal Go standard library function, which is called. The mvm integration passes the arguments as os.Args into the script. Which looks like any normal go binary. How cool is that? Additionally as Build method implements a Builder pattern other things like environment variables or working dir can be passed to the script too.

An another really cool feature is the fact mvm.sh itself provides two flavors of a standard library. Those are

The same split is in the wrapper. The core.Filter type imports the subset of library that dooes not allow code to open any other communication channel besides passes stdio. No writing to files, no network connections, nothing is allowed. This makes it a perfect match for a typical Unix command. And untrusted code.

With the unsafe package there are no guarantees.

Additionally core.Filter type does not enable unsafe and sync/atomic packages to be imported. The typical unix command does not need it, so this keeps the type even more secure.

The other flavor, all.Filter, has an access to the full standard library and can bridge additional symbols from 3rd packages, as long as if they’re imported from the main binary. All external dependencies, which would require donwloading from GOPROXY or reading a go module cache are not available though.

While mvm.sh supports that, this is out of scope of what a scripting should do.

Integration shims

Lets briefly discuss about the integration. As mvm allows to shim every Go symbol, gonix integration do exactly that. Following things are always available and can’t be excluded or changed. The unix.FilterBuilderFunc implemented by Builder has all necessary data via arguments and unix.ShellContextConfig. So shims available to the scripts are

  • os.Args: for receiving command-line arguments
  • os.LookupEnv, os.Getenv and os.Environ: for accessing environment variables
  • os.Getwd: for access the working directory
  • os.Exit: virtualized by mvm.sh (filters cannot terminate the host process directly)

Additionally I wanted to ensure the called script has an access to passed context of Run method. Initially I wanted to shim context.Background(), however the documentation is pretty clear here.

It is never canceled, has no values, and has no deadline

And passing non-empty context is a violation. An another Go proverb says Clear is better than clever. The calling context is available as gonix.Context(). So the script has a full access to the proper context and can act accordingly.

// "codeberg.org/gonix/mvm/core"
// "codeberg.org/gonix/gio/unix"
const code = `
package main

import "fmt"
import "gonix"

func main() {
	ctx := gonix.Context()
	x := ctx.Value("key")
	fmt.Println(x.(string))
	<-ctx.Done()
	fmt.Println(ctx.Err())
}`
b := core.NewBuilder(code)
f, _ := b.Build("test", []string{"mvm"})

var stdout bytes.Buffer
ctx := context.WithValue(ctx, "key", "value")
ctx, _ = context.WithDeadline(ctx, 100*time.Millisecond)
err = f.Run(ctx, unix.NewStdio(nil, &stdout, nil))
// stdout.String == "value\ncontext deadline exceeded\n"

Shell functions

There is only one problem with such integration. While it replaces an individual commands, is not a replacement for shell functions. This is pretty interesting topic on its own I want to explore further.

The idea I have is roughly

// funcs.go - go functions available in a shell
package main
import "fmt"

func HelloWorld() {
  fmt.Println("hello, world")
}

func HelloArgs(args ...string) {
  fmt.Printf("%+v", args...)
}

func HelloContext(ctx context.Context, args ...string) int {
  lookupEnv := gonix.LookupEnvFunc(ctx)
  if len(args) == 0 {
    return 1
  }
  v, ok := lookupEnv(args[1])
  fmt.Printf(
    "%s=(%s, %t)\n",
    args[1],
    v,
    ok,
  )
}

And with an appropriate magic those can be made available for sh and used

source-go ./funcs.go

hello_world
hello_args "Hi" "shell" "in" "go"
FOO=BAR hello_context "FOO"

On txtar

One advantage of shell scripts is those are a single file. This is broken by having parts written in Go. Fortunately Go itself has a solution txtar format. This is trivial to write and to parse format which can contain two different languages in a single file.

#!/bin/gonix-sh
# A showcase of gonix and shell integration

hello "mvm"

-- hello --
package main

import "fmt"
import "os"

func main() {
  fmt.Println("hello " + os.Args[1])
}

Which will output hello mvm.

Conclusion

This addition goes the full circle - all the layers like compiled utilities, shell, monitoring thing, the wasm integration and now the scripted builtins - it is Go all the way round. I am curious how well the shell function integration and txtar format support will go.