Ryan Gerstenkorn home

Bridging the gap between code and serverless

22 November 2020 - Somewhere

Disclaimer: I’m fully aware this is a insane idea. That said…

Sometime ago I started a project called coderun it was meant to make running code in docker containers and lambda stupid easy and secure. I kinda left it in a half finished state because I wasn’t sure where I wanted to take the project.

It was a bit over engineered but the idea was you could simply start the shell and execute any script and it would pull down the right container and run the code. It even had fancy plugins that acted like little snitch as well as FuseFS mounts for intercepting calls to sensitive files, allowing you to accept or block access. All though these where quite interesting, I think the the more interesting part of the project is treating code, docker, and lambda the same, wrapped in a shell you know all too well.

i.e if you run ./test.py in this shell will deploy and run it in docker the same as it would in lambda (if it was running in the lambda mode).

Of course docker and lambda are not the same thing though.. maybe I should have started with an ECS mode here instead. But anyways, I’ll think about that another time. What I want to try to figure out is how can we seamlessly bridge local and remote resources in a way that is as simple as possible. The rest of this post is mostly just going to be a collection of notes more than anything.

Local Shell

> ./first.sh && ./sometimes-second.sh; always-third.sh

Self explanatory, first.sh always executes, sometimes-second.sh executes if the first fails and always-third.sh executes every-time at the end.

Docker (coderun)

> ./first.sh

We can do the same in docker fairly simply either by making our own shell which tries to dynamically determine what the docker image we should be. This is what I attempted to do with coderun.

Docker with shell syntax (jessfraz’s setup)

> ./first.sh && ./sometimes-second.sh; always-third.sh

A way simpler way to do what I tried to do above can be done using jessfraz’s setup. It also just run’s in the normal shell, so nothing special going on here for &&, ;, etc…

lambda (coderun)

./first.sh

coderun also attempted to do this in a similar way as the docker example.

lambda with shell syntax (TODO)

> ./first.sh && ./sometimes-second.sh; always-third.sh

Using event machine we can define a fair bit of shell scripts. I’ve messed around a bit with this but lost my notes on it, but either way though I’ll likely be coming back to this. We should be able to do stdin/out/err piping here as well.

Tying it all together

I think this is the big question that I need to think about more.

Various thoughts

Config

Piped Config

{ Eventtopic: [start, stop, fail], Stdin: [queue|stdin|stdout|stderr], Stdout: queue,… Stderr: [queue|stdin|stdout|stderr], }

echo example

./echo.msh file:

#!/Users/ryan/Code/msh/out/msh dockerfile
FROM alpine:3
ENTRYPOINT ["echo"]

Running it:

> ./echo.msh Hello && echo world
Hello
world

Dockerfile get’s built with anonymous tag and executed.

The internal msh config when running this command is:

{ 
  Stdin:  os.stdin,
  Stdout: os.stdout,
  Stderr: os.stderr,
}

pipe example

./cat.msh file:

#!/Users/ryan/Code/msh/out/msh dockerfile
FROM alpine:3
ENTRYPOINT ["cat"]

Running it:

> ./echo.msh "Hello world | ./cat.msh
Hello world

Dockerfile get’s built with anonymous tag and executed.

The internal msh config for ./echo.sh is:

{ 
  Stdin:  os.stdin,
  Stdout: os.stdout,
  Stderr: os.stderr,
}

It is passed through the pipe to ./cat.sh as the first line (new lines are removed from the json above). Msh in ./cat.msh recieves this and knows that it where it should look for stdout and stderr from the previous command. If os.stdout is being used as output the json is stripped before data from the pipe is passed to ./cat.sh.