Docker-SPK in Build pipeline

[JamborJan]

Hey @zenhack, do you think it would be possible to use docker-spk in a build pipeline? Right now it’s build to run with a local installation of docker and the tool itself. Would be interesting to have that in an automated pipeline too. I’m not yet sure how that could look like. That’s why I’m asking for your opinion.

[zenhack]

Definitely interested in exploring this.

The big question I have is, what does package signing look like in this environment?

It's probably best to start by sketching what we want the workflow to look like, which will inform what changes need to be made.

[zenhack]

If you're just looking to test that the package builds ok, it's pretty common to use docker as part of Travis. In that case you could just do docker-spk build -appkey <test key>, where <test key> is a distinct key from the one in the package manifest, which is used to sign the final package. For the CD side of things it seems a bit more complicated.

[JamborJan]

Thanks for your thoughts on that. Taking the package signing into account, I think something like this should probably work, also for other apps.

Why?
The overarching aim is still to provide an easy to adopt process to maintain apps for sandstorm. I have seen a lot of user feedback complaining about out of date apps. My personal goal is to takle over some of them and ensure they are up to date. Therefore I need a efficient and scalable setup. Right now building a new WordPress package when a new upstream release is available costs me between 20 and 30 hours minimum which is too much.

Overview

(Some) Explenation
Keeping aside the obvious stuff shown in the chart (I can explain it later if needed).

As CI CD tool for the pipelines I would use MS Azure DevOps as my company uses that already and thus I don’t need to take care about where to run the pipelines and how to pay for the resources. Also: since Github has been bought bei MS the integration and compatibility should be rock solid in the future.

The current idea would be to setup a base image repository, basically like you have done it already. My personal favourite would be a Debian testing slim image because I could re-use nearly all my Vagrantfiles for the app specific Dockerfiles. This repo would get a pipeline (Pipeline 1 in the picture) which would rebuild the image when the underlying Dockerfile changes.

Explained with my most updated and used app WordPress on Sandstorm: I currently have best experience when a new repository is created instead of forking the apps main repository for the Sandstorm version. In this repository all the Sandstorm specific stuff is tracked (basically all in .sandstorm). For complex stuff like WordPress I have created a copy of the upstream repository and applied there the changes for sandstorm (“Hack” repository). In other cases like Paperwork there where only few changes and I included the changed files in the repository of the apps Sandstorm package repository. In the more complicated case a separate pipeline (Pipeline 2 in the picture) would take care that the “Hack” repository will be updated and the necessary changes applied. I’m aware that this might cause errors, my experience is: in +90% of the cases the changes just fit in nicely and no issues occure.

The last pipeline (Pipeline 3 in the picture) gets started when the “Hack” repository gets a new master commit. It will build the docker image which runs the app including all sandstorm specific stuff. The docker image should automatically run in an testing environment. This testing environment is not a sandstorm instance. Is a plain docker image running and accessible via web.

So this is the main work which should be automated.

After that I could do some basic manual testing and if everything is okay I use docker-spk locally to create and sign the spk.

Any thoughts and ideas? Did I miss a thing? Is something “unsolvable”?

[zenhack]

For that workflow, pipeline 3 would probably just use docker itself, just doing docker build/docker run. docker-spk just invokes these commands itself, so besides shoving it in the spk there shouldn't be much difference. The issues to be aware of:

1. Obviously, if an app actually does stuff with the sandstorm APIs, running it outside of sandstorm isn't going to work -- for stuff like wordpress that's not an issue.
2. Related to (1) the sandstorm manifest is added to the filesystem after the docker image is built, so it will be missing when running in docker. If (1) isn't a problem, then this probably won't be either, but I mention it for completeness.
3. Testing in docker will not reveal problems caused by sandstorm's containerization.
4. docker-spk currently ignores CMD directives and the like, so you'd still have to make sure the launch commands in sandstorm-pkgdef.capnp agree with what the Dockerfile does when actually running in docker.

(4) strikes me as something that could be improved upon.

Thoughts?

[JamborJan]

Again thanks for your thoughts and you are absolutely right. It doesn’t make a lot sense to test without sandstorm, even in the wordpress case where we have users and permissions via sandstorm.

So I guess the automated process would stop after the pipeline 3 has created the docker image and exported it. I manually have to run docker-spk pack -imagefile my-image.tar and create spk to test this manually and push it to the Sandstorm App Store when everything seems to be okay.

I guess this is still better than today where every bit and byte needs to be done manually.

What would be a major improvement from your point of view? Would you do this completely different?

[zenhack]

One possible improvement I can think of is that if we can tweak things so you can sign the package with the correct key after testing, the build pipeline could build the spk as well, and you could just download and test. There'd be a re-signing step after though, so maybe that's just shuffling things around. That said, it shouldn't be too hard to add the ability to repackage an spk with a different signature; the pipeline would sign it with some dummy key and then after testing you'd do something like docker-spk resign wordpress-debug.spk -o wordpress.spk Not sure whether that's better or worse.

[zenhack]

We could also add a variant of spk publish that re-signs and publishes in one step, so you can effectively just test & upload, which seems like more of a clear win.

[JamborJan]

Thanks for the additional thoughts. Having a ready to test SPK would indeed improve the whole process. But I guess that is something we only know, when we are there. So I’ll start to implement the adjusted process and let you know the results. Hopefully I get something done to show before christmas.

FYI: that’s the adjusted “big picture” I have in mind for the implementation:

[zenhack]

Sounds good, I'll wait to hear.

[JamborJan]

FYI: nothing too amazing happened so far. There is a base image now.

Code: https://github.com/JamborJan/Sandstorm-Docker-Base-Image
Image: https://cloud.docker.com/u/xwrch/repository/docker/xwrch/sandstorm-docker-base-image

Staus: Pipeline 1 done, working on pipeline 2 as soon as I have some time.

[zenhack]

Cool, keep me posted. Quoting JJ (2019-01-18 07:51:20)

…

FYI: nothing too amazing happened so far. There is a base image now. Code: [1]https://github.com/JamborJan/Sandstorm-Docker-Base-Image Image: [2]https://cloud.docker.com/u/xwrch/repository/docker/xwrch/sandstorm-d ocker-base-image Staus: Pipeline 1 done, working on pipeline 2 as soon as I have some time. [3]grafiken ci cd prozess -- You are receiving this because you were mentioned. Reply to this email directly, [4]view it on GitHub, or [5]mute the thread. Verweise 1. https://github.com/JamborJan/Sandstorm-Docker-Base-Image 2. https://cloud.docker.com/u/xwrch/repository/docker/xwrch/sandstorm-docker-base-image 3. https://user-images.githubusercontent.com/7508960/51387914-1f57cb80-1b28-11e9-9592-24ad21afbe71.png 4. #9 (comment) 5. https://github.com/notifications/unsubscribe-auth/AA18PumozauAFHGVp8KLxvX7xFnggwaWks5vEcNIgaJpZM4Y02Zn