Adding CI support for AMD platforms

[saba-kareem]

Hello,

We are working on enabling CI support for sof github Pull requests(For Firmware & Driver) for AMD Platforms.
We have gone through existing scripts located in sof-test project.
We do have few questions on setting up jenkins server on our side.

1. How to add our remote servers for building the code and publish the results ?
2. How to add remote targets for running audio test cases?

@marc-hb @plbossart Could you please help us on the above queries?
Based on your inputs, we will reach out to our DevOps team.

[marc-hb]

Hi @saba-kareem , great news!

On my side the news are not so great unfortunately :-( @plbossart has retired and I'm now working on a different Intel project; same for other Intel people who used to work on SOF CI.

I can still share a short description of how things are set up but a new team is designing a better system right now, maybe @cgturner1 and others who took over can elaborate at some point? I think this new system will not use Jenkins anymore but this should not fundamentally change the very high level design which is not rocket science and described below.

Note the internal CI infrastructure code is mostly Intel-specific and closed source. As you know, 99% of the test code is already open-source in sof-test with a 100% goal. Some of the code to configure and monitor test devices could be open source but AFAIK there's no plan to open any of that. It's not very good anyway ;-) If you're starting from scratch then Ansible is probably one of your best options.

Jenkins is scanning and polling the 3 main git repos. Some internal mirror is needed to avoid hitting GitHub rate limits. Some sort of "Webhooks" should be used instead of polling but we never got to seriously try that (firewalls add significant challenges).

Whenever a Pull Request is created or modified, Jenkins code builds it if needed, deploys it, runs sof-test and and collects the results. WARNING: there is a large attack surface here, so make sure test devices are on a very restricted network and test only PRs from project members. Results are uploaded to sof-ci.01.org (Intel-only server) and standard Github Checks APIs are used to link to them. Standard GitHub APIs can also be used to post comments and in fact Jenkins should have posted links as comments that because Github checks have very bad "memory", making it almost possible to find past test results.

Then the devil is in the details :-)

[lgirdwood]

@saba-kareem let me know what your DevOps inputs are, I think we also need to add some CI user/token to enable the external reporting.

[cgturner1]

@saba-kareem I think it also depends on your team's policy on the level of access they are willing to grant your servers...if they are not behind a firewall and can easily access this repo it may be more straightforward to set up access for your Jenkins instance to the repo (for intel this is not the case so it makes it a bit harder to work with) ... and also as a fairly new person to the project I would assume we would need to double check on the sof's project to get consent and whatnot, since there may be a policy or check I may be unaware of.

We are in the middle of re-thinking how we publish results (as well as test) but currently this is an instance owned and maintained by Intel so not sure if we would be able to push results to it but ideally we find a better open way to push and publish results which is what we are trying to move towards and replace the current process

[marc-hb]

IMHO there would be zero advantage to different companies publishing and hosting test logs to the same server. Drawbacks only. Test logs from different companies are going to come from different network and administrative domains anyway, so it's much simpler to just keep all servers and systems separate all the way. This avoids complex, inter-companies discussions about who has access to what, who pays what, quotas, expiration policies, etc. Unless of course you enjoy more "quality time" with company lawyers? ;-) Also, imagine an accident where test logs leak something confidential. In such an unfortunate case, would anyone want any relevant server to be shared? That would be the "icing on the cake"!

I think we also need to add some CI user/token to enable the external reporting.

This is IMHO the only place that requires some inter-company collaboration: GitHub access control policies and authentication tokens that let CI engines post LINKS from pull requests to test logs (hosted on company-specific servers). That's the unavoidable collaboration minimum.

There could be advantages to sharing/open-sourcing some of the infrastructure code publishing test logs. But not the hosting itself.

PS: while the word "intel" is missing from the hostname https://sof-ci.01.org/, it is very much 100% Intel.