By default our organization on GH only grants a more restricted read
permission to actions for content APIs, which include both writing to
repo for nightly tag creation, and writing to releases for nightly and
tag release creation or updates.
Release will be created as a draft, remaining private until manually published.
All generated artifacts will be uploaded, but will still need to be signed, and
the code archives still need to be created following
MAINTAINING.md#after-tagging
Fix#6345
* Tag is required to create a release, so tag "nightly" is moved to
latest commit on each run.
* Each time the tag is moved, the existing "nightly" release is updated
automatically by github to point to the new tag and bundle the new
commit's source code.
* Artifacts are uploaded with intentionally conflicting names to replace
last commit's version of artifacts, rather than deleting the whole
release and making a new one.
* This has two benefits, it doesn't notify of a new release on each
commit, and it doesn't leave a gap where there's no nightly release
available.
* It is only partially covered by CI and doesn't simplify the build process
much for users. Replacing it in CI with just build-osx.sh significantly reduces
script complexity and is fully tested.
* bootstrap-osx.sh copying system libs and headers locally is unneeded.
Already the DMG file contains them, and re-linking the app against
updated system files may be desirable.
* Update INSTALL.md for macOS to use brewfile, use common dependency build
scripts, and use cmake rather than wrapper scripts.
* Build macOS in Release mode in CI, for release artifact creation
* Don't copy all used libs into a local folder, macdeployerqt already
handles this for the dmg, and for local running of the app using the
system libs and relinking on update is desirable to avoid running out of
date dependencies unexpectedly.
Motiviation:
* Reproducing issues in CI is currently difficult
* Predicting issues in CI is currently difficult if you are not on
ubuntu 18.04
* Reproducing issues submitted from other distros is currently done by
creating a VM of that distro and building qtox for it locally
* Documentation for how to build on different distros is out of date
* Issues on non-ubuntu distributions are not caught by CI
* Cross compiling for windows locally is not trivial
* Iterating when working with custom build scripts is slow, scripts
don't necessarily support re-running without re-starting the docker
container and re-building qtox again
* Updating dependencies is a pain
Changes:
* docker-compose file has been added to the root of our repo.
After `docker compose run --rm ubuntu` (or other supported distros),
you are ready to compile and run qtox
* Dependencies are owned by dependency install scripts in buildscripts/.
This allows us to use the same exact dependencies in our
OSX/windows/linux scripts
* New docker images have been added for a variety of distributions.
These are now run in CI in a variety of configurations
* Docker images are cached in CI so rebuild time for the majority of
jobs is quite quick
* Build scripts have been trimmed to leverage state of docker
containers.
* Windows build script no longer installs anything, dependencies are
now managed by the windows_builder docker images
* Build scripts should now be easily re-runnable. Usage is now `docker
compose run --rm <image>` and then run the scripts
* All artifacts are now uploaded to github after build, this means we
can take an appimage/flatpak/exe/dmg for any given PR and try it out
without having to build it ourselves
Notes:
* Docker image size is quite important. We have a maximum of 5GB cache
space on github actions. The majority of the linux distro docker
images cache at ~300-400MB, which gives us room to test ~6 distros
after accounting for the sizes of flatpak/windows docker images
* Docker layer ordering is relatively intentional. Approximate order
should be that large dependencies that change infrequently should be
farther up. This lowers the amount of rebuilding we have to do when
dependencies are updated
* download_xxx.sh scripts are the cleanest way I could find to implement
a shared dependency map between osx scripts and docker containers.
Although it would be nice to have a single dependency mapping file,
splitting it into individual scripts allows us to only rebuild some
docker layers when dependencies are updated.
* Github actions are split between docker image building and docker
image use. This allows us to re-use the same docker images for
multiple jobs, but only build it once
* Unfortunately I could not find a way to de-duplicate the stitching
between jobs, so we have a lot of copy pasta in that area
Previously, the four matrix Windows jobs would run in parallel and each
build deps, which is redundant since both debug and release qTox use the
same deps.
Now, the win-deps job builds only the needed two versions of deps, and
the four windows builds wait for it to complete before running in
parallel.
So that each commit in repo is tested, including merge commits that
aren't present in PRs. Also prepares for running releases only on push
but not on PRs.
Workflows have a global runs-on, so commit format check needs to be
split out from the shared PR and push workflow to since checking all
commit messages on pushes always fails due to very old commits not
following the proper format.
Partially fix#6345, only PR portion, not nightly or releases.
Unlike travis, we're not caching our brew packages. `actions/cache` doesn't
update the cache on cache hit, making it hard to use a rolling cache like
before. We also don't know what cache key we should use before running, since
it relies on the live package list of brew.
Using a docker image containing brew packages seems like a better option if we
need the speedup going forward. ATM a full build with deps only takes about 12
minutes. Windows builds don't have this issue, since deps there are keyed off of
known versions in our repo.
Current Windows matrix form works, but causes a double-build on dep change due
to both debug/release rebuilding the release deps. Instead, should probably
separate the dep jobs, block build on the dep jobs, and update the cache on dep
jobs, guaranteeing a cache hit on build jobs?
Windows stage 1 and stage 2 can probably be recombined, if they were split due
to travis single build length limits, since GH actions allows much longer single
builds.
Mick Sayson
Anthony Bilinski