cargdev/angular
1
0
Fork 0
Code Issues 16 Pull Requests Actions Packages Projects Releases Wiki Activity

ci(docs-infra): rename 'upload-server' to 'preview-server'

Browse Source 
The server no longer has files uploaded to it. Instead it is more
accurate to refer to it as dealing with "previews" of PRs.
This commit is contained in:
Pete Bacon Darwin
2018-08-15 13:47:45 +01:00
committed by Miško Hevery
parent 2f791ce68b
commit 1c34e02ae6
45 changed files with 270 additions and 267 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
aio/aio-builds-setup/docs/image-config--environment-variables.md
View File

@ -54,14 +54,14 @@ you don't need to specify values for those.
build artifacts publicly served. This is useful for enabling previews for any PR (not only those
from trusted authors).
- `AIO_UPLOAD_HOSTNAME`:
The internal hostname for accessing the Node.js upload-server. This is used by nginx for
delegating upload requests and also for performing a periodic health-check.
- `AIO_PREVIEW_SERVER_HOSTNAME`:
The internal hostname for accessing the Node.js preview-server. This is used by nginx for
delegating web-hook requests and also for performing a periodic health-check.
- `AIO_UPLOAD_MAX_SIZE`:
The maximum allowed size for the uploaded gzip archive containing the build artifacts. Files
larger than this will be rejected.
- `AIO_ARTIFACT_MAX_SIZE`:
The maximum allowed size for the gzip archive containing the build artifacts.
Files larger than this will be rejected.
- `AIO_UPLOAD_PORT`:
The port number on which the Node.js upload-server listens for HTTP connections. This is used by
nginx for delegating upload requests and also for performing a periodic health-check.
- `AIO_PREVIEW_SERVER_PORT`:
The port number on which the Node.js preview-server listens for HTTP connections. This is used by
nginx for delegating web-hook requests and also for performing a periodic health-check.

14
aio/aio-builds-setup/docs/misc--debug-docker-container.md
View File

@ -5,14 +5,14 @@ TODO (gkalpak): Add docs. Mention:
- `aio-health-check`
- `aio-verify-setup`
- Test nginx accessible at:
- `http://$TEST_AIO_NGINX_HOTNAME:$TEST_AIO_NGINX_PORT_HTTP`
- `https://$TEST_AIO_NGINX_HOTNAME:$TEST_AIO_NGINX_PORT_HTTPS`
- Test upload-server accessible at:
- `http://$TEST_AIO_UPLOAD_HOTNAME:$TEST_AIO_UPLOAD_PORT`
- `http://$TEST_AIO_NGINX_HOSTNAME:$TEST_AIO_NGINX_PORT_HTTP`
- `https://$TEST_AIO_NGINX_HOSTNAME:$TEST_AIO_NGINX_PORT_HTTPS`
- Test preview-server accessible at:
- `http://$TEST_AIO_PREVIEW_SERVER_HOSTNAME:$TEST_AIO_PREVIEW_SERVER_PORT`
- Local DNS (via dnsmasq) maps the above hostnames to 127.0.0.1
## Developing the upload server TypeScript files
## Developing the preview server TypeScript files
If you are running Docker on OS/X then you can benefit from linking the built TypeScript
files (i.e. `script-js/dist`) to the JavaScript files inside the Docker container.
@ -39,9 +39,9 @@ aio-verify-setup
```
Sometimes, the errors in the unit test log are not enough to tell you what went wrong.
In that case you can also look at the log of the upload-server itself.
In that case you can also look at the log of the preview-server itself.
A helper script that runs the unit tests (i.e. `aio-verify-setup`) and displays the
last relevant test-upload-server log is:
last relevant test-preview-server log is:
```bash
aio-verify-setup-and-log

32
aio/aio-builds-setup/docs/overview--general.md
View File

@ -3,7 +3,7 @@
## Objective
Whenever a PR job is run on the CI infrastructure (e.g. CircleCI), we want to build `angular.io`
and upload the build artifacts to a publicly accessible server so that collaborators (developers,
and host the build artifacts on a publicly accessible server so that collaborators (developers,
designers, authors, etc) can preview the changes without having to checkout and build the app
locally.
@ -40,31 +40,31 @@ container:
- The CI script checks whether the PR has touched any files that might affect the angular.io app
(currently the `aio/` or `packages/` directories, ignoring spec files).
- The CI script gzips and stores the build artifacts in the CI infrastructure.
- When the build completes CircleCI triggers a webhook on the upload-server.
- When the build completes CircleCI triggers a webhook on the preview-server.
More info on how to set things up on CI can be found [here](misc--integrate-with-ci.md).
### Hosting build artifacts
- nginx receives the webhook trigger and passes it through to the upload server.
- The upload-server makes a request to CircleCI for the URL of the AIO build artifacts.
- The upload-server makes a request to this URL to receive the artifact - failing if the size
- nginx receives the webhook trigger and passes it through to the preview server.
- The preview-server makes a request to CircleCI for the URL of the AIO build artifacts.
- The preview-server makes a request to this URL to receive the artifact - failing if the size
exceeds the specified max file size - and stores it in a temporary location.
- The upload-server runs several checks to determine whether the request should be accepted and
- The preview-server runs several checks to determine whether the request should be accepted and
whether it should be publicly accessible or stored for later verification (more details can be
found [here](overview--security-model.md)).
- The upload-server changes the "visibility" of the associated PR, if necessary. For example, if
- The preview-server changes the "visibility" of the associated PR, if necessary. For example, if
builds for the same PR had been previously deployed as non-public and the current build has been
automatically verified, all previous builds are made public as well.
If the PR transitions from "non-public" to "public", the upload-server posts a comment on the
If the PR transitions from "non-public" to "public", the preview-server posts a comment on the
corresponding PR on GitHub mentioning the SHAs and the links where the previews can be found.
- The upload-server verifies that the uploaded file is not trying to overwrite an existing build.
- The upload-server deploys the artifacts to a sub-directory named after the PR number and the first
- The preview-server verifies that it is not trying to overwrite an existing build.
- The preview-server deploys the artifacts to a sub-directory named after the PR number and the first
few characters of the SHA: `<PR>/<SHA>/`
(Non-publicly accessible PRs will be stored in a different location, but again derived from the PR
number and SHA.)
- If the PR is publicly accessible, the upload-server posts a comment on the corresponding PR on
- If the PR is publicly accessible, the preview-server posts a comment on the corresponding PR on
GitHub mentioning the SHA and the link where the preview can be found.
More info on the possible HTTP status codes and their meaning can be found
@ -73,24 +73,24 @@ More info on the possible HTTP status codes and their meaning can be found
### Updating PR visibility
- nginx receives a natification that a PR has been updated and passes it through to the
upload-server. This could, for example, be sent by a GitHub webhook every time a PR's labels
preview-server. This could, for example, be sent by a GitHub webhook every time a PR's labels
change.
E.g.: `ngbuilds.io/pr-updated` (payload: `{"number":<PR>,"action":"labeled"}`)
- The request contains the PR number (as `number`) and optionally the action that triggered the
request (as `action`) in the payload.
- The upload-server verifies the payload and determines whether the `action` (if specified) could
- The preview-server verifies the payload and determines whether the `action` (if specified) could
have led to PR visibility changes. Only requests that omit the `action` field altogether or
specify an action that can affect visibility are further processed.
(Currently, the only actions that are considered capable of affecting visibility are `labeled` and
`unlabeled`.)
- The upload-server re-checks and if necessary updates the PR's visibility.
- The preview-server re-checks and if necessary updates the PR's visibility.
More info on the possible HTTP status codes and their meaning can be found
[here](overview--http-status-codes.md).
### Serving build artifacts
- nginx receives a request for an uploaded resource on a subdomain corresponding to the PR and SHA.
- nginx receives a request for a hosted preview resource on a subdomain corresponding to the PR and SHA.
E.g.: `pr<PR>-<SHA>.ngbuilds.io/path/to/resource`
- nginx maps the subdomain to the correct sub-directory and serves the resource.
E.g.: `/<PR>/<SHA>/path/to/resource`
@ -108,4 +108,4 @@ that do not correspond with an open PR.
### Health-check
The docker service runs a periodic health-check that verifies the running conditions of the
container. This includes verifying the status of specific system services, the responsiveness of
nginx and the upload-server and internet connectivity.
nginx and the preview-server and internet connectivity.

4
aio/aio-builds-setup/docs/overview--http-status-codes.md
View File

@ -1,8 +1,8 @@
# Overview - HTTP Status Codes
This is a list of all the possible HTTP status codes returned by the nginx anf upload servers, along
with a bried explanation of what they mean:
This is a list of all the possible HTTP status codes returned by the nginx and preview servers, along
with a brief explanation of what they mean:
## `http://*.ngbuilds.io/*`

12
aio/aio-builds-setup/docs/overview--scripts-and-commands.md
View File

@ -40,23 +40,23 @@ purposes. Each command is backed by a corresponding script inside
Initializes the container (mainly by starting the necessary services).
_It is run (by default) when starting the container._
- `aio-upload-server-prod`:
Spins up a Node.js upload-server instance.
- `aio-preview-server-prod`:
Spins up a Node.js preview-server instance.
_It is used in `aio-init` (see above) during initialization._
## Developer Commands
- `aio-upload-server-test`:
Spins up a Node.js upload-server instance for tests.
- `aio-preview-server-test`:
Spins up a Node.js preview-server instance for tests.
_It is used in `aio-verify-setup` (see below) for running tests._
- `aio-verify-setup`:
Runs a suite of e2e-like tests, mainly verifying the correct (inter)operation of nginx and the
Node.js upload-server.
Node.js preview-server.
- `aio-verify-setup-and-log`:
Runs the `aio-verify-setup` command but also then dumps the logs from the upload server, which
Runs the `aio-verify-setup` command but also then dumps the logs from the preview server, which
gives additional useful debugging information. See the [debugging docs](misc--debug-docker-container.md)
for more info.

30
aio/aio-builds-setup/docs/overview--security-model.md
View File

@ -48,7 +48,7 @@ The implemented approach can be broken up to the following sub-tasks:
5. Deploy the artifacts to the corresponding PR's directory.
6. Prevent overwriting previously deployed artifacts (which ensures that the guarantees established
during deployment will remain valid until the artifacts are removed).
7. Prevent uploaded files from accessing anything outside their directory.
7. Prevent hosted preview files from accessing anything outside their directory.
### Implementation details
@ -56,7 +56,7 @@ This section describes how each of the aforementioned sub-tasks is accomplished:
0. **Receive notification from CircleCI of a completed build**
CircleCI is configured to trigger a webhook on our upload-server whenever a build completes.
CircleCI is configured to trigger a webhook on our preview-server whenever a build completes.
The payload contains the number of the build that completed.
1. **Verify that the build is valid and download the artifact.**
@ -71,7 +71,7 @@ This section describes how each of the aforementioned sub-tasks is accomplished:
Next we make another call to the CircleCI API to get a list of the URLS for artifacts of that
build. If there is one that matches the configured artifact path, we download the contents of the
build artifact and store it in a local folder. This download has a maximum size limit to prevent
PRs from producing artifacts that are so large they would cause the upload server to crash.
PRs from producing artifacts that are so large they would cause the preview server to crash.
2. **Fetch the PR's metadata, including author and labels**.
@ -98,7 +98,7 @@ This section describes how each of the aforementioned sub-tasks is accomplished:
Once we have determined whether the PR is considered "trusted", we update its "visibility" (i.e.
whether it is publicly accessible or not), based on the new verification status. For example, if
a PR was initially considered "not trusted" but the check triggered by a new build determined
otherwise, the PR (and all the previously uploaded previews) are made public. It works the same
otherwise, the PR (and all the previously hosted previews) are made public. It works the same
way if a PR has gone from "trusted" to "not trusted".
5. **Deploy the artifacts to the corresponding PR's directory.**
@ -117,23 +117,23 @@ This section describes how each of the aforementioned sub-tasks is accomplished:
Express server) rejects builds that have already been handled.
_Note: A PR can contain multiple builds; one for each SHA that was built on CircleCI._
7. **Prevent uploaded files from accessing anything outside their directory.**
7. **Prevent hosted preview files from accessing anything outside their directory.**
Nginx (which is used to serve the uploaded artifacts) has been configured to not follow symlinks
outside of the directory where the build artifacts are stored.
Nginx (which is used to serve the hosted preview) has been configured to not follow symlinks
outside of the directory where the preview files are stored.
## Assumptions / Things to keep in mind
- Other than the initial webhook trigger, which provides a build number, all requests for data come
from the upload-server making requests to well defined API endpoints (e.g. CircleCI and Github).
This means that any secret access keys need only be stored on the upload-server and not on any of
from the preview-server making requests to well defined API endpoints (e.g. CircleCI and Github).
This means that any secret access keys need only be stored on the preview-server and not on any of
the CI build infrastructure (e.g. CircleCI).
- Each trusted PR author has full control over the content that is uploaded for their PRs. Part of
the security model relies on the trustworthiness of these authors.
- Each trusted PR author has full control over the content that is hosted as a preview for their PRs.
Part of the security model relies on the trustworthiness of these authors.
- Adding the specified label on a PR and marking it as trusted, gives the author full control over
the content that is uploaded for the specific PR (e.g. by pushing more commits to it). The user
adding the label is responsible for ensuring that this control is not abused and that the PR is
either closed (one way of another) or the access is revoked.
- Adding the specified label on a PR to mark it as trusted, gives the author full control over
the content that is hosted for the specific PR preview (e.g. by pushing more commits to it).
The user adding the label is responsible for ensuring that this control is not abused and that
the PR is either closed (one way of another) or the access is revoked.

16
aio/aio-builds-setup/docs/vm-setup--create-host-dirs-and-files.md
View File

@ -12,8 +12,8 @@ More info on how to create `secrets` directory and files can be found
## Create directory for build artifacts
The uploaded build artifacts should be kept on a directory outside the docker container, so it is
easier to replace the container without losing the uploaded builds. For portability across VMs a
The build artifacts should be kept on a directory outside the docker container, so it is
easier to replace the container without losing the builds. For portability across VMs a
persistent disk can be used (as described [here](vm-setup--attach-persistent-disk.md)).
**Note:** The directories created inside that directory will be owned by user `www-data`.
@ -21,7 +21,7 @@ persistent disk can be used (as described [here](vm-setup--attach-persistent-dis
## Create SSL certificates (Optional for dev)
The host VM can attach a directory containing the SSL certificate and key to be used by the nginx
server for serving the uploaded build artifacts. More info on how to attach the directory when
server for serving the hosted previews. More info on how to attach the directory when
starting the container can be found [here](vm-setup--start-docker-container.md).
In order for the container to be able to find the certificate and key, they should be named
@ -61,15 +61,15 @@ The following log files are kept in this directory:
used when running tests locally from inside the container, e.g. with the `aio-verify-setup`
command. (See [here](overview--scripts-and-commands.md) for more info.)
- `upload-server-{prod,test,verify-setup}-*.log`:
The logs produced by the Node.js upload-server while serving either:
- `preview-server-{prod,test,verify-setup}-*.log`:
The logs produced by the Node.js preview-server while serving either:
- `-prod`: "Production" files (g.g during normal operation).
- `-test`: "Test" files (e.g. when a test instance is started with the `aio-upload-server-test`
- `-test`: "Test" files (e.g. when a test instance is started with the `aio-preview-server-test`
command).
- `-verify-setup`: "Test" files, but while running `aio-verify-setup`.
(See [here](overview--scripts-and-commands.md) for more info the commands mentioned above.)
- `verify-setup.log`:
The output of the `aio-verify-setup` command (e.g. Jasmine output), except for upload-server
output which is logged to `upload-server-verify-setup-*.log` (see above).
The output of the `aio-verify-setup` command (e.g. Jasmine output), except for preview-server
output which is logged to `preview-server-verify-setup-*.log` (see above).

4
aio/aio-builds-setup/docs/vm-setup--start-docker-container.md
View File

@ -31,7 +31,7 @@ sudo docker run \
--detach \
# Use the local DNS server.
# (This is necessary for mapping internal URLs, e.g. for the Node.js upload-server.)
# (This is necessary for mapping internal URLs, e.g. for the Node.js preview-server.)
--dns 127.0.0.1 \
# USe `<instance-name>` as an alias for the container.
@ -50,7 +50,7 @@ sudo docker run \
# (See [here](vm-setup--set-up-secrets.md) for more info.)
--volume <host-secrets-dir>:/aio-secrets:ro \
# The uploaded build artifacts will stored to and served from this directory.
# The build artifacts and hosted previews will stored to and served from this directory.
# (If you are using a persistent disk - as described [here](vm-setup--attach-persistent-disk.md) -
# this will be a directory inside the disk.)
--volume <host-builds-dir>:/var/www/aio-builds \