When running in parallel mode, worker processes forward errors thrown
during task processing to the master process, which in turn exits with
an error.
However, there are cases where the error is not directly related to
processing the entry-point. One such case is when there is not enough
memory (for example, due to all the other tasks being processed
simultaneously).
Previously, an `ENOMEM` error thrown on a worker process would propagate
to the master process, eventually causing ngcc to exit with an error.
Example failure: https://circleci.com/gh/angular/angular/682198
This commit improves handling of these low-memory situations by
detecting `ENOMEM` errors and killing the worker process, thus allowing
the master process to decide how to handle that. The master process will
put the task back into the tasks queue and continue processing tasks
with the rest of the worker processes (and thus with lower memory
pressure).
PR Close#36626
Previously, when the last worker process crashed, the master process
would try to re-spawn it indefinitely. This could lead to an infinite
loop (if for some reason the worker process kept crashing).
This commit avoids this by limiting the number of re-spawn attempts to
3, after which ngcc will exit with an error.
PR Close#36626
Previously, when running in parallel mode and a worker process crashed
while processing a task, it was not possible for ngcc to continue
without risking ending up with a corrupted entry-point and therefore it
exited with an error. This, for example, could happen when a worker
process received a `SIGKILL` signal, which was frequently observed in CI
environments. This was probably the result of Docker killing processes
due to increased memory pressure.
One factor that amplifies the problem under Docker (which is often used
in CI) is that it is not possible to distinguish between the available
CPU cores on the host machine and the ones made available to Docker
containers, thus resulting in ngcc spawning too many worker processes.
This commit addresses these issues in the following ways:
1. We take advantage of the fact that files are written to disk only
after an entry-point has been fully analyzed/compiled. The master
process can now determine whether a worker process has not yet
started writing files to disk (even if it was in the middle of
processing a task) and just put the task back into the tasks queue if
the worker process crashes.
2. The master process keeps track of the transformed files that a worker
process will attempt to write to disk. If the worker process crashes
while writing files, the master process can revert any changes and
put the task back into the tasks queue (without risking corruption).
3. When a worker process crashes while processing a task (which can be a
result of increased memory pressure or too many worker processes),
the master process will not try to re-spawn it. This way the number
or worker processes is gradually adjusted to a level that can be
accomodated by the system's resources.
Examples of ngcc being able to recover after a worker process crashed:
- While idling: https://circleci.com/gh/angular/angular/682197
- While compiling: https://circleci.com/gh/angular/angular/682209
- While writing files: https://circleci.com/gh/angular/angular/682267
Jira issue: [FW-2008](https://angular-team.atlassian.net/browse/FW-2008)
Fixes#36278
PR Close#36626
This commit adds a `revertFile()` method to `FileWriter`, which can
revert a transformed file (and its backup - if any) written by the
`FileWriter`.
In a subsequent commit, this will be used to allow ngcc to recover
when a worker process crashes in the middle of processing a task.
PR Close#36626
With this commit, the master process will keep track of the transformed
files that each worker process is intending to write to disk.
In a subsequent commit, this info will be used to allow ngcc to recover
when a worker process crashes in the middle of processing a task.
PR Close#36626
With this commit, worker processes will notify the master process about
the transformed files they are about to write to disk before starting
writing them.
In a subsequent commit, this will be used to allow ngcc to recover when
a worker process crashes in the middle of processing a task.
PR Close#36626
This commit enhances the `CompileFn`, which is used to process each
entry-point, to support running a passed-in callback (and wait for it to
complete) before proceeding with writing the transformed files to disk.
This functionality is currently not used. In a subsequent commit, it
will be used for passing info from worker processes to the master
process that will allow ngcc to recover when a worker process crashes in
the middle of processing a task.
PR Close#36626
Rename the `markTaskCompleted()` method to be consistent with the other
similar methods of `TaskQueue` (`markAsFailed()` and
`markAsUnprocessed()`).
PR Close#36626
This commit adds support for stopping processing an in-progress task
and moving it back to the list of pending tasks.
In a subsequent commit, this will be used to allow ngcc to recover when
a worker process crashes in the middle of processing a task.
PR Close#36626
Previously, ngcc would run in parallel mode (using the
`ClusterExecutor`) when there were at least 2 CPU cores (and all other
requirements where met). On systems with just 2 CPU cores, this meant
there would only be one worker process (since one CPU core is always
reserved for the master process). In these cases, the tasks would still
be processed serially (on the one worker process), but we would also pay
the overhead of communicating between the master and worker processes.
This commit fixes this by only running in parallel mode if there are
more than 2 CPU cores (i.e. at least 2 worker processes).
PR Close#36626
Previously, the "Compiling <entryPoint>" log message was printed before
starting to analyze and transform files, but after creating the
`EntryPointBundle` (which includes creating the TS program).
Since creating the `EntryPointBundle` involves some work, it is more
accurate to move the log message before creating the bundle.
PR Close#36626
This reverts commit d669429bd26dbd753584c331a6d7aa44a4d1f3ee.
The Angular CLI relies upon deep imports into `@angular/localize`. In this
case it relies upon the `source_file_utils.ts` file being in its previous
position.
In `master` (i.e. v10) the CLI will update its import to cope but we need
to revert for 9.1.x.
PR Close#36861
If there's an error during the first creation pass of a `TView`, the data structure may be corrupted which will cause framework assertion failures downstream which can mask the user's error. These changes add a new flag to the `TView` that indicates whether the first creation pass was successful, and if it wasn't we try re-create the `TView`.
Fixes#31221.
PR Close#36381
An enum declaration in TypeScript code will be emitted into JavaScript
as a regular variable declaration, with the enum members being declared
inside an IIFE. For ngcc to support interpreting such variable
declarations as enum declarations with its members, ngcc needs to
recognize the enum declaration emit structure and extract all member
from the statements in the IIFE.
This commit extends the `ConcreteDeclaration` structure in the
`ReflectionHost` abstraction to be able to capture the enum members
on a variable declaration, as a substitute for the original
`ts.EnumDeclaration` as it existed in TypeScript code. The static
interpreter has been extended to handle the extracted enum members
as it would have done for `ts.EnumDeclaration`.
Fixes#35584
Resolves FW-2069
PR Close#36550
There was a lot of duplication and multiline backtick
strings that made it hard to maintain.
Some more text to ensure the commit message is long enough.
PR Close#36834
This will allow the utilities in this file to be shared outside
`translate` code.
Some more text to get to the 100 character commit message requirement.
PR Close#36834
Prior to this change, animations-related runtime logic assumed that the @HostBinding and @HostListener with synthetic (animations) props are used for Components only. However having @HostBinding and @HostListener with synthetic props on Directives is also supported by View Engine. This commit updates the logic to select correct renderer to execute instructions (current renderer for Directives and sub-component renderer for Components).
This PR resolves#35501.
PR Close#35568
After the user edits the file `core.d.ts`, the symbol from the core module will be invalided, which only is created when init the language service. Then the language-service will crash.
PR Close#36783
* Move tools/brotli-cli, tools/browsers, tools/components,
tools/ng_rollup_bundle, and modules/e2e_util to dev-infra/benchmarking
* Fix imports and references to moved folders and files
* Set up BUILD.bazel files for moved folders so they can be packaged with
dev-infra's :npm_package
PR Close#36434
This was originally fixed in #35976, but one of the window.scrollY
assertions was missed. Also updated tests to use toBeGreater/LessThan
to improve failure messages.
PR Close#36742
This PR adds test case to cover a failure that was detected after
merging #36302. That commit will be reverted and will need a new PR that
does not cause this test to fail.
PR Close#36699
Moves the circular deps golden for packages into a subfolder of goldens,
`/goldens/circular-deps/` to more easily target the files for
ownership.
PR Close#36630
Bumping the version of benchpress as a new version needs to be released
as part of the effort to set up more benchmarking accross the
angular/angular and angular/components repos.
PR Close#36457
Previously, we only displayed the new `$localize` id, which is not
currently what most people have in their translation files.
Until people migrate to the new message id system it is confusing
not to display the legacy ids.
PR Close#36761
This reverts commit 1756cced4a59bc1185653c6c9b8f661636fd6116.
The reason for this revert is because of the change being too risky for
a patch release of Angular. Changing date formatting proves to be a
breaking change and can only be apart of the next release.
PR Close#36751
When module overrides (via `TestBed.overrideModule`) are present, it might affect all modules that import (even transitively) an overridden one. For all affected modules we need to recalculate their scopes for a given test run and restore original scopes at the end. Prior to this change, we were recalculating module scopes only for components that are used in a test, without taking into account module hierarchy. This commit updates Ivy TestBed logic to calculate all potentially affected modules are reset cached scopes information for them (so that scopes are recalculated as needed).
Resolves#36619.
PR Close#36649
Prior to this commit unbound attributes were treated as possible inputs to structural directives. Since structural directives can only accepts inputs defined using microsyntax expression (e.g. `<div *dir="exp">`), such unbound attributes should not be considered as inputs. This commit aligns Ivy and View Engine behavior and avoids using unbound attributes as inputs to structural directives.
PR Close#36441
Based on the migration guide, provided classes which don't have
either `@Injectable`, `@Directive`, `@Component` or `@Pipe` need
to be migrated.
This is not correct as provided classes with an `@NgModule` also
have a factory function that can be read by the r3 injector. It's
unclear in which cases the `@NgModule` decorator is used for
provided classes, but this scenario has been reported.
Either we fix this in the migration, or we make sure to report
this as unsupported in the Ivy compiler.
Fixes#35700.
PR Close#36369
The cached file-system was implemented to speed up ngcc
processing, but in reality most files are not accessed many times
and there is no noticeable degradation in speed by removing it.
Benchmarking `ngcc -l debug` for AIO on a local machine
gave a range of 196-236 seconds with the cache and 197-224
seconds without the cache.
Moreover, when running in parallel mode, ngcc has a separate
file cache for each process. This results in excess memory usage.
Notably the master process, which only does analysis of entry-points
holds on to up to 500Mb for AIO when using the cache compared to
only around 30Mb when not using the cache.
Finally, the file-system cache being incorrectly primed with file
contents before being processed has been the cause of a number
of bugs. For example https://github.com/angular/angular-cli/issues/16860#issuecomment-614694269.
PR Close#36687
The flag that determines whether something should be able to inject from `viewProviders` is opt-out and the pipes weren't opted out, resulting in them being able to see the viewProviders if they're placed on a component host node.
Fixes#36146.
PR Close#36512
This commit fixes how the language service evaluates the compatibility
of types to work with arbitrary union types. As a result, compatibility
checks are now more strict and can catch similarities or differences
more clearly.
```
number|string == string|null // OK
number|string == number // OK
number|string == null // not comparable
number == string // not comparable
```
Using Ivy as a backend should provide these diagnoses for free, but we
can backfill them for now.
Closes https://github.com/angular/vscode-ng-language-service/issues/723
PR Close#36529
The change in e041ac6f0d8c519d104601b2b49c008721c074b7
to support sending unlocker process output to the main ngcc
console output prevents messages require that the main process
relinquishes the event-loop to allow the `stdout.on()` handler to
run. This results in none of the messages being written when ngcc
is run in `--no-async` mode, and some messages failing to be
written if the main process is killed (e.g. ctrl-C).
It appears that the problem with Windows and detached processes
is known - see https://github.com/nodejs/node/issues/3596#issuecomment-250890218.
But in the meantime, this commit is a workaround, where non-Windows
`inherit` the main process `stdout` while on Windows it reverts
to the async handler approach, which is better than nothing.
PR Close#36637
Prior to this commit, the unknown property check was unnecessarily invoked for AOT-compiled components (for these components, the check happens at compile time). This commit updates the code to avoid unknown property verification for AOT-compiled components by checking whether schemas information is present (as a way to detect whether this is JIT or AOT compiled component).
Resolves#35945.
PR Close#36072
Prior to this change, there was a problem while matching template attributes, which mistakenly took i18n attributes (that might be present in attrs array after template ones) into account. This commit updates the logic to avoid template attribute matching logic from entering the i18n section and as a result this also allows generating proper i18n attributes sections instead of keeping these attribute in plain form (with their values) in attribute arrays.
PR Close#36422
In certain use-cases it's useful to have an ability to use empty strings as translations. Currently Ivy fails at runtime if empty string is used as a translation, since some parts of internal data structures are not created properly. This commit updates runtime i18n logic to handle empty translations and avoid unnecessary extra processing for such cases.
Fixes#36476.
PR Close#36499
