The current ngcc lock-file strategy spawns a new process in order to
capture a potential `SIGINT` and remove the lock-file. For more
information see #35861.
Previously, this unlocker process was spawned as soon as the `LockFile`
was instantiated in order to have it available as soon as possible
(given that spawning a process is an asynchronous operation). Since the
`LockFile` was instantiated and passed to the `Executor`, this meant
that an unlocker process was spawned for each cluster worker, when
running ngcc in parallel mode. These processes were not needed, since
the `LockFile` was not used in cluster workers, but we still had to pay
the overhead of each process' own memory and V8 instance.
(NOTE: This overhead was small compared to the memory consumed by ngcc's
normal operations, but still unnecessary.)
This commit avoids the extra processes by only spawning an unlocker
process when running on the cluster master process and not on worker
processes.
PR Close#36569
Previously ngcc never preserved whitespaces but this is at odds
with how the ViewEngine compiler works. In ViewEngine, library
templates are recompiled with the current application's tsconfig
settings, which meant that whitespace preservation could be set
in the application tsconfig file.
This commit allows ngcc to use the `preserveWhitespaces` setting
from tsconfig when compiling library templates. One should be aware
that this disallows different projects with different tsconfig settings
to share the same node_modules folder, with regard to whitespace
preservation. But this is already the case in the current ngcc since
this configuration is hard coded right now.
Fixes#35871
PR Close#36189
When computing the dependencies between packages which are not in
node_modules, we may need to rely upon path-mappings to find the path
to the imported entry-point.
This commit allows ngcc to use the path-mappings from a tsconfig
file to find dependencies. By default any tsconfig.json file in the directory
above the `basePath` is loaded but it is possible to use a path to a
specific file by providing the `tsConfigPath` property to mainNgcc,
or to turn off loading any tsconfig file by setting `tsConfigPath` to `null`.
At the command line this is controlled via the `--tsconfig` option.
Fixes#36119
PR Close#36180
When two entry-points overlap, ngcc may attempt to process some
files twice. Previously, when this occured ngcc would just exit with an
error preventing any other entry-points from being processed.
This commit changes ngcc so that if `errorOnFailedEntryPoint` is false, it will
simply log an error and continue to process entry-points. This is useful when
ngcc is processing the entire node_modules folder and there are some invalid
entry-points that the project doesn't actually use.
PR Close#36083
Previously, when an entry-point contained code that caused its compilation
to fail, ngcc would exit in the middle of processing, possibly leaving other
entry-points in a corrupt state.
This change adds a new `errorOnFailedEntryPoint` option to `mainNgcc` that
specifies whether ngcc should exit immediately or log an error and continue
processing other entry-points.
The default is `false` so that ngcc will not error but continue processing
as much as possible. This is useful in post-install hooks, and async CLI
integration, where we do not have as much control over which entry-points
should be processed.
The option is forced to true if the `targetEntryPointPath` is provided,
such as the sync integration with the CLI, since in that case it is targeting
an entry-point that will actually be used in the current project so we do want
ngcc to exit with an error at that point.
PR Close#36083
Later when we implement the ability to continue processing when tasks have
failed to compile, we will also need to avoid processing tasks that depend
upon the failed task.
This refactoring exposes this list of dependent tasks in a way that can be
used to skip processing of tasks that depend upon a failed task.
It also changes the blocking model of the parallel mode of operation so
that non-typings tasks are now blocked on their corresponding typings task.
Previously the non-typings tasks could be triggered to run in parallel to
the typings task, since they do not have a hard dependency on each other,
but this made it difficult to skip task correctly if the typings task failed,
since it was possible that a non-typings task was already in flight when
the typings task failed. The result of this is a small potential degradation
of performance in async parallel processing mode, in the rare cases that
there were not enough unblocked tasks to make use of all the available
workers.
PR Close#36083
Moving the definition of the `onTaskCompleted` callback into `mainNgcc()`
allows it to be configured based on options passed in there more easily.
This will be the case when we want to configure whether to log or throw
an error for tasks that failed to be processed successfully.
This commit also creates two new folders and moves the code around a bit
to make it easier to navigate the code§:
* `execution/tasks`: specific helpers such as task completion handlers
* `execution/tasks/queues`: the `TaskQueue` implementations and helpers
PR Close#36083
In some scenarios it is useful for the developer to indicate
to ngcc that it should not use the entry-point manifest
file, and instead write a new one.
In the ngcc command line tool, this option is set by specfying
```
--invalidate-entry-point-manifest
```
PR Close#35931
The `DirectoryWalkerEntryPointFinder` has to traverse the
entire node_modules library everytime it executes in order to
identify the entry-points that need to be processed. This is
very time consuming (several seconds for big projects on
Windows).
This commit changes the `DirectoryWalkerEntryPointFinder` to
use the `EntryPointManifest` to store the paths to entry-points
that were found when doing this initial node_modules traversal
in a file to be reused for subsequent calls.
This dramatically speeds up ngcc processing when it has been run once
already.
PR Close#35931
This version of `LockFile` creates an "unlocker" child-process that monitors
the main ngcc process and deletes the lock file if it exits unexpectedly.
This resolves the issue where the main process could not be killed by pressing
Ctrl-C at the terminal.
Fixes#35761
PR Close#35861
The previous implementation mixed up the management
of locking a piece of code (both sync and async) with the
management of writing and removing the lockFile that is
used as the flag for which process has locked the code.
This change splits these two concepts up. Apart from
avoiding the awkward base class it allows the `LockFile`
implementation to be replaced cleanly.
PR Close#35861
This commit adds a new ngcc configuration, `ignorableDeepImportMatchers`
for packages. This is a list of regular expressions matching deep imports
that can be safely ignored from that package. Deep imports that are not
ignored cause a warning to be logged.
// FW-1892
Fixes#35615
PR Close#35683
ngcc uses a lockfile to prevent two ngcc instances from executing at the
same time. Previously, if a lockfile was found the current process would
error and exit.
Now, when in async mode, the current process is able to wait for the previous
process to release the lockfile before continuing itself.
PR Close#35131
To support parallel CLI builds we instruct developers to pre-process
their node_modules via ngcc at the command line.
Despite doing this ngcc was still trying to set a lock when it was being
triggered by the CLI for packages that are not going to be processed,
since they are not compiled by Angular for instance.
This commit checks whether a target package needs to be compiled
at all before attempting to set the lock.
Fixes#35000
PR Close#35057
If ngcc gets updated to a new version then the artifacts
left in packages that were processed by the previous
version are possibly invalid.
Previously we just errored if we found packages that
had already been processed by an outdated version.
Now we automatically clean the packages that have
outdated artifacts so that they can be reprocessed
correctly with the current ngcc version.
Fixes#35082
PR Close#35079
Now `hasBeenProcessed()` will no longer throw if there
is an entry-point that has been built with an outdated
version of ngcc.
Instead it just returns `false`, which will include it in this
processing run.
This is a precursor to adding functionality that will
automatically revert outdate build artifacts.
PR Close#35079
The Angular CLI will continue to call ngcc on all possible packages, even if they
have already been processed by ngcc in a postinstall script.
In a parallel build environment, this was causing ngcc to complain that it was
being run in more than one process at the same time.
This commit moves the check for whether the targeted package has been
processed outside the locked code section, since there is no issue with
multiple ngcc processes from doing this check.
PR Close#34722
Previously, it was possible for multiple instance of ngcc to be running
at the same time, but this is not supported and can cause confusing and
flakey errors at build time.
Now, only one instance of ngcc can run at a time. If a second instance
tries to execute it fails with an appropriate error message.
See https://github.com/angular/angular/issues/32431#issuecomment-571825781
PR Close#34722
ngcc computes a dependency graph of entry-points to ensure that
entry-points are processed in the correct order. Previously only the imports
in source files were analysed to determine the dependencies for each
entry-point.
This is not sufficient when an entry-point has a "type-only" dependency
- for example only importing an interface from another entry-point.
In this case the "type-only" import does not appear in the
source code. It only appears in the typings files. This can cause a
dependency to be missed on the entry-point.
This commit fixes this by additionally processing the imports in the
typings program, as well as the source program.
Note that these missing dependencies could cause unexpected flakes when
running ngcc in async mode on multiple processes due to the way that
ngcc caches files when they are first read from disk.
Fixes#34411
// FW-1781
PR Close#34494
By ensuring that legacy i18n message ids are rendered into the templates
of components for packages processed by ngcc, we ensure that these packages
can be used in an application that may provide translations in a legacy
format.
Fixes#34056
PR Close#34135
Placing this configuration in to the bundle avoids having to pass the
value around through lots of function calls, but also could enable
support for different behaviour per bundle in the future.
PR Close#34135
Previously, the list of missing dependencies was not explicitly joined,
which resulted in the default `,` joiner being used during
stringification.
This commit explicitly joins the missing dependency lines to avoid
unnecessary commas.
Before:
```
The target entry-point "some-entry-point" has missing dependencies:
- dependency 1
, - dependency 2
, - dependency 3
```
After:
```
The target entry-point "some-entry-point" has missing dependencies:
- dependency 1
- dependency 2
- dependency 3
```
PR Close#33139
This gives an overview of how much time is spent in each operation/phase
and makes it easy to do rough comparisons of how different
configurations or changes affect performance.
PR Close#32427
`ngcc` supports both synchronous and asynchronous execution. The default
mode when using `ngcc` programmatically (which is how `@angular/cli` is
using it) is synchronous. When running `ngcc` from the command line
(i.e. via the `ivy-ngcc` script), it runs in async mode.
Previously, the work would be executed in the same way in both modes.
This commit improves the performance of `ngcc` in async mode by
processing tasks in parallel on multiple processes. It uses the Node.js
built-in [`cluster` module](https://nodejs.org/api/cluster.html) to
launch a cluster of Node.js processes and take advantage of multi-core
systems.
Preliminary comparisons indicate a 1.8x to 2.6x speed improvement when
processing the angular.io app (apparently depending on the OS, number of
available cores, system load, etc.). Further investigation is needed to
better understand these numbers and identify potential areas of
improvement.
Inspired by/Based on @alxhub's prototype: alxhub/angular@cb631bdb1
Original design doc: https://hackmd.io/uYG9CJrFQZ-6FtKqpnYJAA?view
Jira issue: [FW-1460](https://angular-team.atlassian.net/browse/FW-1460)
PR Close#32427
This change does not alter the current behavior, but makes it easier to
introduce `TaskQueue`s implementing different task selection algorithms,
for example to support executing multiple tasks in parallel (while
respecting interdependencies between them).
Inspired by/Based on @alxhub's prototype: alxhub/angular@cb631bdb1
PR Close#32427
Previously, `ngcc` needed to store some metadata related to the
processing of each entry-point. This metadata was stored in a `Map`, in
the form of `EntryPointProcessingMetadata` and passed around as needed.
After some recent refactorings, it turns out that this metadata (with
its only remaining property, `hasProcessedTypings`) was no longer used,
because the relevant information was extracted from other sources (such
as the `processDts` property on `Task`s).
This commit cleans up the code by removing the unused code and types.
PR Close#32427
In the past, a task's processability didn't use to be known in advance.
It was possible that a task would be created and added to the queue
during the analysis phase and then later (during the compilation phase)
it would be found out that the task (i.e. the associated format
property) was not processable.
As a result, certain checks had to be delayed, until a task's processing
had started or even until all tasks had been processed. Examples of
checks that had to be delayed are:
- Whether a task can be skipped due to `compileAllFormats: false`.
- Whether there were entry-points for which no format at all was
successfully processed.
It turns out that (as made clear by the refactoring in 9537b2ff8), once
a task starts being processed it is expected to either complete
successfully (with the associated format being processed) or throw an
error (in which case the process will exit). In other words, a task's
processability is known in advance.
This commit takes advantage of this fact by moving certain checks
earlier in the process (e.g. in the analysis phase instead of the
compilation phase), which in turn allows avoiding some unnecessary work.
More specifically:
- When `compileAllFormats` is `false`, tasks are created _only_ for the
first suitable format property for each entry-point, since the rest of
the tasks would have been skipped during the compilation phase anyway.
This has the following advantages:
1. It avoids the slight overhead of generating extraneous tasks and
then starting to process them (before realizing they should be
skipped).
2. In a potential future parallel execution mode, unnecessary tasks
might start being processed at the same time as the first (useful)
task, even if their output would be later discarded, wasting
resources. Alternatively, extra logic would have to be added to
prevent this from happening. The change in this commit avoids these
issues.
- When an entry-point is not processable, an error will be thrown
upfront without having to wait for other tasks to be processed before
failing.
PR Close#32427
Previously, `ngcc`'s programmatic API would run and complete
synchronously. This was necessary for specific usecases (such as how the
`@angular/cli` invokes `ngcc` as part of the TypeScript module
resolution process), but not for others (e.g. running `ivy-ngcc` as a
`postinstall` script).
This commit adds a new option (`async`) that enables turning on
asynchronous execution. I.e. it signals that the caller is OK with the
function call to complete asynchronously, which allows `ngcc` to
potentially run in a more efficient mode.
Currently, there is no difference in the way tasks are executed in sync
vs async mode, but this change sets the ground for adding new execution
options (that require asynchronous operation), such as processing tasks
in parallel on multiple processes.
NOTE:
When using the programmatic API, the default value for `async` is
`false`, thus retaining backwards compatibility.
When running `ngcc` from the command line (i.e. via the `ivy-ngcc`
script), it runs in async mode (to be able to take advantage of future
optimizations), but that is transparent to the caller.
PR Close#32427
This change does not alter the current behavior, but makes it easier to
introduce new types of `Executors` , for example to do the required work
in parallel (on multiple processes).
Inspired by/Based on @alxhub's prototype: alxhub/angular@cb631bdb1
PR Close#32427
To persist some of its state, `ngcc` needs to update `package.json`
files (both in memory and on disk).
This refactoring abstracts these operations behind the
`PackageJsonUpdater` interface, making it easier to orchestrate them
from different contexts (e.g. when running tasks in parallel on multiple
processes).
Inspired by/Based on @alxhub's prototype: alxhub/angular@cb631bdb1
PR Close#32427
