Shim generation was built on a lie.
Shims are files added to the program which aren't original files authored by
the user, but files authored effectively by the compiler. These fall into
two categories: files which will be generated (like the .ngfactory shims we
generate for View Engine compatibility) as well as files used internally in
compilation (like the __ng_typecheck__.ts file).
Previously, shim generation was driven by the `rootFiles` passed to the
compiler as input. These are effectively the `files` listed in the
`tsconfig.json`. Each shim generator (e.g. the `FactoryGenerator`) would
examine the `rootFiles` and produce a list of shim file names which it would
be responsible for generating. These names would then be added to the
`rootFiles` when the program was created.
The fatal flaw here is that `rootFiles` does not always account for all of
the files in the program. In fact, it's quite rare that it does. Users don't
typically specify every file directly in `files`. Instead, they rely on
TypeScript, during program creation, starting with a few root files and
transitively discovering all of the files in the program.
This happens, however, during `ts.createProgram`, which is too late to add
new files to the `rootFiles` list.
As a result, shim generation was only including shims for files actually
listed in the `tsconfig.json` file, and not for the transitive set of files
in the user's program as it should.
This commit completely rewrites shim generation to use a different technique
for adding files to the program, inspired by View Engine's shim generator.
In this new technique, as the program is being created and `ts.SourceFile`s
are being requested from the `NgCompilerHost`, shims for those files are
generated and a reference to them is patched onto the original file's
`ts.SourceFile.referencedFiles`. This causes TS to think that the original
file references the shim, and causes the shim to be included in the program.
The original `referencedFiles` array is saved and restored after program
creation, hiding this little hack from the rest of the system.
The new shim generation engine differentiates between two kinds of shims:
top-level shims (such as the flat module entrypoint file and
__ng_typecheck__.ts) and per-file shims such as ngfactory or ngsummary
files. The former are included via `rootFiles` as before, the latter are
included via the `referencedFiles` of their corresponding original files.
As a result of this change, shims are now correctly generated for all files
in the program, not just the ones named in `tsconfig.json`.
A few mitigating factors prevented this bug from being realized until now:
* in g3, `files` does include the transitive closure of files in the program
* in CLI apps, shims are not really used
This change also makes use of a novel technique for associating information
with source files: the use of an `NgExtension` `Symbol` to patch the
information directly onto the AST object. This is used in several
circumstances:
* For shims, metadata about a `ts.SourceFile`'s status as a shim and its
origins are held in the extension data.
* For original files, the original `referencedFiles` are stashed in the
extension data for later restoration.
The main benefit of this technique is a lot less bookkeeping around `Map`s
of `ts.SourceFile`s to various kinds of data, which need to be tracked/
invalidated as part of incremental builds.
This technique is based on designs used internally in the TypeScript
compiler and is serving as a prototype of this design in ngtsc. If it works
well, it could have benefits across the rest of the compiler.
PR Close#36211
Currently, when Angular code is built with Bazel and with Ivy, generated
factory shims (.ngfactory files) are not processed via the majority of
tsickle's transforms. This is a subtle effect of the build infrastructure,
but it boils down to a TsickleHost method `shouldSkipTsickleProcessing`.
For ngc_wrapped builds (Bazel + Angular), this method is defined in the
`@bazel/typescript` (aka bazel rules_typescript) implementation of
`CompilerHost`. The default behavior is to skip tsickle processing for files
which are not present in the original `srcs[]` of the build rule. In
Angular's case, this includes all generated shim files.
For View Engine factories this is probably desirable as they're quite
complex and they've never been tested with tsickle. Ivy factories however
are smaller and very straightforward, and it makes sense to treat them like
any other output.
This commit adjusts two independent implementations of
`shouldSkipTsickleProcessing` to enable transformation of Ivy shims:
* in `@angular/bazel` aka ngc_wrapped, the upstream `@bazel/typescript`
`CompilerHost` is patched to treat .ngfactory files the same as their
original source file, with respect to tsickle processing.
It is currently not possible to test this change as we don't have any test
that inspects tsickle output with bazel. It will be extensively tested in
g3.
* in `ngc`, Angular's own implementation is adjusted to allow for the
processing of shims when compiling with Ivy. This enables a unit test to
be written to validate the correct behavior of tsickle when given a host
that's appropriately configured to process factory shims.
For ngtsc-as-a-plugin, a similar fix will need to be submitted upstream in
tsc_wrapped.
PR Close#35848
PR Close#35975
This commit adds support in the Angular monorepo and in the Angular
compiler(s) for TypeScript 3.8. All packages can now compile with
TS 3.8.
For most of the repo, only a handful few typings adjustments were needed:
* TS 3.8 has a new `CustomElementConstructor` DOM type, which enforces a
zero-argument constructor. The `NgElementConstructor` type previously
declared a required `injector` argument despite the fact that its
implementation allowed `injector` to be optional. The interface type was
updated to reflect the optionality of the argument.
* Certain error messages were changed, and expectations in tests were
updated as a result.
* tsserver (part of language server) now returns performance information in
responses, so test expectations were changed to only assert on the actual
body content of responses.
For compiler-cli and schematics (which use the TypeScript AST) a major
breaking change was the introduction of the export form:
```typescript
export * as foo from 'bar';
```
This is a `ts.NamespaceExport`, and the `exportClause` of a
`ts.ExportDeclaration` can now take this type as well as `ts.NamedExports`.
This broke a lot of places where `exportClause` was assumed to be
`ts.NamedExports`.
For the most part these breakages were in cases where it is not necessary
to handle the new `ts.NamedExports` anyway. ngtsc's design uses the
`ts.TypeChecker` APIs to understand syntax and so automatically supports the
new form of exports.
The View Engine compiler on the other hand extracts TS structures into
metadata.json files, and that format was not designed for namespaced
exports. As a result it will take a nontrivial amount of work if we want to
support such exports in View Engine. For now, these new exports are not
accounted for in metadata.json, and so using them in "folded" Angular
expressions will result in errors (probably claiming that the referenced
exported namespace doesn't exist).
Care was taken to only use TS APIs which are present in 3.7/3.6, as Angular
needs to remain compatible with these for the time being.
This commit does not update angular.io.
PR Close#35864
This commit moves the calculation of `ignoreFiles` - the set of files to be
ignored by a consumer of the `NgCompiler` API - from its `prepareEmit`
operation to its initialization. It's now available as a field on
`NgCompiler`.
This will allow a consumer to skip gathering diagnostics for `ignoreFiles`
as well as skip emit.
PR Close#34792
Previously, NgtscProgram lived in the main @angular/compiler-cli package
alongside the legacy View Engine compiler. As a result, the main package
depended on all of the ngtsc internal packages, and a significant portion of
ngtsc logic lived in NgtscProgram.
This commit refactors NgtscProgram and moves the main logic of compilation
into a new 'core' package. The new package defines a new API which enables
implementers of TypeScript compilers (compilers built using the TS API) to
support Angular transpilation as well. It involves a new NgCompiler type
which takes a ts.Program and performs Angular analysis and transformations,
as well as an NgCompilerHost which wraps an input ts.CompilerHost and adds
any extra Angular files.
Together, these two classes are used to implement a new NgtscProgram which
adapts the legacy api.Program interface used by the View Engine compiler
onto operations on the new types. The new NgtscProgram implementation is
significantly smaller and easier to reason about.
The new NgCompilerHost replaces the previous GeneratedShimsHostWrapper which
lived in the 'shims' package.
A new 'resource' package is added to support the HostResourceLoader which
previously lived in the outer compiler package.
As a result of the refactoring, the dependencies of the outer
@angular/compiler-cli package on ngtsc internal packages are significantly
trimmed.
This refactoring was driven by the desire to build a plugin interface to the
compiler so that tsc_wrapped (another consumer of the TS compiler APIs) can
perform Angular transpilation on user request.
PR Close#34887
The major one that affects the angular repo is the removal of the bootstrap attribute in nodejs_binary, nodejs_test and jasmine_node_test in favor of using templated_args --node_options=--require=/path/to/script. The side-effect of this is that the bootstrap script does not get the require.resolve patches with explicitly loading the targets _loader.js file.
PR Close#34736
The major one that affects the angular repo is the removal of the bootstrap attribute in nodejs_binary, nodejs_test and jasmine_node_test in favor of using templated_args --node_options=--require=/path/to/script. The side-effect of this is that the bootstrap script does not get the require.resolve patches with explicitly loading the targets _loader.js file.
PR Close#34589
Previously 'analyze' in the various `DecoratorHandler`s not only extracts
information from the decorators on the classes being analyzed, but also has
several side effects within the compiler:
* it can register metadata about the types involved in global metadata
trackers.
* it can register information about which .ngfactory symbols are actually
needed.
In this commit, these side-effects are moved into a new 'register' phase,
which runs after the 'analyze' step. Currently this is a no-op refactoring
as 'register' is always called directly after 'analyze'. In the future this
opens the door for re-use of prior analysis work (with only 'register' being
called, to apply the above side effects).
Also as part of this refactoring, the reification of NgModule scope
information into the incremental dependency graph is moved to the
`NgtscProgram` instead of the `TraitCompiler` (which now only manages trait
compilation and does not have other side effects).
PR Close#34288
Similar to https://github.com/angular/angular/pull/33633, this commit is
needed to fix an outage with the Angular Kythe indexer.
Crash logs:
```
TypeError: Cannot read property 'text' of undefined
at NodeObject.getFullText (typescript/stable/lib/typescript.js:121443:57)
at FactoryGenerator.generate (angular2/rc/packages/compiler-cli/src/ngtsc/shims/src/factory_generator.ts:67:34)
at GeneratedShimsHostWrapper.getSourceFile (angular2/rc/packages/compiler-cli/src/ngtsc/shims/src/host.ts:88:26)
at findSourceFile (typescript/stable/lib/typescript.js:90654:29)
at typescript/stable/lib/typescript.js:90553:85
at getSourceFileFromReferenceWorker (typescript/stable/lib/typescript.js:90520:34)
at processSourceFile (typescript/stable/lib/typescript.js:90553:13)
at processRootFile (typescript/stable/lib/typescript.js:90383:13)
at typescript/stable/lib/typescript.js:89399:60
at Object.forEach (typescript/stable/lib/typescript.js:280:30)
```
PR Close#33660
This commit fixes a crash in the Angular Kythe indexer caused by failure
to retrieve `SourceFile` in a `Statement`.
Crash logs:
TypeError: Cannot read property 'text' of undefined
at Object.getTokenPosOfNode (typescript/stable/lib/typescript.js:8957:72)
at NodeObject.getStart (typescript/stable/lib/typescript.js:121419:23)
at NodeObject.getLeadingTriviaWidth (typescript/stable/lib/typescript.js:121439:25)
at FactoryGenerator.generate (angular2/rc/packages/compiler-cli/src/ngtsc/shims/src/factory_generator.ts:64:49)
at GeneratedShimsHostWrapper.getSourceFile (angular2/rc/packages/compiler-cli/src/ngtsc/shims/src/host.ts:88:26)
at findSourceFile (typescript/stable/lib/typescript.js:90654:29)
at typescript/stable/lib/typescript.js:90553:85
at getSourceFileFromReferenceWorker (typescript/stable/lib/typescript.js:90520:34)
at processSourceFile (typescript/stable/lib/typescript.js:90553:13)
at processRootFile (typescript/stable/lib/typescript.js:90383:13)
PR Close#33588
This PR updates Angular to compile with TypeScript 3.6 while retaining
compatibility with TS3.5. We achieve this by inserting several `as any`
casts for compatiblity around `ts.CompilerHost` APIs.
PR Close#32908
To improve cross platform support, all file access (and path manipulation)
is now done through a well known interface (`FileSystem`).
For testing a number of `MockFileSystem` implementations are provided.
These provide an in-memory file-system which emulates operating systems
like OS/X, Unix and Windows.
The current file system is always available via the static method,
`FileSystem.getFileSystem()`. This is also used by a number of static
methods on `AbsoluteFsPath` and `PathSegment`, to avoid having to pass
`FileSystem` objects around all the time. The result of this is that one
must be careful to ensure that the file-system has been initialized before
using any of these static methods. To prevent this happening accidentally
the current file system always starts out as an instance of `InvalidFileSystem`,
which will throw an error if any of its methods are called.
You can set the current file-system by calling `FileSystem.setFileSystem()`.
During testing you can call the helper function `initMockFileSystem(os)`
which takes a string name of the OS to emulate, and will also monkey-patch
aspects of the TypeScript library to ensure that TS is also using the
current file-system.
Finally there is the `NgtscCompilerHost` to be used for any TypeScript
compilation, which uses a given file-system.
All tests that interact with the file-system should be tested against each
of the mock file-systems. A series of helpers have been provided to support
such tests:
* `runInEachFileSystem()` - wrap your tests in this helper to run all the
wrapped tests in each of the mock file-systems.
* `addTestFilesToFileSystem()` - use this to add files and their contents
to the mock file system for testing.
* `loadTestFilesFromDisk()` - use this to load a mirror image of files on
disk into the in-memory mock file-system.
* `loadFakeCore()` - use this to load a fake version of `@angular/core`
into the mock file-system.
All ngcc and ngtsc source and tests now use this virtual file-system setup.
PR Close#30921
Prior to this commit there were no explicit types setup for NgModuleFactory calls in ngfactories, so TypeScript inferred the type based on a given call. In some cases (when generic types were used for Components/Directives) that turned out to be problematic, so we add explicit typing for NgModuleFactory calls.
PR Close#30708
Preserve compatibility with rollup_bundle rule.
Add missing npm dependencies, which are now enforced by the strict_deps plugin in tsc_wrapped
PR Close#30370
Now that ngtsc performs type checking using a dedicated `__ng_typecheck__.ts`
file, `NgtscProgram` always wraps its `ts.CompilerHost` in a shim host. This
shim fails to delegate `resolveModuleNames` so no custom module resolution
logic is considered. This introduces a problem for the CLI, as the compiler
host it passes kicks of ngcc for any imported module such that Ivy's
compatibility compiler runs automatically behind the scenes.
This commit adds delegation of the `resolveModuleNames` to fix the issue.
Fixes#30064
PR Close#30068
The template type-checking engine previously would assemble a type-checking
program by inserting Type Check Blocks (TCBs) into existing user files. This
approach proved expensive, as TypeScript has to re-parse and re-type-check
those files when processing the type-checking program.
Instead, a far more performant approach is to augment the program with a
single type-checking file, into which all TCBs are generated. Additionally,
type constructors are also inlined into this file.
This is not always possible - both TCBs and type constructors can sometimes
require inlining into user code, particularly if bound generic type
parameters are present, so the approach taken is actually a hybrid. These
operations are inlined if necessary, but are otherwise generated in a single
file.
It is critically important that the original program also include an empty
version of the type-checking file, otherwise the shape of the two programs
will be different and TypeScript will throw away all the old program
information. This leads to a painfully slow type checking pass, on the same
order as the original program creation. A shim to generate this file in the
original program is therefore added.
Testing strategy: this commit is largely a refactor with no externally
observable behavioral differences, and thus no tests are needed.
PR Close#29698
Currently when building an Angular project with `ngtsc`
and `flatModuleOutFile` enabled, the Ngtsc build will fail
if there are multiple source files as root file names.
Ngtsc and NGC currently determine the entry-point for multiple
root file names by looking for files ending with `/index.ts`.
This functionality is technically deprecated, but still supported
and currently breaks on Windows as the root file names are not
guaranteed to be normalized POSIX-like paths.
In order to make this logic more reliable in the future, this commit
also switches the shim generators and entry-point logic to the branded
path types. This ensures that we don't break this in the future.
PR Close#29453
ngsummary files were generated with an export for each class declaration.
However, some Angular code declares classes (class Foo) and exports them
(export {Foo}) separately, which was causing incomplete summary files.
This commit expands the set of symbol names for which summary exports will
be generated, fixing this issue.
PR Close#29193
When ngtsc generates a .ngfactory shim, it does so based on the contents of
an original file in the program. Occasionally these original files have
comments at the top which are load-bearing (e.g. they contain jsdoc
annotations which are significant to downstream bundling tools). The
generated factory shims should preserve this comment.
This commit adds a step to the ngfactory generator to preserve the top-level
comment from the original source file.
FW-1006 #resolve
FW-1095 #resolve
PR Close#29065
This change is kind of similar to #27466, but instead of ensuring that
these shims can be generated, we also need to make sure that developers
are able to also use the factory shims like with `ngc`.
This issue is now surfacing because we have various old examples which
are now also built with `ngtsc` (due to the bazel migration). On case insensitive
platforms (e.g. windows) these examples cannot be built because ngtsc fails
the app imports a generated shim file (such as the factory shim files).
This is because the `GeneratedShimsHostWrapper` TypeScript host uses
the `getCanonicalFileName` method in order to check whether a given
file/module exists in the generator file maps. e.g.
```
// Generator Map:
'C:/users/paul/_bazel_paul/lm3s4mgv/execroot/angular/packages/core/index.ngfactory.ts' =>
'C:/users/paul/_bazel_paul/lm3s4mgv/execroot/angular/packages/core/index.ts',
// Path passed into `fileExists`
C:/users/paul/_bazel_paul/lm3s4mgv/execroot/angular/packages/core/index.ngfactory.ts
// After getCanonicalFileName (notice the **lower-case drive name**)
c:/users/paul/_bazel_paul/lm3s4mgv/execroot/angular/packages/core/index.ngfactory.ts
```
As seen above, the generator map does not use the canonical file names, as well as
TypeScript internally does not pass around canonical file names. We can fix this by removing
the manual call to `getCanonicalFileName` and just following TypeScript internal-semantics.
PR Close#28831
At the moment, paths stored in `maps` are not normalized and in Windows is causing files not to be found when enabling factory shimming.
For example, the map contents will be
```
Map {
'C:\\git\\cli-repos\\ng-factory-shims\\index.ngfactory.ts' => 'C:\\git\\cli-repos\\ng-factory-shims\\index.ts' }
```
However, ts compiler normalized the paths and is causing;
```
error TS6053: File 'C:/git/cli-repos/ng-factory-shims/index.ngfactory.ts' not found.
error TS6053: File 'C:/git/cli-repos/ng-factory-shims/index.ngsummary.ts' not found.
```
The changes normalized the paths that are stored within the factory and summary maps.
PR Close#28006
Generated factory shims can import from @angular/core. However, we have
special logic in place to rewrite self-imports when generating code for
@angular/core.
This commit leverages the new standalone ImportRewriter interface to
properly rewrite imports in generated factory shims. Before this fix,
a generated factory file for core would look like:
```typescript
import * as i0 from './r3_symbols';
export var ApplicationModuleNgFactory = new ɵNgModuleFactory(...);
```
This is invalid, as ɵNgModuleFactory is just NgModuleFactory when imported
via r3_symbols.
FW-881 #resolve
PR Close#27998
This refactoring moves code around between a few of the ngtsc subpackages,
with the goal of having a more logical package structure. Additional
interfaces are also introduced where they make sense.
The 'metadata' package formerly contained both the partial evaluator,
the TypeScriptReflectionHost as well as some other reflection functions,
and the Reference interface and various implementations. This package
was split into 3 parts.
The partial evaluator now has its own package 'partial_evaluator', and
exists behind an interface PartialEvaluator instead of a top-level
function. In the future this will be useful for reducing churn as the
partial evaluator becomes more complicated.
The TypeScriptReflectionHost and other miscellaneous functions have moved
into a new 'reflection' package. The former 'host' package which contained
the ReflectionHost interface and associated types was also merged into this
new 'reflection' package.
Finally, the Reference APIs were moved to the 'imports' package, which will
consolidate all import-related logic in ngtsc.
PR Close#27743
This commit moves the FlatIndexGenerator to its own package, in preparation
to expand its capabilities and support re-exporting of private declarations
from NgModules.
PR Close#27743
Previously, ngtsc did not respect the angularCompilerOptions settings
for generating flat module indices. This commit adds a
FlatIndexGenerator which is used to implement those options.
FW-738 #resolve
PR Close#27497
Previously the ngtsc ShimGenerator interface expected that all shims would
be generated using the contents of existing ts.SourceFiles. This assumption
was true for ngfactory and ngsummary files, but breaks down for flat module
index files, which are standalone.
This commit prepares for flat module index generation by enabling shim
generators which don't require an existing file.
PR Close#27497
Common insensitive platforms are `win32/win64` (see:
[here](3e4c5c95ab/src/compiler/sys.ts (L681-L682)))
Currently when running `bazel build packages/core --define=compile=aot`, the `compiler-cli` will throw because it cannot find the `index.ngfactory.ts` file in the compiler host. This is because the shim host wrapper is not properly generating the requested `ngfactory` file.
This happens because we call `getCanonicalFileName` that returns a path that is different to the actual program filenames that are used to construct a map of generated files. Since the generators always use the paths which are not "canonical" and pases them internally like that, we can just stop manually calling `getCanonicalFileName`.
PR Close#27466
ngfactory files have a ɵNonEmptyModule constant included if there are no
other exported factory symbols. Previously this extra export was added
dynamically in a TS transformer.
However, synthetically constructed exports don't get properly downleveled
during JS emit, and this generated constant caused issues with downstream
tests.
Instead, this commit configures the shim to always have this export to
begin with, and to filter it out if it's not required.
Testing strategy: covered by existing ngtsc_spec tests which verify the
presence of the ɵNonEmptyModule symbol.
PR Close#27483
In ngtsc, files loaded into the ts.Program have a "module name", set via
ts.SourceFile.moduleName, which ends up being written into an AMD module
name triple-slash directive in the generated .js file.
For generated shim files (ngfactories, ngsummaries) that are constructed
synthetically, there was previously no moduleName set, which caused some
issues with downstream tests.
This commit adds logic to compute and set moduleNames for both generated
ngfactory and ngsummary shims.
PR Close#27483
The method `ts.CompilerHost.directoryExists` is optional, and was not
previously handled by our ts.CompilerHost wrapper for factory and
summary shims (GeneratedShimsHostWrapper).
TypeScript checks for the existence of this method and silently ignores
things like typeRoots if it's not found. This commit adds proper handling
of directoryExists() to the shim.
A test is also added which verifies typeRoots behavior works when shims
are enabled.
PR Close#27470
Previously the ngfactory shim generator in ngtsc would always write two
imports in the factory file shims:
1) an import to @angular/core
2) an import to the base file
If the base file has no exports, import #2 would be empty. This turns out
to cause issues downstream.
This commit changes the generated shim so if there are no exports in the
base file, the generated shim is empty too.
PR Close#27470
This commit adds generation of .ngsummary.js shims alongside .ngfactory.js
shims when generated files are enabled.
Generated .ngsummary shims contain a single, null export for every exported
class with decorators that exists in the original source files. Ivy code
does not depend on summaries, so these exist only as a placeholder to allow
them to be imported and their values passed to old APIs. This preserves
backwards compatibility.
Testing strategy: this commit adds a compiler test to verify the correct
shape and contents of the generated .ngsummary.js files.
PR Close#26495
This commit refactors the shim host to be agnostic to the shims being
generated, and provides an API for generating additional shims besides
the .ngfactory.js. This will be used in a following commit to generate
.ngsummary.js shims.
Testing strategy: this refactor introduces no new behavior, so it's
sufficient that the existing tests for factory shim generation continue
to pass.
PR Close#26495
This simple refactor of the build rules renames the .ngfactory.js shim
generator to 'shims' instead of 'factories', in preparation for adding
.ngsummary.js shim generation.
Testing strategy: this commit does not introduce any new behavior and
merely moves files and symbols around. It's sufficient that the existing
ngtsc tests pass.
PR Close#26495
