perf(compiler-cli): fix regressions in incremental program reuse (#37641)

Commit 4213e8d5 introduced shim reference tagging into the compiler, and
changed how the `TypeCheckProgramHost` worked under the hood during the
creation of a template type-checking program. This work enabled a more
incremental flow for template type-checking, but unintentionally introduced
several regressions in performance, caused by poor incrementality during
`ts.Program` creation.

1. The `TypeCheckProgramHost` was made to rely on the `ts.CompilerHost` to
   retrieve instances of `ts.SourceFile`s from the original program. If the
   host does not return the original instance of such files, but instead
   creates new instances, this has two negative effects: it incurs
   additional parsing time, and it interferes with TypeScript's ability to
   reuse information about such files.

2. During the incremental creation of a `ts.Program`, TypeScript compares
   the `referencedFiles` of `ts.SourceFile` instances from the old program
   with those in the new program. If these arrays differ, TypeScript cannot
   fully reuse the old program. The implementation of reference tagging
   introduced in 4213e8d5 restores the original `referencedFiles` array
   after a `ts.Program` is created, which means that future incremental
   operations involving that program will always fail this comparison,
   effectively limiting the incrementality TypeScript can achieve.

Problem 1 exacerbates problem 2: if a new `ts.SourceFile` is created by the
host after shim generation has been disabled, it will have an untagged
`referencedFiles` array even if the original file's `referencedFiles` was
not restored, triggering problem 2 when creating the template type-checking
program.

To fix these issues, `referencedFiles` arrays are now restored on the old
`ts.Program` prior to the creation of a new incremental program. This allows
TypeScript to get the most out of reusing the old program's data.

Additionally, the `TypeCheckProgramHost` now uses the original `ts.Program`
to retrieve original instances of `ts.SourceFile`s where possible,
preventing issues when a host would otherwise return fresh instances.

Together, these fixes ensure that program reuse is as incremental as
possible, and tests have been added to verify this for certain scenarios.

An optimization was further added to prevent the creation of a type-checking
`ts.Program` in the first place if no type-checking is necessary.

PR Close #37641

packages/compiler-cli/src/ngtsc/program.ts

@@ -17,6 +17,7 @@ import {NgCompilerOptions} from './core/api';
 import {TrackedIncrementalBuildStrategy} from './incremental';
 import {IndexedComponent} from './indexer';
 import {NOOP_PERF_RECORDER, PerfRecorder, PerfTracker} from './perf';
+import {retagAllTsFiles, untagAllTsFiles} from './shims';
 import {ReusedProgramStrategy} from './typecheck';
 
 
@@ -68,14 +69,26 @@ export class NgtscProgram implements api.Program {
     }
     this.closureCompilerEnabled = !!options.annotateForClosureCompiler;
 
-    const reuseProgram = oldProgram && oldProgram.reuseTsProgram;
+    const reuseProgram = oldProgram?.reuseTsProgram;
     this.host = NgCompilerHost.wrap(delegateHost, rootNames, options, reuseProgram ?? null);
 
+    if (reuseProgram !== undefined) {
+      // Prior to reusing the old program, restore shim tagging for all its `ts.SourceFile`s.
+      // TypeScript checks the `referencedFiles` of `ts.SourceFile`s for changes when evaluating
+      // incremental reuse of data from the old program, so it's important that these match in order
+      // to get the most benefit out of reuse.
+      retagAllTsFiles(reuseProgram);
+    }
+
     this.tsProgram = ts.createProgram(this.host.inputFiles, options, this.host, reuseProgram);
     this.reuseTsProgram = this.tsProgram;
 
     this.host.postProgramCreationCleanup();
 
+    // Shim tagging has served its purpose, and tags can now be removed from all `ts.SourceFile`s in
+    // the program.
+    untagAllTsFiles(this.tsProgram);
+
     const reusedProgramStrategy = new ReusedProgramStrategy(
         this.tsProgram, this.host, this.options, this.host.shimExtensionPrefixes);
 
@@ -93,6 +106,10 @@ export class NgtscProgram implements api.Program {
     return this.tsProgram;
   }
 
+  getReuseTsProgram(): ts.Program {
+    return this.reuseTsProgram;
+  }
+
   getTsOptionDiagnostics(cancellationToken?: ts.CancellationToken|
                          undefined): readonly ts.Diagnostic[] {
     return this.tsProgram.getOptionsDiagnostics(cancellationToken);
@@ -248,6 +265,7 @@ export class NgtscProgram implements api.Program {
       }));
       this.perfRecorder.stop(fileEmitSpan);
     }
+
     this.perfRecorder.stop(emitSpan);
 
     if (this.perfTracker !== null && this.options.tracePerformance !== undefined) {