refactor: move angular source to /packages rather than modules/@angular

packages/benchpress/README.md

@@ -0,0 +1,305 @@
+# Benchpress
+
+Benchpress is a framework for e2e performance tests.
+See [here for an example project](https://github.com/angular/benchpress-tree).
+
+The sources for this package are in the main [Angular](https://github.com/angular/angular) repo. Please file issues and pull requests against that repo.
+
+License: MIT
+
+# Why?
+
+There are so called "micro benchmarks" that essentially use a stop watch in the browser to measure time
+(e.g. via `performance.now()`). This approach is limited to time, and in some cases memory
+(Chrome with special flags), as metric. It does not allow to measure:
+
+- rendering time: e.g. the time the browser spends to layout or paint elements. This can e.g. used to
+  test the performance impact of stylesheet changes.
+- garbage collection: e.g. how long the browser paused script execution, and how much memory was collected.
+  This can be used to stabilize script execution time, as garbage collection times are usually very
+  unpredictable. This data can also be used to measure and improve memory usage of applications,
+  as the garbage collection amount directly affects garbage collection time.
+- distinguish script execution time from waiting: e.g. to measure the client side only time that is spent
+  in a complex user interaction, ignoring backend calls.
+- measure fps to assert the smoothness of scrolling and animations.
+
+This kind of data is already available in the DevTools of modern browsers. However, there is no standard way to
+use those tools in an automated way to measure web app performance, especially not across platforms.
+
+Benchpress tries to fill this gap, i.e. allow to access all kinds of performance metrics in an automated way.
+
+
+# How it works
+
+Benchpress uses webdriver to read out the so called "performance log" of browsers. This contains all kinds of interesting
+data, e.g. when a script started/ended executing, gc started/ended, the browser painted something to the screen, ...
+
+As browsers are different, benchpress has plugins to normalizes these events.
+
+
+# Features
+
+* Provides a loop (so called "Sampler") that executes the benchmark multiple times
+* Automatically waits/detects until the browser is "warm"
+* Reporters provide a normalized way to store results:
+  - console reporter
+  - file reporter
+  - Google Big Query reporter (coming soon)
+* Supports micro benchmarks as well via `console.time()` / `console.timeEnd()`
+  - `console.time()` / `console.timeEnd()` mark the timeline in the DevTools, so it makes sense
+    to use them in micro benchmark to visualize and understand them, with or without benchpress.
+  - running micro benchmarks in benchpress leverages the already existing reporters,
+    the sampler and the auto warmup feature of benchpress.
+
+
+# Supported browsers
+
+* Chrome on all platforms
+* Mobile Safari (iOS)
+* Firefox (work in progress)
+
+
+# How to write a benchmark
+
+A benchmark in benchpress is made by an application under test
+and a benchmark driver. The application under test is the
+actual application consisting of html/css/js that should be tests.
+A benchmark driver is a webdriver test that interacts with the
+application under test.
+
+
+## A simple benchmark
+
+Let's assume we want to measure the script execution time, as well as the render time
+that it takes to fill a container element with a complex html string.
+
+The application under test could look like this:
+
+```
+index.html:
+
+<button id="reset" onclick="reset()">Reset</button>
+<button id="fill" onclick="fill()">fill innerHTML</button>
+<div id="container"></div>
+<script>
+  var container = document.getElementById('container');
+  var complexHtmlString = '...'; // TODO
+
+  function reset() { container.innerHTML = ''; }
+
+  function fill() {
+    container.innerHTML = complexHtmlString;
+  }
+</script>
+```
+
+A benchmark driver could look like this:
+
+```
+// A runner contains the shared configuration
+// and can be shared across multiple tests.
+var runner = new Runner(...);
+
+driver.get('http://myserver/index.html');
+
+var resetBtn = driver.findElement(By.id('reset'));
+var fillBtn = driver.findElement(By.id('fill'));
+
+runner.sample({
+  id: 'fillElement',
+  // Prepare is optional...
+  prepare: () {
+    resetBtn.click();
+  },
+  execute: () {
+    fillBtn.click();
+    // Note: if fillBtn would use some asynchronous code,
+    // we would need to wait here for its end.
+  }
+});
+```
+
+## Measuring in the browser
+
+If the application under test would like to, it can measure on its own.
+E.g.
+
+```
+index.html:
+
+<button id="measure" onclick="measure()">Measure document.createElement</button>
+<script>
+  function measure() {
+    console.time('createElement*10000');
+    for (var i=0; i<100000; i++) {
+      document.createElement('div');
+    }
+    console.timeEnd('createElement*10000');
+  }
+</script>
+```
+
+When the `measure` button is clicked, it marks the timeline and creates 10000 elements.
+It uses the special names `createElement*10000` to tell benchpress that the
+time that was measured is for 10000 calls to createElement and that benchpress should
+take the average for it.
+
+A test driver for this would look like this:
+
+````
+driver.get('.../index.html');
+
+var measureBtn = driver.findElement(By.id('measure'));
+runner.sample({
+  id: 'createElement test',
+  microMetrics: {
+    'createElement': 'time to create an element (ms)'
+  },
+  execute: () {
+    measureBtn.click();
+  }
+});
+````
+
+When looking into the DevTools Timeline, we see a marker as well:
+![Marked Timeline](docs/marked_timeline.png)
+
+### Custom Metrics Without Using `console.time`
+
+It's also possible to measure any "user metric" within the browser
+by setting a numeric value on the `window` object. For example:
+
+```js
+bootstrap(App)
+  .then(() => {
+    window.timeToBootstrap = Date.now() - performance.timing.navigationStart;
+  });
+```
+
+A test driver for this user metric could be written as follows:
+
+```js
+
+describe('home page load', function() {
+  it('should log load time for a 2G connection', done => {
+    runner.sample({
+      execute: () => {
+        browser.get(`http://localhost:8080`);
+      },
+      userMetrics: {
+        timeToBootstrap: 'The time in milliseconds to bootstrap'
+      },
+      providers: [
+        {provide: RegressionSlopeValidator.METRIC, useValue: 'timeToBootstrap'}
+      ]
+    }).then(done);
+  });
+});
+```
+
+Using this strategy, benchpress will wait until the specified property name,
+`timeToBootstrap` in this case, is defined as a number on the `window` object
+inside the application under test.
+
+# Smoothness Metrics
+
+Benchpress can also measure the "smoothness" of scrolling and animations. In order to do that, the following set of metrics can be collected by benchpress:
+
+- `frameTime.mean`: mean frame time in ms (target: 16.6ms for 60fps)
+- `frameTime.worst`: worst frame time in ms
+- `frameTime.best`: best frame time in ms
+- `frameTime.smooth`: percentage of frames that hit 60fps
+
+To collect these metrics, you need to execute `console.time('frameCapture')` and `console.timeEnd('frameCapture')` either in your benchmark application or in you benchmark driver via webdriver. The metrics mentioned above will only be collected between those two calls and it is recommended to wrap the time/timeEnd calls as closely as possible around the action you want to evaluate to get accurate measurements.
+
+In addition to that, one extra provider needs to be passed to benchpress in tests that want to collect these metrics:
+
+    benchpress.sample(providers: [{provide: bp.Options.CAPTURE_FRAMES, useValue: true}], ... )
+
+# Requests Metrics
+
+Benchpress can also record the number of requests sent and count the received "encoded" bytes since [window.performance.timing.navigationStart](http://www.w3.org/TR/navigation-timing/#dom-performancetiming-navigationstart):
+
+- `receivedData`: number of bytes received since the last navigation start
+- `requestCount`: number of requests sent since the last navigation start
+
+To collect these metrics, you need the following corresponding extra providers:
+
+    benchpress.sample(providers: [
+      {provide: bp.Options.RECEIVED_DATA, useValue: true},
+      {provide: bp.Options.REQUEST_COUNT, useValue: true}
+    ], ... )
+
+# Best practices
+
+* Use normalized environments
+  - metrics that are dependent on the performance of the execution environment must be executed on a normalized machine
+  - e.g. a real mobile device whose cpu frequency is set to a fixed value.
+      * see our [build script](https://github.com/angular/angular/blob/master/scripts/ci/android_cpu.sh)
+      * this requires root access, e.g. via a userdebug build of Android on a Google Nexus device
+        (see [here](https://source.android.com/source/building-running.html) and [here](https://source.android.com/source/building-devices.html#obtaining-proprietary-binaries))
+  - e.g. a calibrated machine that does not run background jobs, has a fixed cpu frequency, ...
+
+* Use relative comparisons
+  - relative comparisons are less likely to change over time and help to interpret the results of benchmarks
+  - e.g. compare an example written using a ui framework against a hand coded example and track the ratio
+
+* Assert post-commit for commit ranges
+  - running benchmarks can take some time. Running them before every commit is usually too slow.
+  - when a regression is detected for a commit range, use bisection to find the problematic commit
+
+* Repeat benchmarks multiple times in a fresh window
+  - run the same benchmark multiple times in a fresh window and then take the minimal average value of each benchmark run
+
+* Use force gc with care
+  - forcing gc can skew the script execution time and gcTime numbers,
+    but might be needed to get stable gc time / gc amount numbers
+
+* Open a new window for every test
+  - browsers (e.g. chrome) might keep JIT statistics over page reloads and optimize pages differently depending on what has been loaded before
+
+# Detailed overview
+
+![Overview](docs/overview.png)
+
+Definitions:
+
+* valid sample: a sample that represents the world that should be measured in a good way.
+* complete sample: sample of all measure values collected so far
+
+Components:
+
+* Runner
+  - contains a default configuration
+  - creates a new injector for every sample call, via which all other components are created
+
+* Sampler
+  - gets data from the metrics
+  - reports measure values immediately to the reporters
+  - loops until the validator is able to extract a valid sample out of the complete sample (see below).
+  - reports the valid sample and the complete sample to the reporters
+
+* Metric
+  - gets measure values from the browser
+  - e.g. reads out performance logs, DOM values, JavaScript values
+
+* Validator
+  - extracts a valid sample out of the complete sample of all measure values.
+  - e.g. wait until there are 10 samples and take them as valid sample (would include warmup time)
+  - e.g. wait until the regression slope for the metric `scriptTime` through the last 10 measure values is >=0, i.e. the values for the `scriptTime` metric are no more decreasing
+
+* Reporter
+  - reports measure values, the valid sample and the complete sample to backends
+  - e.g. a reporter that prints to the console, a reporter that reports values into Google BigQuery, ...
+
+* WebDriverAdapter
+  - abstraction over the used web driver client
+  - one implementation for every webdriver client
+    E.g. one for selenium-webdriver Node.js module, dart async webdriver, dart sync webdriver, ...
+
+* WebDriverExtension
+  - implements additional methods that are standardized in the webdriver protocol using the WebDriverAdapter
+  - provides functionality like force gc, read out performance logs in a normalized format
+  - one implementation per browser, e.g. one for Chrome, one for mobile Safari, one for Firefox
+
+

packages/benchpress/index.ts

@@ -0,0 +1,34 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+// Must be imported first, because angular2 decorators throws on load.
+import 'reflect-metadata';
+
+export {InjectionToken, Injector, Provider, ReflectiveInjector} from '@angular/core';
+export {Options} from './src/common_options';
+export {MeasureValues} from './src/measure_values';
+export {Metric} from './src/metric';
+export {MultiMetric} from './src/metric/multi_metric';
+export {PerflogMetric} from './src/metric/perflog_metric';
+export {UserMetric} from './src/metric/user_metric';
+export {Reporter} from './src/reporter';
+export {ConsoleReporter} from './src/reporter/console_reporter';
+export {JsonFileReporter} from './src/reporter/json_file_reporter';
+export {MultiReporter} from './src/reporter/multi_reporter';
+export {Runner} from './src/runner';
+export {SampleDescription} from './src/sample_description';
+export {SampleState, Sampler} from './src/sampler';
+export {Validator} from './src/validator';
+export {RegressionSlopeValidator} from './src/validator/regression_slope_validator';
+export {SizeValidator} from './src/validator/size_validator';
+export {WebDriverAdapter} from './src/web_driver_adapter';
+export {PerfLogEvent, PerfLogFeatures, WebDriverExtension} from './src/web_driver_extension';
+export {ChromeDriverExtension} from './src/webdriver/chrome_driver_extension';
+export {FirefoxDriverExtension} from './src/webdriver/firefox_driver_extension';
+export {IOsDriverExtension} from './src/webdriver/ios_driver_extension';
+export {SeleniumWebDriverAdapter} from './src/webdriver/selenium_webdriver_adapter';

packages/benchpress/package.json

@@ -0,0 +1,31 @@
+{
+  "name": "@angular/benchpress",
+  "version": "0.1.0",
+  "description": "Benchpress - a framework for e2e performance tests",
+  "main": "index.js",
+  "typings": "./typings/index.d.ts",
+  "dependencies": {
+    "@angular/core": "^2.0.0-rc.7",
+    "reflect-metadata": "^0.1.2",
+    "rxjs": "^5.0.1",
+    "jpm": "1.1.4",
+    "firefox-profile": "0.4.0",
+    "selenium-webdriver": "^2.53.3"
+  },
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/angular/angular.git"
+  },
+  "keywords": [
+    "angular",
+    "benchmarks"
+  ],
+  "contributors": [
+    "Tobias Bosch <tbosch@google.com> (https://angular.io/)"
+  ],
+  "license": "MIT",
+  "bugs": {
+    "url": "https://github.com/angular/angular/issues"
+  },
+  "homepage": "https://github.com/angular/angular/tree/master/modules/@angular/compiler-cli"
+}

packages/benchpress/publish.sh

@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+
+set -ex
+
+cd $(dirname $0)/../../..
+ROOTDIR=$(pwd)
+SRCDIR=${ROOTDIR}/modules/@angular/benchpress
+DESTDIR=${ROOTDIR}/dist/packages-dist/benchpress
+
+rm -fr ${DESTDIR}
+
+echo "====== BUILDING... ====="
+./build.sh --packages=core,benchpress --bundle=false
+
+echo "====== PUBLISHING: ${DESTDIR} ====="
+npm publish ${DESTDIR} --access public

packages/benchpress/src/common_options.ts

@@ -0,0 +1,53 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {InjectionToken} from '@angular/core';
+import * as fs from 'fs';
+
+export class Options {
+  static SAMPLE_ID = new InjectionToken('Options.sampleId');
+  static DEFAULT_DESCRIPTION = new InjectionToken('Options.defaultDescription');
+  static SAMPLE_DESCRIPTION = new InjectionToken('Options.sampleDescription');
+  static FORCE_GC = new InjectionToken('Options.forceGc');
+  static NO_PREPARE = () => true;
+  static PREPARE = new InjectionToken('Options.prepare');
+  static EXECUTE = new InjectionToken('Options.execute');
+  static CAPABILITIES = new InjectionToken('Options.capabilities');
+  static USER_AGENT = new InjectionToken('Options.userAgent');
+  static MICRO_METRICS = new InjectionToken('Options.microMetrics');
+  static USER_METRICS = new InjectionToken('Options.userMetrics');
+  static NOW = new InjectionToken('Options.now');
+  static WRITE_FILE = new InjectionToken('Options.writeFile');
+  static RECEIVED_DATA = new InjectionToken('Options.receivedData');
+  static REQUEST_COUNT = new InjectionToken('Options.requestCount');
+  static CAPTURE_FRAMES = new InjectionToken('Options.frameCapture');
+  static DEFAULT_PROVIDERS = [
+    {provide: Options.DEFAULT_DESCRIPTION, useValue: {}},
+    {provide: Options.SAMPLE_DESCRIPTION, useValue: {}},
+    {provide: Options.FORCE_GC, useValue: false},
+    {provide: Options.PREPARE, useValue: Options.NO_PREPARE},
+    {provide: Options.MICRO_METRICS, useValue: {}}, {provide: Options.USER_METRICS, useValue: {}},
+    {provide: Options.NOW, useValue: () => new Date()},
+    {provide: Options.RECEIVED_DATA, useValue: false},
+    {provide: Options.REQUEST_COUNT, useValue: false},
+    {provide: Options.CAPTURE_FRAMES, useValue: false},
+    {provide: Options.WRITE_FILE, useValue: writeFile}
+  ];
+}
+
+function writeFile(filename: string, content: string): Promise<any> {
+  return new Promise(function(resolve, reject) {
+    fs.writeFile(filename, content, (error) => {
+      if (error) {
+        reject(error);
+      } else {
+        resolve();
+      }
+    });
+  });
+}

packages/benchpress/src/firefox_extension/.gitignore

@@ -0,0 +1,2 @@
+*.xpi
+addon-sdk*

packages/benchpress/src/firefox_extension/data/installed_script.ts

@@ -0,0 +1,38 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+declare var exportFunction: any;
+declare var unsafeWindow: any;
+
+exportFunction(function() {
+  const curTime = unsafeWindow.performance.now();
+  (<any>self).port.emit('startProfiler', curTime);
+}, unsafeWindow, {defineAs: 'startProfiler'});
+
+exportFunction(function() {
+  (<any>self).port.emit('stopProfiler');
+}, unsafeWindow, {defineAs: 'stopProfiler'});
+
+exportFunction(function(cb: Function) {
+  (<any>self).port.once('perfProfile', cb);
+  (<any>self).port.emit('getProfile');
+}, unsafeWindow, {defineAs: 'getProfile'});
+
+exportFunction(function() {
+  (<any>self).port.emit('forceGC');
+}, unsafeWindow, {defineAs: 'forceGC'});
+
+exportFunction(function(name: string) {
+  const curTime = unsafeWindow.performance.now();
+  (<any>self).port.emit('markStart', name, curTime);
+}, unsafeWindow, {defineAs: 'markStart'});
+
+exportFunction(function(name: string) {
+  const curTime = unsafeWindow.performance.now();
+  (<any>self).port.emit('markEnd', name, curTime);
+}, unsafeWindow, {defineAs: 'markEnd'});

packages/benchpress/src/firefox_extension/lib/main.ts

@@ -0,0 +1,78 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+const {Cc, Ci, Cu} = require('chrome');
+const os = Cc['@mozilla.org/observer-service;1'].getService(Ci.nsIObserverService);
+const ParserUtil = require('./parser_util');
+
+class Profiler {
+  private _profiler: any;
+  private _markerEvents: any[];
+  private _profilerStartTime: number;
+
+  constructor() { this._profiler = Cc['@mozilla.org/tools/profiler;1'].getService(Ci.nsIProfiler); }
+
+  start(entries: any, interval: any, features: any, timeStarted: any) {
+    this._profiler.StartProfiler(entries, interval, features, features.length);
+    this._profilerStartTime = timeStarted;
+    this._markerEvents = [];
+  }
+
+  stop() { this._profiler.StopProfiler(); }
+
+  getProfilePerfEvents() {
+    const profileData = this._profiler.getProfileData();
+    let perfEvents = ParserUtil.convertPerfProfileToEvents(profileData);
+    perfEvents = this._mergeMarkerEvents(perfEvents);
+    perfEvents.sort(function(event1: any, event2: any) {
+      return event1.ts - event2.ts;
+    });  // Sort by ts
+    return perfEvents;
+  }
+
+  /** @internal */
+  private _mergeMarkerEvents(perfEvents: any[]): any[] {
+    this._markerEvents.forEach(function(markerEvent) { perfEvents.push(markerEvent); });
+    return perfEvents;
+  }
+
+  addStartEvent(name: string, timeStarted: number) {
+    this._markerEvents.push({ph: 'B', ts: timeStarted - this._profilerStartTime, name: name});
+  }
+
+  addEndEvent(name: string, timeEnded: number) {
+    this._markerEvents.push({ph: 'E', ts: timeEnded - this._profilerStartTime, name: name});
+  }
+}
+
+function forceGC() {
+  Cu.forceGC();
+  os.notifyObservers(null, 'child-gc-request', null);
+};
+
+const mod = require('sdk/page-mod');
+const data = require('sdk/self').data;
+const profiler = new Profiler();
+mod.PageMod({
+  include: ['*'],
+  contentScriptFile: data.url('installed_script.js'),
+  onAttach: (worker: any) => {
+    worker.port.on(
+        'startProfiler',
+        (timeStarted: any) => profiler.start(
+            /* = profiler memory */ 3000000, 0.1, ['leaf', 'js', 'stackwalk', 'gc'], timeStarted));
+    worker.port.on('stopProfiler', () => profiler.stop());
+    worker.port.on(
+        'getProfile', () => worker.port.emit('perfProfile', profiler.getProfilePerfEvents()));
+    worker.port.on('forceGC', forceGC);
+    worker.port.on(
+        'markStart', (name: string, timeStarted: any) => profiler.addStartEvent(name, timeStarted));
+    worker.port.on(
+        'markEnd', (name: string, timeEnded: any) => profiler.addEndEvent(name, timeEnded));
+  }
+});

packages/benchpress/src/firefox_extension/lib/parser_util.ts

@@ -0,0 +1,92 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+/**
+ * @param {Object} perfProfile The perf profile JSON object.
+ * @return {Object[]} An array of recognized events that are captured
+ *     within the perf profile.
+ */
+export function convertPerfProfileToEvents(perfProfile: any): any[] {
+  const inProgressEvents = new Map();                 // map from event name to start time
+  const finishedEvents: {[key: string]: any}[] = [];  // Event[] finished events
+  const addFinishedEvent = function(eventName: string, startTime: number, endTime: number) {
+    const categorizedEventName = categorizeEvent(eventName);
+    let args: {[key: string]: any} = undefined;
+    if (categorizedEventName == 'gc') {
+      // TODO: We cannot measure heap size at the moment
+      args = {usedHeapSize: 0};
+    }
+    if (startTime == endTime) {
+      // Finished instantly
+      finishedEvents.push({ph: 'X', ts: startTime, name: categorizedEventName, args: args});
+    } else {
+      // Has duration
+      finishedEvents.push({ph: 'B', ts: startTime, name: categorizedEventName, args: args});
+      finishedEvents.push({ph: 'E', ts: endTime, name: categorizedEventName, args: args});
+    }
+  };
+
+  const samples = perfProfile.threads[0].samples;
+  // In perf profile, firefox samples all the frames in set time intervals. Here
+  // we go through all the samples and construct the start and end time for each
+  // event.
+  for (let i = 0; i < samples.length; ++i) {
+    const sample = samples[i];
+    const sampleTime = sample.time;
+
+    // Add all the frames into a set so it's easier/faster to find the set
+    // differences
+    const sampleFrames = new Set();
+    sample.frames.forEach(function(frame: {[key: string]: any}) {
+      sampleFrames.add(frame['location']);
+    });
+
+    // If an event is in the inProgressEvents map, but not in the current sample,
+    // then it must have just finished. We add this event to the finishedEvents
+    // array and remove it from the inProgressEvents map.
+    const previousSampleTime = (i == 0 ? /* not used */ -1 : samples[i - 1].time);
+    inProgressEvents.forEach(function(startTime, eventName) {
+      if (!(sampleFrames.has(eventName))) {
+        addFinishedEvent(eventName, startTime, previousSampleTime);
+        inProgressEvents.delete(eventName);
+      }
+    });
+
+    // If an event is in the current sample, but not in the inProgressEvents map,
+    // then it must have just started. We add this event to the inProgressEvents
+    // map.
+    sampleFrames.forEach(function(eventName) {
+      if (!(inProgressEvents.has(eventName))) {
+        inProgressEvents.set(eventName, sampleTime);
+      }
+    });
+  }
+
+  // If anything is still in progress, we need to included it as a finished event
+  // since recording ended.
+  const lastSampleTime = samples[samples.length - 1].time;
+  inProgressEvents.forEach(function(startTime, eventName) {
+    addFinishedEvent(eventName, startTime, lastSampleTime);
+  });
+
+  // Remove all the unknown categories.
+  return finishedEvents.filter(function(event) { return event['name'] != 'unknown'; });
+}
+
+// TODO: this is most likely not exhaustive.
+export function categorizeEvent(eventName: string): string {
+  if (eventName.indexOf('PresShell::Paint') > -1) {
+    return 'render';
+  } else if (eventName.indexOf('FirefoxDriver.prototype.executeScript') > -1) {
+    return 'script';
+  } else if (eventName.indexOf('forceGC') > -1) {
+    return 'gc';
+  } else {
+    return 'unknown';
+  }
+}

packages/benchpress/src/firefox_extension/lib/test_helper.ts

@@ -0,0 +1,51 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+const q = require('q');
+const FirefoxProfile = require('firefox-profile');
+const jpm = require('jpm/lib/xpi');
+const pathUtil = require('path');
+
+const PERF_ADDON_PACKAGE_JSON_DIR = '..';
+
+exports.getAbsolutePath = function(path: string) {
+  const normalizedPath = pathUtil.normalize(path);
+  if (pathUtil.resolve(normalizedPath) == normalizedPath) {
+    // Already absolute path
+    return normalizedPath;
+  } else {
+    return pathUtil.join(__dirname, normalizedPath);
+  }
+};
+
+exports.getFirefoxProfile = function(extensionPath: string) {
+  const deferred = q.defer();
+
+  const firefoxProfile = new FirefoxProfile();
+  firefoxProfile.addExtensions([extensionPath], () => {
+    firefoxProfile.encoded((encodedProfile: any) => {
+      const multiCapabilities = [{browserName: 'firefox', firefox_profile: encodedProfile}];
+      deferred.resolve(multiCapabilities);
+    });
+  });
+
+  return deferred.promise;
+};
+
+exports.getFirefoxProfileWithExtension = function() {
+  const absPackageJsonDir = pathUtil.join(__dirname, PERF_ADDON_PACKAGE_JSON_DIR);
+  const packageJson = require(pathUtil.join(absPackageJsonDir, 'package.json'));
+
+  const savedCwd = process.cwd();
+  process.chdir(absPackageJsonDir);
+
+  return jpm(packageJson).then((xpiPath: string) => {
+    process.chdir(savedCwd);
+    return exports.getFirefoxProfile(xpiPath);
+  });
+};

packages/benchpress/src/firefox_extension/package.json

@@ -0,0 +1 @@
+{ "version" : "0.0.1", "main" : "lib/main.js", "name" : "ffperf-addon" }

packages/benchpress/src/measure_values.ts

@@ -0,0 +1,20 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+export class MeasureValues {
+  constructor(
+      public runIndex: number, public timeStamp: Date, public values: {[key: string]: any}) {}
+
+  toJson() {
+    return {
+      'timeStamp': this.timeStamp.toJSON(),
+      'runIndex': this.runIndex,
+      'values': this.values,
+    };
+  }
+}

packages/benchpress/src/metric.ts

@@ -0,0 +1,31 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+
+/**
+ * A metric is measures values
+ */
+export abstract class Metric {
+  /**
+   * Starts measuring
+   */
+  beginMeasure(): Promise<any> { throw new Error('NYI'); }
+
+  /**
+   * Ends measuring and reports the data
+   * since the begin call.
+   * @param restart: Whether to restart right after this.
+   */
+  endMeasure(restart: boolean): Promise<{[key: string]: any}> { throw new Error('NYI'); }
+
+  /**
+   * Describes the metrics provided by this metric implementation.
+   * (e.g. units, ...)
+   */
+  describe(): {[key: string]: string} { throw new Error('NYI'); }
+}

packages/benchpress/src/metric/multi_metric.ts

@@ -0,0 +1,63 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {InjectionToken, Injector} from '@angular/core';
+
+import {Metric} from '../metric';
+
+export class MultiMetric extends Metric {
+  static provideWith(childTokens: any[]): any[] {
+    return [
+      {
+        provide: _CHILDREN,
+        useFactory: (injector: Injector) => childTokens.map(token => injector.get(token)),
+        deps: [Injector]
+      },
+      {
+        provide: MultiMetric,
+        useFactory: (children: Metric[]) => new MultiMetric(children),
+        deps: [_CHILDREN]
+      }
+    ];
+  }
+
+  constructor(private _metrics: Metric[]) { super(); }
+
+  /**
+   * Starts measuring
+   */
+  beginMeasure(): Promise<any> {
+    return Promise.all(this._metrics.map(metric => metric.beginMeasure()));
+  }
+
+  /**
+   * Ends measuring and reports the data
+   * since the begin call.
+   * @param restart: Whether to restart right after this.
+   */
+  endMeasure(restart: boolean): Promise<{[key: string]: any}> {
+    return Promise.all(this._metrics.map(metric => metric.endMeasure(restart)))
+        .then(values => mergeStringMaps(<any>values));
+  }
+
+  /**
+   * Describes the metrics provided by this metric implementation.
+   * (e.g. units, ...)
+   */
+  describe(): {[key: string]: any} {
+    return mergeStringMaps(this._metrics.map((metric) => metric.describe()));
+  }
+}
+
+function mergeStringMaps(maps: {[key: string]: string}[]): {[key: string]: string} {
+  const result: {[key: string]: string} = {};
+  maps.forEach(map => { Object.keys(map).forEach(prop => { result[prop] = map[prop]; }); });
+  return result;
+}
+
+const _CHILDREN = new InjectionToken('MultiMetric.children');

packages/benchpress/src/metric/perflog_metric.ts

@@ -0,0 +1,371 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {Inject, Injectable, InjectionToken} from '@angular/core';
+
+import {Options} from '../common_options';
+import {Metric} from '../metric';
+import {PerfLogEvent, PerfLogFeatures, WebDriverExtension} from '../web_driver_extension';
+
+
+/**
+ * A metric that reads out the performance log
+ */
+@Injectable()
+export class PerflogMetric extends Metric {
+  static SET_TIMEOUT = new InjectionToken('PerflogMetric.setTimeout');
+  static PROVIDERS = [
+    PerflogMetric, {
+      provide: PerflogMetric.SET_TIMEOUT,
+      useValue: (fn: Function, millis: number) => <any>setTimeout(fn, millis)
+    }
+  ];
+
+  private _remainingEvents: PerfLogEvent[];
+  private _measureCount: number;
+  private _perfLogFeatures: PerfLogFeatures;
+
+  /**
+   * @param driverExtension
+   * @param setTimeout
+   * @param microMetrics Name and description of metrics provided via console.time / console.timeEnd
+   **/
+  constructor(
+      private _driverExtension: WebDriverExtension,
+      @Inject(PerflogMetric.SET_TIMEOUT) private _setTimeout: Function,
+      @Inject(Options.MICRO_METRICS) private _microMetrics: {[key: string]: string},
+      @Inject(Options.FORCE_GC) private _forceGc: boolean,
+      @Inject(Options.CAPTURE_FRAMES) private _captureFrames: boolean,
+      @Inject(Options.RECEIVED_DATA) private _receivedData: boolean,
+      @Inject(Options.REQUEST_COUNT) private _requestCount: boolean) {
+    super();
+
+    this._remainingEvents = [];
+    this._measureCount = 0;
+    this._perfLogFeatures = _driverExtension.perfLogFeatures();
+    if (!this._perfLogFeatures.userTiming) {
+      // User timing is needed for navigationStart.
+      this._receivedData = false;
+      this._requestCount = false;
+    }
+  }
+
+  describe(): {[key: string]: string} {
+    const res: {[key: string]: any} = {
+      'scriptTime': 'script execution time in ms, including gc and render',
+      'pureScriptTime': 'script execution time in ms, without gc nor render'
+    };
+    if (this._perfLogFeatures.render) {
+      res['renderTime'] = 'render time in ms';
+    }
+    if (this._perfLogFeatures.gc) {
+      res['gcTime'] = 'gc time in ms';
+      res['gcAmount'] = 'gc amount in kbytes';
+      res['majorGcTime'] = 'time of major gcs in ms';
+      if (this._forceGc) {
+        res['forcedGcTime'] = 'forced gc time in ms';
+        res['forcedGcAmount'] = 'forced gc amount in kbytes';
+      }
+    }
+    if (this._receivedData) {
+      res['receivedData'] = 'encoded bytes received since navigationStart';
+    }
+    if (this._requestCount) {
+      res['requestCount'] = 'count of requests sent since navigationStart';
+    }
+    if (this._captureFrames) {
+      if (!this._perfLogFeatures.frameCapture) {
+        const warningMsg = 'WARNING: Metric requested, but not supported by driver';
+        // using dot syntax for metric name to keep them grouped together in console reporter
+        res['frameTime.mean'] = warningMsg;
+        res['frameTime.worst'] = warningMsg;
+        res['frameTime.best'] = warningMsg;
+        res['frameTime.smooth'] = warningMsg;
+      } else {
+        res['frameTime.mean'] = 'mean frame time in ms (target: 16.6ms for 60fps)';
+        res['frameTime.worst'] = 'worst frame time in ms';
+        res['frameTime.best'] = 'best frame time in ms';
+        res['frameTime.smooth'] = 'percentage of frames that hit 60fps';
+      }
+    }
+    for (const name in this._microMetrics) {
+      res[name] = this._microMetrics[name];
+    }
+    return res;
+  }
+
+  beginMeasure(): Promise<any> {
+    let resultPromise = Promise.resolve(null);
+    if (this._forceGc) {
+      resultPromise = resultPromise.then((_) => this._driverExtension.gc());
+    }
+    return resultPromise.then((_) => this._beginMeasure());
+  }
+
+  endMeasure(restart: boolean): Promise<{[key: string]: number}> {
+    if (this._forceGc) {
+      return this._endPlainMeasureAndMeasureForceGc(restart);
+    } else {
+      return this._endMeasure(restart);
+    }
+  }
+
+  /** @internal */
+  private _endPlainMeasureAndMeasureForceGc(restartMeasure: boolean) {
+    return this._endMeasure(true).then((measureValues) => {
+      // disable frame capture for measurements during forced gc
+      const originalFrameCaptureValue = this._captureFrames;
+      this._captureFrames = false;
+      return this._driverExtension.gc()
+          .then((_) => this._endMeasure(restartMeasure))
+          .then((forceGcMeasureValues) => {
+            this._captureFrames = originalFrameCaptureValue;
+            measureValues['forcedGcTime'] = forceGcMeasureValues['gcTime'];
+            measureValues['forcedGcAmount'] = forceGcMeasureValues['gcAmount'];
+            return measureValues;
+          });
+    });
+  }
+
+  private _beginMeasure(): Promise<any> {
+    return this._driverExtension.timeBegin(this._markName(this._measureCount++));
+  }
+
+  private _endMeasure(restart: boolean): Promise<{[key: string]: number}> {
+    const markName = this._markName(this._measureCount - 1);
+    const nextMarkName = restart ? this._markName(this._measureCount++) : null;
+    return this._driverExtension.timeEnd(markName, nextMarkName)
+        .then((_) => this._readUntilEndMark(markName));
+  }
+
+  private _readUntilEndMark(
+      markName: string, loopCount: number = 0, startEvent: PerfLogEvent = null) {
+    if (loopCount > _MAX_RETRY_COUNT) {
+      throw new Error(`Tried too often to get the ending mark: ${loopCount}`);
+    }
+    return this._driverExtension.readPerfLog().then((events) => {
+      this._addEvents(events);
+      const result = this._aggregateEvents(this._remainingEvents, markName);
+      if (result) {
+        this._remainingEvents = events;
+        return result;
+      }
+      let resolve: (result: any) => void;
+      const promise = new Promise(res => { resolve = res; });
+      this._setTimeout(() => resolve(this._readUntilEndMark(markName, loopCount + 1)), 100);
+      return promise;
+    });
+  }
+
+  private _addEvents(events: PerfLogEvent[]) {
+    let needSort = false;
+    events.forEach(event => {
+      if (event['ph'] === 'X') {
+        needSort = true;
+        const startEvent: PerfLogEvent = {};
+        const endEvent: PerfLogEvent = {};
+        for (const prop in event) {
+          startEvent[prop] = event[prop];
+          endEvent[prop] = event[prop];
+        }
+        startEvent['ph'] = 'B';
+        endEvent['ph'] = 'E';
+        endEvent['ts'] = startEvent['ts'] + startEvent['dur'];
+        this._remainingEvents.push(startEvent);
+        this._remainingEvents.push(endEvent);
+      } else {
+        this._remainingEvents.push(event);
+      }
+    });
+    if (needSort) {
+      // Need to sort because of the ph==='X' events
+      this._remainingEvents.sort((a, b) => {
+        const diff = a['ts'] - b['ts'];
+        return diff > 0 ? 1 : diff < 0 ? -1 : 0;
+      });
+    }
+  }
+
+  private _aggregateEvents(events: PerfLogEvent[], markName: string): {[key: string]: number} {
+    const result: {[key: string]: number} = {'scriptTime': 0, 'pureScriptTime': 0};
+    if (this._perfLogFeatures.gc) {
+      result['gcTime'] = 0;
+      result['majorGcTime'] = 0;
+      result['gcAmount'] = 0;
+    }
+    if (this._perfLogFeatures.render) {
+      result['renderTime'] = 0;
+    }
+    if (this._captureFrames) {
+      result['frameTime.mean'] = 0;
+      result['frameTime.best'] = 0;
+      result['frameTime.worst'] = 0;
+      result['frameTime.smooth'] = 0;
+    }
+    for (const name in this._microMetrics) {
+      result[name] = 0;
+    }
+    if (this._receivedData) {
+      result['receivedData'] = 0;
+    }
+    if (this._requestCount) {
+      result['requestCount'] = 0;
+    }
+
+    let markStartEvent: PerfLogEvent = null;
+    let markEndEvent: PerfLogEvent = null;
+    events.forEach((event) => {
+      const ph = event['ph'];
+      const name = event['name'];
+      if (ph === 'B' && name === markName) {
+        markStartEvent = event;
+      } else if (ph === 'I' && name === 'navigationStart') {
+        // if a benchmark measures reload of a page, use the last
+        // navigationStart as begin event
+        markStartEvent = event;
+      } else if (ph === 'E' && name === markName) {
+        markEndEvent = event;
+      }
+    });
+    if (!markStartEvent || !markEndEvent) {
+      // not all events have been received, no further processing for now
+      return null;
+    }
+
+    let gcTimeInScript = 0;
+    let renderTimeInScript = 0;
+
+    const frameTimestamps: number[] = [];
+    const frameTimes: number[] = [];
+    let frameCaptureStartEvent: PerfLogEvent = null;
+    let frameCaptureEndEvent: PerfLogEvent = null;
+
+    const intervalStarts: {[key: string]: PerfLogEvent} = {};
+    const intervalStartCount: {[key: string]: number} = {};
+
+    let inMeasureRange = false;
+    events.forEach((event) => {
+      const ph = event['ph'];
+      let name = event['name'];
+      let microIterations = 1;
+      const microIterationsMatch = name.match(_MICRO_ITERATIONS_REGEX);
+      if (microIterationsMatch) {
+        name = microIterationsMatch[1];
+        microIterations = parseInt(microIterationsMatch[2], 10);
+      }
+      if (event === markStartEvent) {
+        inMeasureRange = true;
+      } else if (event === markEndEvent) {
+        inMeasureRange = false;
+      }
+      if (!inMeasureRange || event['pid'] !== markStartEvent['pid']) {
+        return;
+      }
+
+      if (this._requestCount && name === 'sendRequest') {
+        result['requestCount'] += 1;
+      } else if (this._receivedData && name === 'receivedData' && ph === 'I') {
+        result['receivedData'] += event['args']['encodedDataLength'];
+      }
+      if (ph === 'B' && name === _MARK_NAME_FRAME_CAPUTRE) {
+        if (frameCaptureStartEvent) {
+          throw new Error('can capture frames only once per benchmark run');
+        }
+        if (!this._captureFrames) {
+          throw new Error(
+              'found start event for frame capture, but frame capture was not requested in benchpress');
+        }
+        frameCaptureStartEvent = event;
+      } else if (ph === 'E' && name === _MARK_NAME_FRAME_CAPUTRE) {
+        if (!frameCaptureStartEvent) {
+          throw new Error('missing start event for frame capture');
+        }
+        frameCaptureEndEvent = event;
+      }
+
+      if (ph === 'I' && frameCaptureStartEvent && !frameCaptureEndEvent && name === 'frame') {
+        frameTimestamps.push(event['ts']);
+        if (frameTimestamps.length >= 2) {
+          frameTimes.push(
+              frameTimestamps[frameTimestamps.length - 1] -
+              frameTimestamps[frameTimestamps.length - 2]);
+        }
+      }
+
+      if (ph === 'B') {
+        if (!intervalStarts[name]) {
+          intervalStartCount[name] = 1;
+          intervalStarts[name] = event;
+        } else {
+          intervalStartCount[name]++;
+        }
+      } else if ((ph === 'E') && intervalStarts[name]) {
+        intervalStartCount[name]--;
+        if (intervalStartCount[name] === 0) {
+          const startEvent = intervalStarts[name];
+          const duration = (event['ts'] - startEvent['ts']);
+          intervalStarts[name] = null;
+          if (name === 'gc') {
+            result['gcTime'] += duration;
+            const amount =
+                (startEvent['args']['usedHeapSize'] - event['args']['usedHeapSize']) / 1000;
+            result['gcAmount'] += amount;
+            const majorGc = event['args']['majorGc'];
+            if (majorGc && majorGc) {
+              result['majorGcTime'] += duration;
+            }
+            if (intervalStarts['script']) {
+              gcTimeInScript += duration;
+            }
+          } else if (name === 'render') {
+            result['renderTime'] += duration;
+            if (intervalStarts['script']) {
+              renderTimeInScript += duration;
+            }
+          } else if (name === 'script') {
+            result['scriptTime'] += duration;
+          } else if (this._microMetrics[name]) {
+            (<any>result)[name] += duration / microIterations;
+          }
+        }
+      }
+    });
+
+    if (frameCaptureStartEvent && !frameCaptureEndEvent) {
+      throw new Error('missing end event for frame capture');
+    }
+    if (this._captureFrames && !frameCaptureStartEvent) {
+      throw new Error('frame capture requested in benchpress, but no start event was found');
+    }
+    if (frameTimes.length > 0) {
+      this._addFrameMetrics(result, frameTimes);
+    }
+    result['pureScriptTime'] = result['scriptTime'] - gcTimeInScript - renderTimeInScript;
+    return result;
+  }
+
+  private _addFrameMetrics(result: {[key: string]: number}, frameTimes: any[]) {
+    result['frameTime.mean'] = frameTimes.reduce((a, b) => a + b, 0) / frameTimes.length;
+    const firstFrame = frameTimes[0];
+    result['frameTime.worst'] = frameTimes.reduce((a, b) => a > b ? a : b, firstFrame);
+    result['frameTime.best'] = frameTimes.reduce((a, b) => a < b ? a : b, firstFrame);
+    result['frameTime.smooth'] =
+        frameTimes.filter(t => t < _FRAME_TIME_SMOOTH_THRESHOLD).length / frameTimes.length;
+  }
+
+  private _markName(index: number) { return `${_MARK_NAME_PREFIX}${index}`; }
+}
+
+const _MICRO_ITERATIONS_REGEX = /(.+)\*(\d+)$/;
+
+const _MAX_RETRY_COUNT = 20;
+const _MARK_NAME_PREFIX = 'benchpress';
+
+const _MARK_NAME_FRAME_CAPUTRE = 'frameCapture';
+// using 17ms as a somewhat looser threshold, instead of 16.6666ms
+const _FRAME_TIME_SMOOTH_THRESHOLD = 17;

packages/benchpress/src/metric/user_metric.ts

@@ -0,0 +1,69 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {Inject, Injectable} from '@angular/core';
+
+import {Options} from '../common_options';
+import {Metric} from '../metric';
+import {WebDriverAdapter} from '../web_driver_adapter';
+
+@Injectable()
+export class UserMetric extends Metric {
+  static PROVIDERS = [UserMetric];
+
+  constructor(
+      @Inject(Options.USER_METRICS) private _userMetrics: {[key: string]: string},
+      private _wdAdapter: WebDriverAdapter) {
+    super();
+  }
+
+  /**
+   * Starts measuring
+   */
+  beginMeasure(): Promise<any> { return Promise.resolve(true); }
+
+  /**
+   * Ends measuring.
+   */
+  endMeasure(restart: boolean): Promise<{[key: string]: any}> {
+    let resolve: (result: any) => void;
+    let reject: (error: any) => void;
+    const promise = new Promise((res, rej) => {
+      resolve = res;
+      reject = rej;
+    });
+    const adapter = this._wdAdapter;
+    const names = Object.keys(this._userMetrics);
+
+    function getAndClearValues() {
+      Promise.all(names.map(name => adapter.executeScript(`return window.${name}`)))
+          .then((values: any[]) => {
+            if (values.every(v => typeof v === 'number')) {
+              Promise.all(names.map(name => adapter.executeScript(`delete window.${name}`)))
+                  .then((_: any[]) => {
+                    const map: {[k: string]: any} = {};
+                    for (let i = 0, n = names.length; i < n; i++) {
+                      map[names[i]] = values[i];
+                    }
+                    resolve(map);
+                  }, reject);
+            } else {
+              <any>setTimeout(getAndClearValues, 100);
+            }
+          }, reject);
+    }
+    getAndClearValues();
+    return promise;
+  }
+
+  /**
+   * Describes the metrics provided by this metric implementation.
+   * (e.g. units, ...)
+   */
+  describe(): {[key: string]: any} { return this._userMetrics; }
+}

packages/benchpress/src/reporter.ts

@@ -0,0 +1,20 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {MeasureValues} from './measure_values';
+
+/**
+ * A reporter reports measure values and the valid sample.
+ */
+export abstract class Reporter {
+  reportMeasureValues(values: MeasureValues): Promise<any> { throw new Error('NYI'); }
+
+  reportSample(completeSample: MeasureValues[], validSample: MeasureValues[]): Promise<any> {
+    throw new Error('NYI');
+  }
+}

packages/benchpress/src/reporter/console_reporter.ts

@@ -0,0 +1,87 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {Inject, Injectable, InjectionToken} from '@angular/core';
+import {MeasureValues} from '../measure_values';
+import {Reporter} from '../reporter';
+import {SampleDescription} from '../sample_description';
+
+import {formatNum, formatStats, sortedProps} from './util';
+
+
+/**
+ * A reporter for the console
+ */
+@Injectable()
+export class ConsoleReporter extends Reporter {
+  static PRINT = new InjectionToken('ConsoleReporter.print');
+  static COLUMN_WIDTH = new InjectionToken('ConsoleReporter.columnWidth');
+  static PROVIDERS = [
+    ConsoleReporter, {provide: ConsoleReporter.COLUMN_WIDTH, useValue: 18}, {
+      provide: ConsoleReporter.PRINT,
+      useValue: function(v: any) {
+        // tslint:disable-next-line:no-console
+        console.log(v);
+      }
+    }
+  ];
+
+  private static _lpad(value: string, columnWidth: number, fill = ' ') {
+    let result = '';
+    for (let i = 0; i < columnWidth - value.length; i++) {
+      result += fill;
+    }
+    return result + value;
+  }
+
+  private _metricNames: string[];
+
+  constructor(
+      @Inject(ConsoleReporter.COLUMN_WIDTH) private _columnWidth: number,
+      sampleDescription: SampleDescription,
+      @Inject(ConsoleReporter.PRINT) private _print: Function) {
+    super();
+    this._metricNames = sortedProps(sampleDescription.metrics);
+    this._printDescription(sampleDescription);
+  }
+
+  private _printDescription(sampleDescription: SampleDescription) {
+    this._print(`BENCHMARK ${sampleDescription.id}`);
+    this._print('Description:');
+    const props = sortedProps(sampleDescription.description);
+    props.forEach((prop) => { this._print(`- ${prop}: ${sampleDescription.description[prop]}`); });
+    this._print('Metrics:');
+    this._metricNames.forEach((metricName) => {
+      this._print(`- ${metricName}: ${sampleDescription.metrics[metricName]}`);
+    });
+    this._print('');
+    this._printStringRow(this._metricNames);
+    this._printStringRow(this._metricNames.map((_) => ''), '-');
+  }
+
+  reportMeasureValues(measureValues: MeasureValues): Promise<any> {
+    const formattedValues = this._metricNames.map(metricName => {
+      const value = measureValues.values[metricName];
+      return formatNum(value);
+    });
+    this._printStringRow(formattedValues);
+    return Promise.resolve(null);
+  }
+
+  reportSample(completeSample: MeasureValues[], validSamples: MeasureValues[]): Promise<any> {
+    this._printStringRow(this._metricNames.map((_) => ''), '=');
+    this._printStringRow(
+        this._metricNames.map(metricName => formatStats(validSamples, metricName)));
+    return Promise.resolve(null);
+  }
+
+  private _printStringRow(parts: any[], fill = ' ') {
+    this._print(
+        parts.map(part => ConsoleReporter._lpad(part, this._columnWidth, fill)).join(' | '));
+  }
+}

packages/benchpress/src/reporter/json_file_reporter.ts

@@ -0,0 +1,52 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {Inject, Injectable, InjectionToken} from '@angular/core';
+
+import {Options} from '../common_options';
+import {MeasureValues} from '../measure_values';
+import {Reporter} from '../reporter';
+import {SampleDescription} from '../sample_description';
+
+import {formatStats, sortedProps} from './util';
+
+
+/**
+ * A reporter that writes results into a json file.
+ */
+@Injectable()
+export class JsonFileReporter extends Reporter {
+  static PATH = new InjectionToken('JsonFileReporter.path');
+  static PROVIDERS = [JsonFileReporter, {provide: JsonFileReporter.PATH, useValue: '.'}];
+
+  constructor(
+      private _description: SampleDescription, @Inject(JsonFileReporter.PATH) private _path: string,
+      @Inject(Options.WRITE_FILE) private _writeFile: Function,
+      @Inject(Options.NOW) private _now: Function) {
+    super();
+  }
+
+  reportMeasureValues(measureValues: MeasureValues): Promise<any> { return Promise.resolve(null); }
+
+  reportSample(completeSample: MeasureValues[], validSample: MeasureValues[]): Promise<any> {
+    const stats: {[key: string]: string} = {};
+    sortedProps(this._description.metrics).forEach((metricName) => {
+      stats[metricName] = formatStats(validSample, metricName);
+    });
+    const content = JSON.stringify(
+        {
+          'description': this._description,
+          'stats': stats,
+          'completeSample': completeSample,
+          'validSample': validSample,
+        },
+        null, 2);
+    const filePath = `${this._path}/${this._description.id}_${this._now().getTime()}.json`;
+    return this._writeFile(filePath, content);
+  }
+}

packages/benchpress/src/reporter/multi_reporter.ts

@@ -0,0 +1,42 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {InjectionToken, Injector} from '@angular/core';
+
+import {MeasureValues} from '../measure_values';
+import {Reporter} from '../reporter';
+
+export class MultiReporter extends Reporter {
+  static provideWith(childTokens: any[]): any[] {
+    return [
+      {
+        provide: _CHILDREN,
+        useFactory: (injector: Injector) => childTokens.map(token => injector.get(token)),
+        deps: [Injector],
+      },
+      {
+        provide: MultiReporter,
+        useFactory: (children: Reporter[]) => new MultiReporter(children),
+        deps: [_CHILDREN]
+      }
+    ];
+  }
+
+  constructor(private _reporters: Reporter[]) { super(); }
+
+  reportMeasureValues(values: MeasureValues): Promise<any[]> {
+    return Promise.all(this._reporters.map(reporter => reporter.reportMeasureValues(values)));
+  }
+
+  reportSample(completeSample: MeasureValues[], validSample: MeasureValues[]): Promise<any[]> {
+    return Promise.all(
+        this._reporters.map(reporter => reporter.reportSample(completeSample, validSample)));
+  }
+}
+
+const _CHILDREN = new InjectionToken('MultiReporter.children');

packages/benchpress/src/reporter/util.ts

@@ -0,0 +1,28 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {MeasureValues} from '../measure_values';
+import {Statistic} from '../statistic';
+
+export function formatNum(n: number) {
+  return n.toFixed(2);
+}
+
+export function sortedProps(obj: {[key: string]: any}) {
+  return Object.keys(obj).sort();
+}
+
+export function formatStats(validSamples: MeasureValues[], metricName: string): string {
+  const samples = validSamples.map(measureValues => measureValues.values[metricName]);
+  const mean = Statistic.calculateMean(samples);
+  const cv = Statistic.calculateCoefficientOfVariation(samples, mean);
+  const formattedMean = formatNum(mean);
+  // Note: Don't use the unicode character for +- as it might cause
+  // hickups for consoles...
+  return isNaN(cv) ? formattedMean : `${formattedMean}+-${Math.floor(cv)}%`;
+}

packages/benchpress/src/runner.ts

@@ -0,0 +1,109 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {Provider, ReflectiveInjector} from '@angular/core';
+
+import {Options} from './common_options';
+import {Metric} from './metric';
+import {MultiMetric} from './metric/multi_metric';
+import {PerflogMetric} from './metric/perflog_metric';
+import {UserMetric} from './metric/user_metric';
+import {Reporter} from './reporter';
+import {ConsoleReporter} from './reporter/console_reporter';
+import {MultiReporter} from './reporter/multi_reporter';
+import {SampleDescription} from './sample_description';
+import {SampleState, Sampler} from './sampler';
+import {Validator} from './validator';
+import {RegressionSlopeValidator} from './validator/regression_slope_validator';
+import {SizeValidator} from './validator/size_validator';
+import {WebDriverAdapter} from './web_driver_adapter';
+import {WebDriverExtension} from './web_driver_extension';
+import {ChromeDriverExtension} from './webdriver/chrome_driver_extension';
+import {FirefoxDriverExtension} from './webdriver/firefox_driver_extension';
+import {IOsDriverExtension} from './webdriver/ios_driver_extension';
+
+
+
+/**
+ * The Runner is the main entry point for executing a sample run.
+ * It provides defaults, creates the injector and calls the sampler.
+ */
+export class Runner {
+  constructor(private _defaultProviders: Provider[] = []) {}
+
+  sample({id, execute, prepare, microMetrics, providers, userMetrics}: {
+    id: string,
+    execute?: Function,
+    prepare?: Function,
+    microMetrics?: {[key: string]: string},
+    providers?: Provider[],
+    userMetrics?: {[key: string]: string}
+  }): Promise<SampleState> {
+    const sampleProviders: Provider[] = [
+      _DEFAULT_PROVIDERS, this._defaultProviders, {provide: Options.SAMPLE_ID, useValue: id},
+      {provide: Options.EXECUTE, useValue: execute}
+    ];
+    if (prepare != null) {
+      sampleProviders.push({provide: Options.PREPARE, useValue: prepare});
+    }
+    if (microMetrics != null) {
+      sampleProviders.push({provide: Options.MICRO_METRICS, useValue: microMetrics});
+    }
+    if (userMetrics != null) {
+      sampleProviders.push({provide: Options.USER_METRICS, useValue: userMetrics});
+    }
+    if (providers != null) {
+      sampleProviders.push(providers);
+    }
+
+    const inj = ReflectiveInjector.resolveAndCreate(sampleProviders);
+    const adapter: WebDriverAdapter = inj.get(WebDriverAdapter);
+
+    return Promise
+        .all([adapter.capabilities(), adapter.executeScript('return window.navigator.userAgent;')])
+        .then((args) => {
+          const capabilities = args[0];
+          const userAgent = args[1];
+
+          // This might still create instances twice. We are creating a new injector with all the
+          // providers.
+          // Only WebDriverAdapter is reused.
+          // TODO vsavkin consider changing it when toAsyncFactory is added back or when child
+          // injectors are handled better.
+          const injector = ReflectiveInjector.resolveAndCreate([
+            sampleProviders, {provide: Options.CAPABILITIES, useValue: capabilities},
+            {provide: Options.USER_AGENT, useValue: userAgent},
+            {provide: WebDriverAdapter, useValue: adapter}
+          ]);
+
+          const sampler = injector.get(Sampler);
+          return sampler.sample();
+        });
+  }
+}
+
+const _DEFAULT_PROVIDERS = [
+  Options.DEFAULT_PROVIDERS,
+  Sampler.PROVIDERS,
+  ConsoleReporter.PROVIDERS,
+  RegressionSlopeValidator.PROVIDERS,
+  SizeValidator.PROVIDERS,
+  ChromeDriverExtension.PROVIDERS,
+  FirefoxDriverExtension.PROVIDERS,
+  IOsDriverExtension.PROVIDERS,
+  PerflogMetric.PROVIDERS,
+  UserMetric.PROVIDERS,
+  SampleDescription.PROVIDERS,
+  MultiReporter.provideWith([ConsoleReporter]),
+  MultiMetric.provideWith([PerflogMetric, UserMetric]),
+  {provide: Reporter, useExisting: MultiReporter},
+  {provide: Validator, useExisting: RegressionSlopeValidator},
+  WebDriverExtension.provideFirstSupported(
+      [ChromeDriverExtension, FirefoxDriverExtension, IOsDriverExtension]),
+  {provide: Metric, useExisting: MultiMetric},
+];

packages/benchpress/src/sample_description.ts

@@ -0,0 +1,49 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {InjectionToken} from '@angular/core';
+
+import {Options} from './common_options';
+import {Metric} from './metric';
+import {Validator} from './validator';
+
+
+/**
+ * SampleDescription merges all available descriptions about a sample
+ */
+export class SampleDescription {
+  static PROVIDERS = [{
+    provide: SampleDescription,
+    useFactory:
+        (metric: Metric, id: string, forceGc: boolean, userAgent: string, validator: Validator,
+         defaultDesc: {[key: string]: string}, userDesc: {[key: string]: string}) =>
+            new SampleDescription(
+                id,
+                [
+                  {'forceGc': forceGc, 'userAgent': userAgent}, validator.describe(), defaultDesc,
+                  userDesc
+                ],
+                metric.describe()),
+    deps: [
+      Metric, Options.SAMPLE_ID, Options.FORCE_GC, Options.USER_AGENT, Validator,
+      Options.DEFAULT_DESCRIPTION, Options.SAMPLE_DESCRIPTION
+    ]
+  }];
+  description: {[key: string]: any};
+
+  constructor(
+      public id: string, descriptions: Array<{[key: string]: any}>,
+      public metrics: {[key: string]: any}) {
+    this.description = {};
+    descriptions.forEach(description => {
+      Object.keys(description).forEach(prop => { this.description[prop] = description[prop]; });
+    });
+  }
+
+  toJson() { return {'id': this.id, 'description': this.description, 'metrics': this.metrics}; }
+}

packages/benchpress/src/sampler.ts

@@ -0,0 +1,80 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {Inject, Injectable} from '@angular/core';
+
+import {Options} from './common_options';
+import {MeasureValues} from './measure_values';
+import {Metric} from './metric';
+import {Reporter} from './reporter';
+import {Validator} from './validator';
+import {WebDriverAdapter} from './web_driver_adapter';
+
+
+/**
+ * The Sampler owns the sample loop:
+ * 1. calls the prepare/execute callbacks,
+ * 2. gets data from the metric
+ * 3. asks the validator for a valid sample
+ * 4. reports the new data to the reporter
+ * 5. loop until there is a valid sample
+ */
+@Injectable()
+export class Sampler {
+  static PROVIDERS = [Sampler];
+
+  constructor(
+      private _driver: WebDriverAdapter, private _metric: Metric, private _reporter: Reporter,
+      private _validator: Validator, @Inject(Options.PREPARE) private _prepare: Function,
+      @Inject(Options.EXECUTE) private _execute: Function,
+      @Inject(Options.NOW) private _now: Function) {}
+
+  sample(): Promise<SampleState> {
+    const loop = (lastState: SampleState): Promise<SampleState> => {
+      return this._iterate(lastState).then((newState) => {
+        if (newState.validSample != null) {
+          return newState;
+        } else {
+          return loop(newState);
+        }
+      });
+    };
+    return loop(new SampleState([], null));
+  }
+
+  private _iterate(lastState: SampleState): Promise<SampleState> {
+    let resultPromise: Promise<SampleState>;
+    if (this._prepare !== Options.NO_PREPARE) {
+      resultPromise = this._driver.waitFor(this._prepare);
+    } else {
+      resultPromise = Promise.resolve(null);
+    }
+    if (this._prepare !== Options.NO_PREPARE || lastState.completeSample.length === 0) {
+      resultPromise = resultPromise.then((_) => this._metric.beginMeasure());
+    }
+    return resultPromise.then((_) => this._driver.waitFor(this._execute))
+        .then((_) => this._metric.endMeasure(this._prepare === Options.NO_PREPARE))
+        .then((measureValues) => this._report(lastState, measureValues));
+  }
+
+  private _report(state: SampleState, metricValues: {[key: string]: any}): Promise<SampleState> {
+    const measureValues = new MeasureValues(state.completeSample.length, this._now(), metricValues);
+    const completeSample = state.completeSample.concat([measureValues]);
+    const validSample = this._validator.validate(completeSample);
+    let resultPromise = this._reporter.reportMeasureValues(measureValues);
+    if (validSample != null) {
+      resultPromise =
+          resultPromise.then((_) => this._reporter.reportSample(completeSample, validSample));
+    }
+    return resultPromise.then((_) => new SampleState(completeSample, validSample));
+  }
+}
+
+export class SampleState {
+  constructor(public completeSample: MeasureValues[], public validSample: MeasureValues[]) {}
+}

packages/benchpress/src/statistic.ts

@@ -0,0 +1,41 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+export class Statistic {
+  static calculateCoefficientOfVariation(sample: number[], mean: number) {
+    return Statistic.calculateStandardDeviation(sample, mean) / mean * 100;
+  }
+
+  static calculateMean(samples: number[]) {
+    let total = 0;
+    // TODO: use reduce
+    samples.forEach(x => total += x);
+    return total / samples.length;
+  }
+
+  static calculateStandardDeviation(samples: number[], mean: number) {
+    let deviation = 0;
+    // TODO: use reduce
+    samples.forEach(x => deviation += Math.pow(x - mean, 2));
+    deviation = deviation / (samples.length);
+    deviation = Math.sqrt(deviation);
+    return deviation;
+  }
+
+  static calculateRegressionSlope(
+      xValues: number[], xMean: number, yValues: number[], yMean: number) {
+    // See http://en.wikipedia.org/wiki/Simple_linear_regression
+    let dividendSum = 0;
+    let divisorSum = 0;
+    for (let i = 0; i < xValues.length; i++) {
+      dividendSum += (xValues[i] - xMean) * (yValues[i] - yMean);
+      divisorSum += Math.pow(xValues[i] - xMean, 2);
+    }
+    return dividendSum / divisorSum;
+  }
+}

packages/benchpress/src/validator.ts

@@ -0,0 +1,27 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {MeasureValues} from './measure_values';
+
+/**
+ * A Validator calculates a valid sample out of the complete sample.
+ * A valid sample is a sample that represents the population that should be observed
+ * in the correct way.
+ */
+export abstract class Validator {
+  /**
+   * Calculates a valid sample out of the complete sample
+   */
+  validate(completeSample: MeasureValues[]): MeasureValues[] { throw new Error('NYI'); }
+
+  /**
+   * Returns a Map that describes the properties of the validator
+   * (e.g. sample size, ...)
+   */
+  describe(): {[key: string]: any} { throw new Error('NYI'); }
+}

packages/benchpress/src/validator/regression_slope_validator.ts

@@ -0,0 +1,57 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {Inject, Injectable, InjectionToken} from '@angular/core';
+
+import {MeasureValues} from '../measure_values';
+import {Statistic} from '../statistic';
+import {Validator} from '../validator';
+
+/**
+ * A validator that checks the regression slope of a specific metric.
+ * Waits for the regression slope to be >=0.
+ */
+@Injectable()
+export class RegressionSlopeValidator extends Validator {
+  static SAMPLE_SIZE = new InjectionToken('RegressionSlopeValidator.sampleSize');
+  static METRIC = new InjectionToken('RegressionSlopeValidator.metric');
+  static PROVIDERS = [
+    RegressionSlopeValidator, {provide: RegressionSlopeValidator.SAMPLE_SIZE, useValue: 10},
+    {provide: RegressionSlopeValidator.METRIC, useValue: 'scriptTime'}
+  ];
+
+  constructor(
+      @Inject(RegressionSlopeValidator.SAMPLE_SIZE) private _sampleSize: number,
+      @Inject(RegressionSlopeValidator.METRIC) private _metric: string) {
+    super();
+  }
+
+  describe(): {[key: string]: any} {
+    return {'sampleSize': this._sampleSize, 'regressionSlopeMetric': this._metric};
+  }
+
+  validate(completeSample: MeasureValues[]): MeasureValues[] {
+    if (completeSample.length >= this._sampleSize) {
+      const latestSample =
+          completeSample.slice(completeSample.length - this._sampleSize, completeSample.length);
+      const xValues: number[] = [];
+      const yValues: number[] = [];
+      for (let i = 0; i < latestSample.length; i++) {
+        // For now, we only use the array index as x value.
+        // TODO(tbosch): think about whether we should use time here instead
+        xValues.push(i);
+        yValues.push(latestSample[i].values[this._metric]);
+      }
+      const regressionSlope = Statistic.calculateRegressionSlope(
+          xValues, Statistic.calculateMean(xValues), yValues, Statistic.calculateMean(yValues));
+      return regressionSlope >= 0 ? latestSample : null;
+    } else {
+      return null;
+    }
+  }
+}

packages/benchpress/src/validator/size_validator.ts

@@ -0,0 +1,33 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {Inject, Injectable, InjectionToken} from '@angular/core';
+
+import {MeasureValues} from '../measure_values';
+import {Validator} from '../validator';
+
+/**
+ * A validator that waits for the sample to have a certain size.
+ */
+@Injectable()
+export class SizeValidator extends Validator {
+  static SAMPLE_SIZE = new InjectionToken('SizeValidator.sampleSize');
+  static PROVIDERS = [SizeValidator, {provide: SizeValidator.SAMPLE_SIZE, useValue: 10}];
+
+  constructor(@Inject(SizeValidator.SAMPLE_SIZE) private _sampleSize: number) { super(); }
+
+  describe(): {[key: string]: any} { return {'sampleSize': this._sampleSize}; }
+
+  validate(completeSample: MeasureValues[]): MeasureValues[] {
+    if (completeSample.length >= this._sampleSize) {
+      return completeSample.slice(completeSample.length - this._sampleSize, completeSample.length);
+    } else {
+      return null;
+    }
+  }
+}

packages/benchpress/src/web_driver_adapter.ts

@@ -0,0 +1,22 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+
+
+/**
+ * A WebDriverAdapter bridges API differences between different WebDriver clients,
+ * e.g. JS vs Dart Async vs Dart Sync webdriver.
+ * Needs one implementation for every supported WebDriver client.
+ */
+export abstract class WebDriverAdapter {
+  waitFor(callback: Function): Promise<any> { throw new Error('NYI'); }
+  executeScript(script: string): Promise<any> { throw new Error('NYI'); }
+  executeAsyncScript(script: string): Promise<any> { throw new Error('NYI'); }
+  capabilities(): Promise<{[key: string]: any}> { throw new Error('NYI'); }
+  logs(type: string): Promise<any[]> { throw new Error('NYI'); }
+}

packages/benchpress/src/web_driver_extension.ts

@@ -0,0 +1,104 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {InjectionToken, Injector} from '@angular/core';
+
+import {Options} from './common_options';
+
+export type PerfLogEvent = {
+  [key: string]: any
+} & {
+  ph?: 'X' | 'B' | 'E' | 'I',
+  ts?: number,
+  dur?: number,
+  name?: string,
+  pid?: string,
+  args?: {
+    encodedDataLength?: number,
+    usedHeapSize?: number,
+    majorGc?: boolean,
+    url?: string,
+    method?: string
+  }
+};
+
+/**
+ * A WebDriverExtension implements extended commands of the webdriver protocol
+ * for a given browser, independent of the WebDriverAdapter.
+ * Needs one implementation for every supported Browser.
+ */
+export abstract class WebDriverExtension {
+  static provideFirstSupported(childTokens: any[]): any[] {
+    const res = [
+      {
+        provide: _CHILDREN,
+        useFactory: (injector: Injector) => childTokens.map(token => injector.get(token)),
+        deps: [Injector]
+      },
+      {
+        provide: WebDriverExtension,
+        useFactory: (children: WebDriverExtension[], capabilities: {[key: string]: any}) => {
+          let delegate: WebDriverExtension;
+          children.forEach(extension => {
+            if (extension.supports(capabilities)) {
+              delegate = extension;
+            }
+          });
+          if (!delegate) {
+            throw new Error('Could not find a delegate for given capabilities!');
+          }
+          return delegate;
+        },
+        deps: [_CHILDREN, Options.CAPABILITIES]
+      }
+    ];
+    return res;
+  }
+
+  gc(): Promise<any> { throw new Error('NYI'); }
+
+  timeBegin(name: string): Promise<any> { throw new Error('NYI'); }
+
+  timeEnd(name: string, restartName: string): Promise<any> { throw new Error('NYI'); }
+
+  /**
+   * Format:
+   * - cat: category of the event
+   * - name: event name: 'script', 'gc', 'render', ...
+   * - ph: phase: 'B' (begin), 'E' (end), 'X' (Complete event), 'I' (Instant event)
+   * - ts: timestamp in ms, e.g. 12345
+   * - pid: process id
+   * - args: arguments, e.g. {heapSize: 1234}
+   *
+   * Based on [Chrome Trace Event
+   *Format](https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/edit)
+   **/
+  readPerfLog(): Promise<PerfLogEvent[]> { throw new Error('NYI'); }
+
+  perfLogFeatures(): PerfLogFeatures { throw new Error('NYI'); }
+
+  supports(capabilities: {[key: string]: any}): boolean { return true; }
+}
+
+export class PerfLogFeatures {
+  render: boolean;
+  gc: boolean;
+  frameCapture: boolean;
+  userTiming: boolean;
+
+  constructor(
+      {render = false, gc = false, frameCapture = false, userTiming = false}:
+          {render?: boolean, gc?: boolean, frameCapture?: boolean, userTiming?: boolean} = {}) {
+    this.render = render;
+    this.gc = gc;
+    this.frameCapture = frameCapture;
+    this.userTiming = userTiming;
+  }
+}
+
+const _CHILDREN = new InjectionToken('WebDriverExtension.children');

packages/benchpress/src/webdriver/chrome_driver_extension.ts

@@ -0,0 +1,208 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {Inject, Injectable} from '@angular/core';
+
+import {Options} from '../common_options';
+import {WebDriverAdapter} from '../web_driver_adapter';
+import {PerfLogEvent, PerfLogFeatures, WebDriverExtension} from '../web_driver_extension';
+
+/**
+ * Set the following 'traceCategories' to collect metrics in Chrome:
+ * 'v8,blink.console,disabled-by-default-devtools.timeline,devtools.timeline,blink.user_timing'
+ *
+ * In order to collect the frame rate related metrics, add 'benchmark'
+ * to the list above.
+ */
+@Injectable()
+export class ChromeDriverExtension extends WebDriverExtension {
+  static PROVIDERS = [ChromeDriverExtension];
+
+  private _majorChromeVersion: number;
+
+  constructor(private _driver: WebDriverAdapter, @Inject(Options.USER_AGENT) userAgent: string) {
+    super();
+    this._majorChromeVersion = this._parseChromeVersion(userAgent);
+  }
+
+  private _parseChromeVersion(userAgent: string): number {
+    if (!userAgent) {
+      return -1;
+    }
+    let v = userAgent.split(/Chrom(e|ium)\//g)[2];
+    if (!v) {
+      return -1;
+    }
+    v = v.split('.')[0];
+    if (!v) {
+      return -1;
+    }
+    return parseInt(v, 10);
+  }
+
+  gc() { return this._driver.executeScript('window.gc()'); }
+
+  timeBegin(name: string): Promise<any> {
+    return this._driver.executeScript(`console.time('${name}');`);
+  }
+
+  timeEnd(name: string, restartName: string = null): Promise<any> {
+    let script = `console.timeEnd('${name}');`;
+    if (restartName) {
+      script += `console.time('${restartName}');`;
+    }
+    return this._driver.executeScript(script);
+  }
+
+  // See [Chrome Trace Event
+  // Format](https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/edit)
+  readPerfLog(): Promise<PerfLogEvent[]> {
+    // TODO(tbosch): Chromedriver bug https://code.google.com/p/chromedriver/issues/detail?id=1098
+    // Need to execute at least one command so that the browser logs can be read out!
+    return this._driver.executeScript('1+1')
+        .then((_) => this._driver.logs('performance'))
+        .then((entries) => {
+          const events: PerfLogEvent[] = [];
+          entries.forEach((entry: any) => {
+            const message = JSON.parse(entry['message'])['message'];
+            if (message['method'] === 'Tracing.dataCollected') {
+              events.push(message['params']);
+            }
+            if (message['method'] === 'Tracing.bufferUsage') {
+              throw new Error('The DevTools trace buffer filled during the test!');
+            }
+          });
+          return this._convertPerfRecordsToEvents(events);
+        });
+  }
+
+  private _convertPerfRecordsToEvents(
+      chromeEvents: Array<{[key: string]: any}>, normalizedEvents: PerfLogEvent[] = null) {
+    if (!normalizedEvents) {
+      normalizedEvents = [];
+    }
+    chromeEvents.forEach((event) => {
+      const categories = this._parseCategories(event['cat']);
+      const normalizedEvent = this._convertEvent(event, categories);
+      if (normalizedEvent != null) normalizedEvents.push(normalizedEvent);
+    });
+    return normalizedEvents;
+  }
+
+  private _convertEvent(event: {[key: string]: any}, categories: string[]) {
+    const name = event['name'];
+    const args = event['args'];
+    if (this._isEvent(categories, name, ['blink.console'])) {
+      return normalizeEvent(event, {'name': name});
+    } else if (this._isEvent(
+                   categories, name, ['benchmark'],
+                   'BenchmarkInstrumentation::ImplThreadRenderingStats')) {
+      // TODO(goderbauer): Instead of BenchmarkInstrumentation::ImplThreadRenderingStats the
+      // following events should be used (if available) for more accurate measurments:
+      //   1st choice: vsync_before - ground truth on Android
+      //   2nd choice: BenchmarkInstrumentation::DisplayRenderingStats - available on systems with
+      //               new surfaces framework (not broadly enabled yet)
+      //   3rd choice: BenchmarkInstrumentation::ImplThreadRenderingStats - fallback event that is
+      //               always available if something is rendered
+      const frameCount = event['args']['data']['frame_count'];
+      if (frameCount > 1) {
+        throw new Error('multi-frame render stats not supported');
+      }
+      if (frameCount == 1) {
+        return normalizeEvent(event, {'name': 'frame'});
+      }
+    } else if (
+        this._isEvent(categories, name, ['disabled-by-default-devtools.timeline'], 'Rasterize') ||
+        this._isEvent(
+            categories, name, ['disabled-by-default-devtools.timeline'], 'CompositeLayers')) {
+      return normalizeEvent(event, {'name': 'render'});
+    } else if (this._isEvent(categories, name, ['devtools.timeline', 'v8'], 'MajorGC')) {
+      const normArgs = {
+        'majorGc': true,
+        'usedHeapSize': args['usedHeapSizeAfter'] !== undefined ? args['usedHeapSizeAfter'] :
+                                                                  args['usedHeapSizeBefore']
+      };
+      return normalizeEvent(event, {'name': 'gc', 'args': normArgs});
+    } else if (this._isEvent(categories, name, ['devtools.timeline', 'v8'], 'MinorGC')) {
+      const normArgs = {
+        'majorGc': false,
+        'usedHeapSize': args['usedHeapSizeAfter'] !== undefined ? args['usedHeapSizeAfter'] :
+                                                                  args['usedHeapSizeBefore']
+      };
+      return normalizeEvent(event, {'name': 'gc', 'args': normArgs});
+    } else if (
+        this._isEvent(categories, name, ['devtools.timeline'], 'FunctionCall') &&
+        (!args || !args['data'] ||
+         (args['data']['scriptName'] !== 'InjectedScript' && args['data']['scriptName'] !== ''))) {
+      return normalizeEvent(event, {'name': 'script'});
+    } else if (this._isEvent(categories, name, ['devtools.timeline'], 'EvaluateScript')) {
+      return normalizeEvent(event, {'name': 'script'});
+    } else if (this._isEvent(
+                   categories, name, ['devtools.timeline', 'blink'], 'UpdateLayoutTree')) {
+      return normalizeEvent(event, {'name': 'render'});
+    } else if (
+        this._isEvent(categories, name, ['devtools.timeline'], 'UpdateLayerTree') ||
+        this._isEvent(categories, name, ['devtools.timeline'], 'Layout') ||
+        this._isEvent(categories, name, ['devtools.timeline'], 'Paint')) {
+      return normalizeEvent(event, {'name': 'render'});
+    } else if (this._isEvent(categories, name, ['devtools.timeline'], 'ResourceReceivedData')) {
+      const normArgs = {'encodedDataLength': args['data']['encodedDataLength']};
+      return normalizeEvent(event, {'name': 'receivedData', 'args': normArgs});
+    } else if (this._isEvent(categories, name, ['devtools.timeline'], 'ResourceSendRequest')) {
+      const data = args['data'];
+      const normArgs = {'url': data['url'], 'method': data['requestMethod']};
+      return normalizeEvent(event, {'name': 'sendRequest', 'args': normArgs});
+    } else if (this._isEvent(categories, name, ['blink.user_timing'], 'navigationStart')) {
+      return normalizeEvent(event, {'name': 'navigationStart'});
+    }
+    return null;  // nothing useful in this event
+  }
+
+  private _parseCategories(categories: string): string[] { return categories.split(','); }
+
+  private _isEvent(
+      eventCategories: string[], eventName: string, expectedCategories: string[],
+      expectedName: string = null): boolean {
+    const hasCategories = expectedCategories.reduce(
+        (value, cat) => value && eventCategories.indexOf(cat) !== -1, true);
+    return !expectedName ? hasCategories : hasCategories && eventName === expectedName;
+  }
+
+  perfLogFeatures(): PerfLogFeatures {
+    return new PerfLogFeatures({render: true, gc: true, frameCapture: true, userTiming: true});
+  }
+
+  supports(capabilities: {[key: string]: any}): boolean {
+    return this._majorChromeVersion >= 44 && capabilities['browserName'].toLowerCase() === 'chrome';
+  }
+}
+
+function normalizeEvent(chromeEvent: {[key: string]: any}, data: PerfLogEvent): PerfLogEvent {
+  let ph = chromeEvent['ph'].toUpperCase();
+  if (ph === 'S') {
+    ph = 'B';
+  } else if (ph === 'F') {
+    ph = 'E';
+  } else if (ph === 'R') {
+    // mark events from navigation timing
+    ph = 'I';
+  }
+  const result: {[key: string]: any} =
+      {'pid': chromeEvent['pid'], 'ph': ph, 'cat': 'timeline', 'ts': chromeEvent['ts'] / 1000};
+  if (ph === 'X') {
+    let dur = chromeEvent['dur'];
+    if (dur === undefined) {
+      dur = chromeEvent['tdur'];
+    }
+    result['dur'] = !dur ? 0.0 : dur / 1000;
+  }
+  for (const prop in data) {
+    result[prop] = data[prop];
+  }
+  return result;
+}

packages/benchpress/src/webdriver/firefox_driver_extension.ts

@@ -0,0 +1,52 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {Injectable} from '@angular/core';
+
+import {WebDriverAdapter} from '../web_driver_adapter';
+import {PerfLogEvent, PerfLogFeatures, WebDriverExtension} from '../web_driver_extension';
+
+@Injectable()
+export class FirefoxDriverExtension extends WebDriverExtension {
+  static PROVIDERS = [FirefoxDriverExtension];
+
+  private _profilerStarted: boolean;
+
+  constructor(private _driver: WebDriverAdapter) {
+    super();
+    this._profilerStarted = false;
+  }
+
+  gc() { return this._driver.executeScript('window.forceGC()'); }
+
+  timeBegin(name: string): Promise<any> {
+    if (!this._profilerStarted) {
+      this._profilerStarted = true;
+      this._driver.executeScript('window.startProfiler();');
+    }
+    return this._driver.executeScript('window.markStart("' + name + '");');
+  }
+
+  timeEnd(name: string, restartName: string = null): Promise<any> {
+    let script = 'window.markEnd("' + name + '");';
+    if (restartName != null) {
+      script += 'window.markStart("' + restartName + '");';
+    }
+    return this._driver.executeScript(script);
+  }
+
+  readPerfLog(): Promise<PerfLogEvent> {
+    return this._driver.executeAsyncScript('var cb = arguments[0]; window.getProfile(cb);');
+  }
+
+  perfLogFeatures(): PerfLogFeatures { return new PerfLogFeatures({render: true, gc: true}); }
+
+  supports(capabilities: {[key: string]: any}): boolean {
+    return capabilities['browserName'].toLowerCase() === 'firefox';
+  }
+}

packages/benchpress/src/webdriver/ios_driver_extension.ts

@@ -0,0 +1,126 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {Injectable} from '@angular/core';
+
+import {WebDriverAdapter} from '../web_driver_adapter';
+import {PerfLogEvent, PerfLogFeatures, WebDriverExtension} from '../web_driver_extension';
+
+@Injectable()
+export class IOsDriverExtension extends WebDriverExtension {
+  static PROVIDERS = [IOsDriverExtension];
+
+  constructor(private _driver: WebDriverAdapter) { super(); }
+
+  gc(): Promise<any> { throw new Error('Force GC is not supported on iOS'); }
+
+  timeBegin(name: string): Promise<any> {
+    return this._driver.executeScript(`console.time('${name}');`);
+  }
+
+  timeEnd(name: string, restartName: string = null): Promise<any> {
+    let script = `console.timeEnd('${name}');`;
+    if (restartName != null) {
+      script += `console.time('${restartName}');`;
+    }
+    return this._driver.executeScript(script);
+  }
+
+  // See https://github.com/WebKit/webkit/tree/master/Source/WebInspectorUI/Versions
+  readPerfLog() {
+    // TODO(tbosch): Bug in IOsDriver: Need to execute at least one command
+    // so that the browser logs can be read out!
+    return this._driver.executeScript('1+1')
+        .then((_) => this._driver.logs('performance'))
+        .then((entries) => {
+          const records: any[] = [];
+          entries.forEach((entry: any) => {
+            const message = JSON.parse(entry['message'])['message'];
+            if (message['method'] === 'Timeline.eventRecorded') {
+              records.push(message['params']['record']);
+            }
+          });
+          return this._convertPerfRecordsToEvents(records);
+        });
+  }
+
+  /** @internal */
+  private _convertPerfRecordsToEvents(records: any[], events: PerfLogEvent[] = null) {
+    if (!events) {
+      events = [];
+    }
+    records.forEach((record) => {
+      let endEvent: PerfLogEvent = null;
+      const type = record['type'];
+      const data = record['data'];
+      const startTime = record['startTime'];
+      const endTime = record['endTime'];
+
+      if (type === 'FunctionCall' && (data == null || data['scriptName'] !== 'InjectedScript')) {
+        events.push(createStartEvent('script', startTime));
+        endEvent = createEndEvent('script', endTime);
+      } else if (type === 'Time') {
+        events.push(createMarkStartEvent(data['message'], startTime));
+      } else if (type === 'TimeEnd') {
+        events.push(createMarkEndEvent(data['message'], startTime));
+      } else if (
+          type === 'RecalculateStyles' || type === 'Layout' || type === 'UpdateLayerTree' ||
+          type === 'Paint' || type === 'Rasterize' || type === 'CompositeLayers') {
+        events.push(createStartEvent('render', startTime));
+        endEvent = createEndEvent('render', endTime);
+      }
+      // Note: ios used to support GCEvent up until iOS 6 :-(
+      if (record['children'] != null) {
+        this._convertPerfRecordsToEvents(record['children'], events);
+      }
+      if (endEvent != null) {
+        events.push(endEvent);
+      }
+    });
+    return events;
+  }
+
+  perfLogFeatures(): PerfLogFeatures { return new PerfLogFeatures({render: true}); }
+
+  supports(capabilities: {[key: string]: any}): boolean {
+    return capabilities['browserName'].toLowerCase() === 'safari';
+  }
+}
+
+function createEvent(
+    ph: 'X' | 'B' | 'E' | 'B' | 'E', name: string, time: number, args: any = null) {
+  const result: PerfLogEvent = {
+    'cat': 'timeline',
+    'name': name,
+    'ts': time,
+    'ph': ph,
+    // The ios protocol does not support the notions of multiple processes in
+    // the perflog...
+    'pid': 'pid0'
+  };
+  if (args != null) {
+    result['args'] = args;
+  }
+  return result;
+}
+
+function createStartEvent(name: string, time: number, args: any = null) {
+  return createEvent('B', name, time, args);
+}
+
+function createEndEvent(name: string, time: number, args: any = null) {
+  return createEvent('E', name, time, args);
+}
+
+function createMarkStartEvent(name: string, time: number) {
+  return createEvent('B', name, time);
+}
+
+function createMarkEndEvent(name: string, time: number) {
+  return createEvent('E', name, time);
+}

packages/benchpress/src/webdriver/selenium_webdriver_adapter.ts

@@ -0,0 +1,70 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {WebDriverAdapter} from '../web_driver_adapter';
+
+/**
+ * Adapter for the selenium-webdriver.
+ */
+export class SeleniumWebDriverAdapter extends WebDriverAdapter {
+  static PROTRACTOR_PROVIDERS = [{
+    provide: WebDriverAdapter,
+    useFactory: () => new SeleniumWebDriverAdapter((<any>global).browser)
+  }];
+
+  constructor(private _driver: any) { super(); }
+
+  waitFor(callback: () => any): Promise<any> { return this._driver.call(callback); }
+
+  executeScript(script: string): Promise<any> { return this._driver.executeScript(script); }
+
+  executeAsyncScript(script: string): Promise<any> {
+    return this._driver.executeAsyncScript(script);
+  }
+
+  capabilities(): Promise<{[key: string]: any}> {
+    return this._driver.getCapabilities().then((capsObject: any) => {
+      const localData: {[key: string]: any} = {};
+      capsObject.forEach((value: any, key: string) => { localData[key] = value; });
+      return localData;
+    });
+  }
+
+  logs(type: string): Promise<any> {
+    // Needed as selenium-webdriver does not forward
+    // performance logs in the correct way via manage().logs
+    return this._driver.schedule(
+        new Command('getLog').setParameter('type', type),
+        'WebDriver.manage().logs().get(' + type + ')');
+  }
+}
+
+/**
+ * Copy of the `Command` class of webdriver as
+ * it is not exposed via index.js in selenium-webdriver.
+ */
+class Command {
+  private parameters_: {[key: string]: any} = {};
+  constructor(private name_: string) {}
+
+  getName() { return this.name_; }
+
+  setParameter(name: string, value: any) {
+    this.parameters_[name] = value;
+    return this;
+  }
+
+  setParameters(parameters: {[key: string]: any}) {
+    this.parameters_ = parameters;
+    return this;
+  }
+
+  getParameter(key: string) { return this.parameters_[key]; }
+
+  getParameters() { return this.parameters_; }
+}

packages/benchpress/test/firefox_extension/conf.ts

@@ -0,0 +1,21 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+require('core-js');
+require('reflect-metadata');
+const testHelper = require('../../src/firefox_extension/lib/test_helper.js');
+
+exports.config = {
+  specs: ['spec.js', 'sample_benchmark.js'],
+
+  framework: 'jasmine2',
+
+  jasmineNodeOpts: {showColors: true, defaultTimeoutInterval: 1200000},
+
+  getMultiCapabilities: function() { return testHelper.getFirefoxProfileWithExtension(); }
+};

packages/benchpress/test/firefox_extension/parser_util_spec.ts

@@ -0,0 +1,100 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {convertPerfProfileToEvents} from '../../src/firefox_extension/lib/parser_util';
+
+function assertEventsEqual(actualEvents: any[], expectedEvents: any[]) {
+  expect(actualEvents.length == expectedEvents.length);
+  for (let i = 0; i < actualEvents.length; ++i) {
+    const actualEvent = actualEvents[i];
+    const expectedEvent = expectedEvents[i];
+    for (const key in actualEvent) {
+      expect(actualEvent[key]).toEqual(expectedEvent[key]);
+    }
+  }
+};
+
+export function main() {
+  describe('convertPerfProfileToEvents', function() {
+    it('should convert single instantaneous event', function() {
+      const profileData = {
+        threads: [
+          {samples: [{time: 1, frames: [{location: 'FirefoxDriver.prototype.executeScript'}]}]}
+        ]
+      };
+      const perfEvents = convertPerfProfileToEvents(profileData);
+      assertEventsEqual(perfEvents, [{ph: 'X', ts: 1, name: 'script'}]);
+    });
+
+    it('should convert single non-instantaneous event', function() {
+      const profileData = {
+        threads: [{
+          samples: [
+            {time: 1, frames: [{location: 'FirefoxDriver.prototype.executeScript'}]},
+            {time: 2, frames: [{location: 'FirefoxDriver.prototype.executeScript'}]},
+            {time: 100, frames: [{location: 'FirefoxDriver.prototype.executeScript'}]}
+          ]
+        }]
+      };
+      const perfEvents = convertPerfProfileToEvents(profileData);
+      assertEventsEqual(
+          perfEvents, [{ph: 'B', ts: 1, name: 'script'}, {ph: 'E', ts: 100, name: 'script'}]);
+    });
+
+    it('should convert multiple instantaneous events', function() {
+      const profileData = {
+        threads: [{
+          samples: [
+            {time: 1, frames: [{location: 'FirefoxDriver.prototype.executeScript'}]},
+            {time: 2, frames: [{location: 'PresShell::Paint'}]}
+          ]
+        }]
+      };
+      const perfEvents = convertPerfProfileToEvents(profileData);
+      assertEventsEqual(
+          perfEvents, [{ph: 'X', ts: 1, name: 'script'}, {ph: 'X', ts: 2, name: 'render'}]);
+    });
+
+    it('should convert multiple mixed events', function() {
+      const profileData = {
+        threads: [{
+          samples: [
+            {time: 1, frames: [{location: 'FirefoxDriver.prototype.executeScript'}]},
+            {time: 2, frames: [{location: 'PresShell::Paint'}]},
+            {time: 5, frames: [{location: 'FirefoxDriver.prototype.executeScript'}]},
+            {time: 10, frames: [{location: 'FirefoxDriver.prototype.executeScript'}]}
+          ]
+        }]
+      };
+      const perfEvents = convertPerfProfileToEvents(profileData);
+      assertEventsEqual(perfEvents, [
+        {ph: 'X', ts: 1, name: 'script'}, {ph: 'X', ts: 2, name: 'render'},
+        {ph: 'B', ts: 5, name: 'script'}, {ph: 'E', ts: 10, name: 'script'}
+      ]);
+    });
+
+    it('should add args to gc events', function() {
+      const profileData = {threads: [{samples: [{time: 1, frames: [{location: 'forceGC'}]}]}]};
+      const perfEvents = convertPerfProfileToEvents(profileData);
+      assertEventsEqual(perfEvents, [{ph: 'X', ts: 1, name: 'gc', args: {usedHeapSize: 0}}]);
+    });
+
+    it('should skip unknown events', function() {
+      const profileData = {
+        threads: [{
+          samples: [
+            {time: 1, frames: [{location: 'FirefoxDriver.prototype.executeScript'}]},
+            {time: 2, frames: [{location: 'foo'}]}
+          ]
+        }]
+      };
+      const perfEvents = convertPerfProfileToEvents(profileData);
+      assertEventsEqual(perfEvents, [{ph: 'X', ts: 1, name: 'script'}]);
+    });
+  });
+};

packages/benchpress/test/firefox_extension/sample_benchmark.ts

@@ -0,0 +1,42 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {$, browser} from 'protractor';
+
+const benchpress = require('../../index.js');
+const runner = new benchpress.Runner([
+  // use protractor as Webdriver client
+  benchpress.SeleniumWebDriverAdapter.PROTRACTOR_PROVIDERS,
+  // use RegressionSlopeValidator to validate samples
+  benchpress.Validator.bindTo(benchpress.RegressionSlopeValidator),
+  // use 10 samples to calculate slope regression
+  benchpress.bind(benchpress.RegressionSlopeValidator.SAMPLE_SIZE).toValue(20),
+  // use the script metric to calculate slope regression
+  benchpress.bind(benchpress.RegressionSlopeValidator.METRIC).toValue('scriptTime'),
+  benchpress.bind(benchpress.Options.FORCE_GC).toValue(true)
+]);
+
+describe('deep tree baseline', function() {
+  it('should be fast!', function(done) {
+    browser.ignoreSynchronization = true;
+    browser.get('http://localhost:8001/playground/src/benchpress/');
+
+    /*
+     * Tell benchpress to click the buttons to destroy and re-create the tree for each sample.
+     * Benchpress will log the collected metrics after each sample is collected, and will stop
+     * sampling as soon as the calculated regression slope for last 20 samples is stable.
+     */
+    runner
+        .sample({
+          id: 'baseline',
+          execute: function() { $('button').click(); },
+          providers: [benchpress.bind(benchpress.Options.SAMPLE_DESCRIPTION).toValue({depth: 9})]
+        })
+        .then(done, done.fail);
+  });
+});

packages/benchpress/test/firefox_extension/spec.ts

@@ -0,0 +1,44 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+/* tslint:disable:no-console  */
+import {browser} from 'protractor';
+
+const assertEventsContainsName = function(events: any[], eventName: string) {
+  let found = false;
+  for (let i = 0; i < events.length; ++i) {
+    if (events[i].name == eventName) {
+      found = true;
+      break;
+    }
+  }
+  expect(found).toBeTruthy();
+};
+
+describe('firefox extension', function() {
+  const TEST_URL = 'http://localhost:8001/playground/src/hello_world/index.html';
+
+  it('should measure performance', function() {
+    browser.sleep(3000);  // wait for extension to load
+
+    browser.driver.get(TEST_URL);
+
+    browser.executeScript('window.startProfiler()').then(function() {
+      console.log('started measuring perf');
+    });
+
+    browser.executeAsyncScript('setTimeout(arguments[0], 1000);');
+    browser.executeScript('window.forceGC()');
+
+    browser.executeAsyncScript('var cb = arguments[0]; window.getProfile(cb);')
+        .then(function(profile: any) {
+          assertEventsContainsName(profile, 'gc');
+          assertEventsContainsName(profile, 'script');
+        });
+  });
+});

packages/benchpress/test/metric/multi_metric_spec.ts

@@ -0,0 +1,69 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {AsyncTestCompleter, describe, expect, inject, it} from '@angular/core/testing/testing_internal';
+import {Metric, MultiMetric, ReflectiveInjector} from '../../index';
+
+export function main() {
+  function createMetric(ids: any[]) {
+    const m = ReflectiveInjector
+                  .resolveAndCreate([
+                    ids.map(id => ({provide: id, useValue: new MockMetric(id)})),
+                    MultiMetric.provideWith(ids)
+                  ])
+                  .get(MultiMetric);
+    return Promise.resolve(m);
+  }
+
+  describe('multi metric', () => {
+    it('should merge descriptions', inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createMetric(['m1', 'm2']).then((m) => {
+           expect(m.describe()).toEqual({'m1': 'describe', 'm2': 'describe'});
+           async.done();
+         });
+       }));
+
+    it('should merge all beginMeasure calls',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createMetric(['m1', 'm2']).then((m) => m.beginMeasure()).then((values) => {
+           expect(values).toEqual(['m1_beginMeasure', 'm2_beginMeasure']);
+           async.done();
+         });
+       }));
+
+    [false, true].forEach((restartFlag) => {
+      it(`should merge all endMeasure calls for restart=${restartFlag}`,
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           createMetric(['m1', 'm2']).then((m) => m.endMeasure(restartFlag)).then((values) => {
+             expect(values).toEqual(
+                 {'m1': {'restart': restartFlag}, 'm2': {'restart': restartFlag}});
+             async.done();
+           });
+         }));
+    });
+
+  });
+}
+
+class MockMetric extends Metric {
+  constructor(private _id: string) { super(); }
+
+  beginMeasure(): Promise<string> { return Promise.resolve(`${this._id}_beginMeasure`); }
+
+  endMeasure(restart: boolean): Promise<{[key: string]: any}> {
+    const result: {[key: string]: any} = {};
+    result[this._id] = {'restart': restart};
+    return Promise.resolve(result);
+  }
+
+  describe(): {[key: string]: string} {
+    const result: {[key: string]: string} = {};
+    result[this._id] = 'describe';
+    return result;
+  }
+}

packages/benchpress/test/metric/perflog_metric_spec.ts

@@ -0,0 +1,699 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {Provider} from '@angular/core';
+import {AsyncTestCompleter, beforeEach, describe, expect, inject, it} from '@angular/core/testing/testing_internal';
+
+import {Metric, Options, PerfLogEvent, PerfLogFeatures, PerflogMetric, ReflectiveInjector, WebDriverExtension} from '../../index';
+import {TraceEventFactory} from '../trace_event_factory';
+
+export function main() {
+  let commandLog: any[];
+  const eventFactory = new TraceEventFactory('timeline', 'pid0');
+
+  function createMetric(
+      perfLogs: PerfLogEvent[], perfLogFeatures: PerfLogFeatures,
+      {microMetrics, forceGc, captureFrames, receivedData, requestCount}: {
+        microMetrics?: {[key: string]: string},
+        forceGc?: boolean,
+        captureFrames?: boolean,
+        receivedData?: boolean,
+        requestCount?: boolean
+      } = {}): Metric {
+    commandLog = [];
+    if (!perfLogFeatures) {
+      perfLogFeatures =
+          new PerfLogFeatures({render: true, gc: true, frameCapture: true, userTiming: true});
+    }
+    if (!microMetrics) {
+      microMetrics = {};
+    }
+    const providers: Provider[] = [
+      Options.DEFAULT_PROVIDERS, PerflogMetric.PROVIDERS,
+      {provide: Options.MICRO_METRICS, useValue: microMetrics}, {
+        provide: PerflogMetric.SET_TIMEOUT,
+        useValue: (fn: Function, millis: number) => {
+          commandLog.push(['setTimeout', millis]);
+          fn();
+        },
+      },
+      {
+        provide: WebDriverExtension,
+        useValue: new MockDriverExtension(perfLogs, commandLog, perfLogFeatures)
+      }
+    ];
+    if (forceGc != null) {
+      providers.push({provide: Options.FORCE_GC, useValue: forceGc});
+    }
+    if (captureFrames != null) {
+      providers.push({provide: Options.CAPTURE_FRAMES, useValue: captureFrames});
+    }
+    if (receivedData != null) {
+      providers.push({provide: Options.RECEIVED_DATA, useValue: receivedData});
+    }
+    if (requestCount != null) {
+      providers.push({provide: Options.REQUEST_COUNT, useValue: requestCount});
+    }
+    return ReflectiveInjector.resolveAndCreate(providers).get(PerflogMetric);
+  }
+
+  describe('perflog metric', () => {
+
+    function sortedKeys(stringMap: {[key: string]: any}) {
+      const res: string[] = [];
+      res.push(...Object.keys(stringMap));
+      res.sort();
+      return res;
+    }
+
+    it('should describe itself based on the perfLogFeatrues', () => {
+      expect(sortedKeys(createMetric([[]], new PerfLogFeatures()).describe())).toEqual([
+        'pureScriptTime', 'scriptTime'
+      ]);
+
+      expect(
+          sortedKeys(createMetric([[]], new PerfLogFeatures({render: true, gc: false})).describe()))
+          .toEqual(['pureScriptTime', 'renderTime', 'scriptTime']);
+
+      expect(sortedKeys(createMetric([[]], null).describe())).toEqual([
+        'gcAmount', 'gcTime', 'majorGcTime', 'pureScriptTime', 'renderTime', 'scriptTime'
+      ]);
+
+      expect(sortedKeys(createMetric([[]], new PerfLogFeatures({render: true, gc: true}), {
+                          forceGc: true
+                        }).describe()))
+          .toEqual([
+            'forcedGcAmount', 'forcedGcTime', 'gcAmount', 'gcTime', 'majorGcTime', 'pureScriptTime',
+            'renderTime', 'scriptTime'
+          ]);
+
+
+      expect(sortedKeys(createMetric([[]], new PerfLogFeatures({userTiming: true}), {
+                          receivedData: true,
+                          requestCount: true
+                        }).describe()))
+          .toEqual(['pureScriptTime', 'receivedData', 'requestCount', 'scriptTime']);
+    });
+
+    it('should describe itself based on micro metrics', () => {
+      const description =
+          createMetric([[]], null, {microMetrics: {'myMicroMetric': 'someDesc'}}).describe();
+      expect(description['myMicroMetric']).toEqual('someDesc');
+    });
+
+    it('should describe itself if frame capture is requested and available', () => {
+      const description = createMetric([[]], new PerfLogFeatures({frameCapture: true}), {
+                            captureFrames: true
+                          }).describe();
+      expect(description['frameTime.mean']).not.toContain('WARNING');
+      expect(description['frameTime.best']).not.toContain('WARNING');
+      expect(description['frameTime.worst']).not.toContain('WARNING');
+      expect(description['frameTime.smooth']).not.toContain('WARNING');
+    });
+
+    it('should describe itself if frame capture is requested and not available', () => {
+      const description = createMetric([[]], new PerfLogFeatures({frameCapture: false}), {
+                            captureFrames: true
+                          }).describe();
+      expect(description['frameTime.mean']).toContain('WARNING');
+      expect(description['frameTime.best']).toContain('WARNING');
+      expect(description['frameTime.worst']).toContain('WARNING');
+      expect(description['frameTime.smooth']).toContain('WARNING');
+    });
+
+    describe('beginMeasure', () => {
+
+      it('should not force gc and mark the timeline',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           const metric = createMetric([[]], null);
+           metric.beginMeasure().then((_) => {
+             expect(commandLog).toEqual([['timeBegin', 'benchpress0']]);
+
+             async.done();
+           });
+         }));
+
+      it('should force gc and mark the timeline',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           const metric = createMetric([[]], null, {forceGc: true});
+           metric.beginMeasure().then((_) => {
+             expect(commandLog).toEqual([['gc'], ['timeBegin', 'benchpress0']]);
+
+             async.done();
+           });
+         }));
+
+    });
+
+    describe('endMeasure', () => {
+
+      it('should mark and aggregate events in between the marks',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           const events = [[
+             eventFactory.markStart('benchpress0', 0), eventFactory.start('script', 4),
+             eventFactory.end('script', 6), eventFactory.markEnd('benchpress0', 10)
+           ]];
+           const metric = createMetric(events, null);
+           metric.beginMeasure().then((_) => metric.endMeasure(false)).then((data) => {
+             expect(commandLog).toEqual([
+               ['timeBegin', 'benchpress0'], ['timeEnd', 'benchpress0', null], 'readPerfLog'
+             ]);
+             expect(data['scriptTime']).toBe(2);
+
+             async.done();
+           });
+         }));
+
+      it('should mark and aggregate events since navigationStart',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           const events = [[
+             eventFactory.markStart('benchpress0', 0), eventFactory.start('script', 4),
+             eventFactory.end('script', 6), eventFactory.instant('navigationStart', 7),
+             eventFactory.start('script', 8), eventFactory.end('script', 9),
+             eventFactory.markEnd('benchpress0', 10)
+           ]];
+           const metric = createMetric(events, null);
+           metric.beginMeasure().then((_) => metric.endMeasure(false)).then((data) => {
+             expect(data['scriptTime']).toBe(1);
+
+             async.done();
+           });
+         }));
+
+      it('should restart timing', inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           const events = [
+             [
+               eventFactory.markStart('benchpress0', 0),
+               eventFactory.markEnd('benchpress0', 1),
+               eventFactory.markStart('benchpress1', 2),
+             ],
+             [eventFactory.markEnd('benchpress1', 3)]
+           ];
+           const metric = createMetric(events, null);
+           metric.beginMeasure()
+               .then((_) => metric.endMeasure(true))
+               .then((_) => metric.endMeasure(true))
+               .then((_) => {
+                 expect(commandLog).toEqual([
+                   ['timeBegin', 'benchpress0'], ['timeEnd', 'benchpress0', 'benchpress1'],
+                   'readPerfLog', ['timeEnd', 'benchpress1', 'benchpress2'], 'readPerfLog'
+                 ]);
+
+                 async.done();
+               });
+         }));
+
+      it('should loop and aggregate until the end mark is present',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           const events = [
+             [eventFactory.markStart('benchpress0', 0), eventFactory.start('script', 1)],
+             [eventFactory.end('script', 2)],
+             [
+               eventFactory.start('script', 3), eventFactory.end('script', 5),
+               eventFactory.markEnd('benchpress0', 10)
+             ]
+           ];
+           const metric = createMetric(events, null);
+           metric.beginMeasure().then((_) => metric.endMeasure(false)).then((data) => {
+             expect(commandLog).toEqual([
+               ['timeBegin', 'benchpress0'], ['timeEnd', 'benchpress0', null], 'readPerfLog',
+               ['setTimeout', 100], 'readPerfLog', ['setTimeout', 100], 'readPerfLog'
+             ]);
+             expect(data['scriptTime']).toBe(3);
+
+             async.done();
+           });
+         }));
+
+      it('should store events after the end mark for the next call',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           const events = [
+             [
+               eventFactory.markStart('benchpress0', 0), eventFactory.markEnd('benchpress0', 1),
+               eventFactory.markStart('benchpress1', 1), eventFactory.start('script', 1),
+               eventFactory.end('script', 2)
+             ],
+             [
+               eventFactory.start('script', 3), eventFactory.end('script', 5),
+               eventFactory.markEnd('benchpress1', 6)
+             ]
+           ];
+           const metric = createMetric(events, null);
+           metric.beginMeasure()
+               .then((_) => metric.endMeasure(true))
+               .then((data) => {
+                 expect(data['scriptTime']).toBe(0);
+                 return metric.endMeasure(true);
+               })
+               .then((data) => {
+                 expect(commandLog).toEqual([
+                   ['timeBegin', 'benchpress0'], ['timeEnd', 'benchpress0', 'benchpress1'],
+                   'readPerfLog', ['timeEnd', 'benchpress1', 'benchpress2'], 'readPerfLog'
+                 ]);
+                 expect(data['scriptTime']).toBe(3);
+
+                 async.done();
+               });
+         }));
+
+      describe('with forced gc', () => {
+        let events: PerfLogEvent[][];
+        beforeEach(() => {
+          events = [[
+            eventFactory.markStart('benchpress0', 0), eventFactory.start('script', 4),
+            eventFactory.end('script', 6), eventFactory.markEnd('benchpress0', 10),
+            eventFactory.markStart('benchpress1', 11),
+            eventFactory.start('gc', 12, {'usedHeapSize': 2500}),
+            eventFactory.end('gc', 15, {'usedHeapSize': 1000}),
+            eventFactory.markEnd('benchpress1', 20)
+          ]];
+        });
+
+        it('should measure forced gc', inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+             const metric = createMetric(events, null, {forceGc: true});
+             metric.beginMeasure().then((_) => metric.endMeasure(false)).then((data) => {
+               expect(commandLog).toEqual([
+                 ['gc'], ['timeBegin', 'benchpress0'], ['timeEnd', 'benchpress0', 'benchpress1'],
+                 'readPerfLog', ['gc'], ['timeEnd', 'benchpress1', null], 'readPerfLog'
+               ]);
+               expect(data['forcedGcTime']).toBe(3);
+               expect(data['forcedGcAmount']).toBe(1.5);
+
+               async.done();
+             });
+           }));
+
+        it('should restart after the forced gc if needed',
+           inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+             const metric = createMetric(events, null, {forceGc: true});
+             metric.beginMeasure().then((_) => metric.endMeasure(true)).then((data) => {
+               expect(commandLog[5]).toEqual(['timeEnd', 'benchpress1', 'benchpress2']);
+
+               async.done();
+             });
+           }));
+
+      });
+
+    });
+
+    describe('aggregation', () => {
+
+      function aggregate(events: any[], {microMetrics, captureFrames, receivedData, requestCount}: {
+        microMetrics?: {[key: string]: string},
+        captureFrames?: boolean,
+        receivedData?: boolean,
+        requestCount?: boolean
+      } = {}) {
+        events.unshift(eventFactory.markStart('benchpress0', 0));
+        events.push(eventFactory.markEnd('benchpress0', 10));
+        const metric = createMetric([events], null, {
+          microMetrics: microMetrics,
+          captureFrames: captureFrames,
+          receivedData: receivedData,
+          requestCount: requestCount
+        });
+        return metric.beginMeasure().then((_) => metric.endMeasure(false));
+      }
+
+      describe('frame metrics', () => {
+        it('should calculate mean frame time',
+           inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+             aggregate(
+                 [
+                   eventFactory.markStart('frameCapture', 0), eventFactory.instant('frame', 1),
+                   eventFactory.instant('frame', 3), eventFactory.instant('frame', 4),
+                   eventFactory.markEnd('frameCapture', 5)
+                 ],
+                 {captureFrames: true})
+                 .then((data) => {
+                   expect(data['frameTime.mean']).toBe(((3 - 1) + (4 - 3)) / 2);
+                   async.done();
+                 });
+           }));
+
+        it('should throw if no start event',
+           inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+
+             aggregate(
+                 [eventFactory.instant('frame', 4), eventFactory.markEnd('frameCapture', 5)],
+                 {captureFrames: true})
+                 .catch((err): any => {
+                   expect(() => {
+                     throw err;
+                   }).toThrowError('missing start event for frame capture');
+                   async.done();
+                 });
+           }));
+
+        it('should throw if no end event',
+           inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+
+             aggregate(
+                 [eventFactory.markStart('frameCapture', 3), eventFactory.instant('frame', 4)],
+                 {captureFrames: true})
+                 .catch((err): any => {
+                   expect(() => { throw err; }).toThrowError('missing end event for frame capture');
+                   async.done();
+                 });
+           }));
+
+        it('should throw if trying to capture twice',
+           inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+
+             aggregate(
+                 [
+                   eventFactory.markStart('frameCapture', 3),
+                   eventFactory.markStart('frameCapture', 4)
+                 ],
+                 {captureFrames: true})
+                 .catch((err): any => {
+                   expect(() => {
+                     throw err;
+                   }).toThrowError('can capture frames only once per benchmark run');
+                   async.done();
+                 });
+           }));
+
+        it('should throw if trying to capture when frame capture is disabled',
+           inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+             aggregate([eventFactory.markStart('frameCapture', 3)]).catch((err) => {
+               expect(() => { throw err; })
+                   .toThrowError(
+                       'found start event for frame capture, but frame capture was not requested in benchpress');
+               async.done();
+               return null;
+             });
+           }));
+
+        it('should throw if frame capture is enabled, but nothing is captured',
+           inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+             aggregate([], {captureFrames: true}).catch((err): any => {
+               expect(() => { throw err; })
+                   .toThrowError(
+                       'frame capture requested in benchpress, but no start event was found');
+               async.done();
+             });
+           }));
+
+        it('should calculate best and worst frame time',
+           inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+             aggregate(
+                 [
+                   eventFactory.markStart('frameCapture', 0), eventFactory.instant('frame', 1),
+                   eventFactory.instant('frame', 9), eventFactory.instant('frame', 15),
+                   eventFactory.instant('frame', 18), eventFactory.instant('frame', 28),
+                   eventFactory.instant('frame', 32), eventFactory.markEnd('frameCapture', 10)
+                 ],
+                 {captureFrames: true})
+                 .then((data) => {
+                   expect(data['frameTime.worst']).toBe(10);
+                   expect(data['frameTime.best']).toBe(3);
+                   async.done();
+                 });
+           }));
+
+        it('should calculate percentage of smoothness to be good',
+           inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+             aggregate(
+                 [
+                   eventFactory.markStart('frameCapture', 0), eventFactory.instant('frame', 1),
+                   eventFactory.instant('frame', 2), eventFactory.instant('frame', 3),
+                   eventFactory.markEnd('frameCapture', 4)
+                 ],
+                 {captureFrames: true})
+                 .then((data) => {
+                   expect(data['frameTime.smooth']).toBe(1.0);
+                   async.done();
+                 });
+           }));
+
+        it('should calculate percentage of smoothness to be bad',
+           inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+             aggregate(
+                 [
+                   eventFactory.markStart('frameCapture', 0), eventFactory.instant('frame', 1),
+                   eventFactory.instant('frame', 2), eventFactory.instant('frame', 22),
+                   eventFactory.instant('frame', 23), eventFactory.instant('frame', 24),
+                   eventFactory.markEnd('frameCapture', 4)
+                 ],
+                 {captureFrames: true})
+                 .then((data) => {
+                   expect(data['frameTime.smooth']).toBe(0.75);
+                   async.done();
+                 });
+           }));
+
+      });
+
+      it('should report a single interval',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           aggregate([
+             eventFactory.start('script', 0), eventFactory.end('script', 5)
+           ]).then((data) => {
+             expect(data['scriptTime']).toBe(5);
+             async.done();
+           });
+         }));
+
+      it('should sum up multiple intervals',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           aggregate([
+             eventFactory.start('script', 0), eventFactory.end('script', 5),
+             eventFactory.start('script', 10), eventFactory.end('script', 17)
+           ]).then((data) => {
+             expect(data['scriptTime']).toBe(12);
+             async.done();
+           });
+         }));
+
+      it('should ignore not started intervals',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           aggregate([eventFactory.end('script', 10)]).then((data) => {
+             expect(data['scriptTime']).toBe(0);
+             async.done();
+           });
+         }));
+
+      it('should ignore not ended intervals',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           aggregate([eventFactory.start('script', 10)]).then((data) => {
+             expect(data['scriptTime']).toBe(0);
+             async.done();
+           });
+         }));
+
+      it('should ignore nested intervals',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           aggregate([
+             eventFactory.start('script', 0), eventFactory.start('script', 5),
+             eventFactory.end('script', 10), eventFactory.end('script', 17)
+           ]).then((data) => {
+             expect(data['scriptTime']).toBe(17);
+             async.done();
+           });
+         }));
+
+      it('should ignore events from different processed as the start mark',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           const otherProcessEventFactory = new TraceEventFactory('timeline', 'pid1');
+           const metric = createMetric(
+               [[
+                 eventFactory.markStart('benchpress0', 0), eventFactory.start('script', 0, null),
+                 eventFactory.end('script', 5, null),
+                 otherProcessEventFactory.start('script', 10, null),
+                 otherProcessEventFactory.end('script', 17, null),
+                 eventFactory.markEnd('benchpress0', 20)
+               ]],
+               null);
+           metric.beginMeasure().then((_) => metric.endMeasure(false)).then((data) => {
+             expect(data['scriptTime']).toBe(5);
+             async.done();
+           });
+         }));
+
+      it('should support scriptTime metric',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           aggregate([
+             eventFactory.start('script', 0), eventFactory.end('script', 5)
+           ]).then((data) => {
+             expect(data['scriptTime']).toBe(5);
+             async.done();
+           });
+         }));
+
+      it('should support renderTime metric',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           aggregate([
+             eventFactory.start('render', 0), eventFactory.end('render', 5)
+           ]).then((data) => {
+             expect(data['renderTime']).toBe(5);
+             async.done();
+           });
+         }));
+
+      it('should support gcTime/gcAmount metric',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           aggregate([
+             eventFactory.start('gc', 0, {'usedHeapSize': 2500}),
+             eventFactory.end('gc', 5, {'usedHeapSize': 1000})
+           ]).then((data) => {
+             expect(data['gcTime']).toBe(5);
+             expect(data['gcAmount']).toBe(1.5);
+             expect(data['majorGcTime']).toBe(0);
+             async.done();
+           });
+         }));
+
+      it('should support majorGcTime metric',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           aggregate([
+             eventFactory.start('gc', 0, {'usedHeapSize': 2500}),
+             eventFactory.end('gc', 5, {'usedHeapSize': 1000, 'majorGc': true})
+           ]).then((data) => {
+             expect(data['gcTime']).toBe(5);
+             expect(data['majorGcTime']).toBe(5);
+             async.done();
+           });
+         }));
+
+      it('should support pureScriptTime = scriptTime-gcTime-renderTime',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           aggregate([
+             eventFactory.start('script', 0), eventFactory.start('gc', 1, {'usedHeapSize': 1000}),
+             eventFactory.end('gc', 4, {'usedHeapSize': 0}), eventFactory.start('render', 4),
+             eventFactory.end('render', 5), eventFactory.end('script', 6)
+           ]).then((data) => {
+             expect(data['scriptTime']).toBe(6);
+             expect(data['pureScriptTime']).toBe(2);
+             async.done();
+           });
+         }));
+
+      describe('receivedData', () => {
+        it('should report received data since last navigationStart',
+           inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+             aggregate(
+                 [
+                   eventFactory.instant('receivedData', 0, {'encodedDataLength': 1}),
+                   eventFactory.instant('navigationStart', 1),
+                   eventFactory.instant('receivedData', 2, {'encodedDataLength': 2}),
+                   eventFactory.instant('navigationStart', 3),
+                   eventFactory.instant('receivedData', 4, {'encodedDataLength': 4}),
+                   eventFactory.instant('receivedData', 5, {'encodedDataLength': 8})
+                 ],
+                 {receivedData: true})
+                 .then((data) => {
+                   expect(data['receivedData']).toBe(12);
+                   async.done();
+                 });
+           }));
+      });
+
+      describe('requestCount', () => {
+        it('should report count of requests sent since last navigationStart',
+           inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+             aggregate(
+                 [
+                   eventFactory.instant('sendRequest', 0),
+                   eventFactory.instant('navigationStart', 1),
+                   eventFactory.instant('sendRequest', 2),
+                   eventFactory.instant('navigationStart', 3),
+                   eventFactory.instant('sendRequest', 4), eventFactory.instant('sendRequest', 5)
+                 ],
+                 {requestCount: true})
+                 .then((data) => {
+                   expect(data['requestCount']).toBe(2);
+                   async.done();
+                 });
+           }));
+      });
+
+      describe('microMetrics', () => {
+
+        it('should report micro metrics',
+           inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+             aggregate(
+                 [
+                   eventFactory.markStart('mm1', 0),
+                   eventFactory.markEnd('mm1', 5),
+                 ],
+                 {microMetrics: {'mm1': 'micro metric 1'}})
+                 .then((data) => {
+                   expect(data['mm1']).toBe(5.0);
+                   async.done();
+                 });
+           }));
+
+        it('should ignore micro metrics that were not specified',
+           inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+             aggregate([
+               eventFactory.markStart('mm1', 0),
+               eventFactory.markEnd('mm1', 5),
+             ]).then((data) => {
+               expect(data['mm1']).toBeFalsy();
+               async.done();
+             });
+           }));
+
+        it('should report micro metric averages',
+           inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+             aggregate(
+                 [
+                   eventFactory.markStart('mm1*20', 0),
+                   eventFactory.markEnd('mm1*20', 5),
+                 ],
+                 {microMetrics: {'mm1': 'micro metric 1'}})
+                 .then((data) => {
+                   expect(data['mm1']).toBe(5 / 20);
+                   async.done();
+                 });
+           }));
+
+      });
+
+    });
+
+  });
+}
+
+class MockDriverExtension extends WebDriverExtension {
+  constructor(
+      private _perfLogs: any[], private _commandLog: any[],
+      private _perfLogFeatures: PerfLogFeatures) {
+    super();
+  }
+
+  timeBegin(name: string): Promise<any> {
+    this._commandLog.push(['timeBegin', name]);
+    return Promise.resolve(null);
+  }
+
+  timeEnd(name: string, restartName: string): Promise<any> {
+    this._commandLog.push(['timeEnd', name, restartName]);
+    return Promise.resolve(null);
+  }
+
+  perfLogFeatures(): PerfLogFeatures { return this._perfLogFeatures; }
+
+  readPerfLog(): Promise<any> {
+    this._commandLog.push('readPerfLog');
+    if (this._perfLogs.length > 0) {
+      const next = this._perfLogs[0];
+      this._perfLogs.shift();
+      return Promise.resolve(next);
+    } else {
+      return Promise.resolve([]);
+    }
+  }
+
+  gc(): Promise<any> {
+    this._commandLog.push(['gc']);
+    return Promise.resolve(null);
+  }
+}

packages/benchpress/test/metric/user_metric_spec.ts

@@ -0,0 +1,82 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {Provider, ReflectiveInjector} from '@angular/core';
+import {AsyncTestCompleter, describe, expect, inject, it} from '@angular/core/testing/testing_internal';
+
+import {Options, PerfLogEvent, PerfLogFeatures, UserMetric, WebDriverAdapter} from '../../index';
+
+export function main() {
+  let wdAdapter: MockDriverAdapter;
+
+  function createMetric(
+      perfLogs: PerfLogEvent[], perfLogFeatures: PerfLogFeatures,
+      {userMetrics}: {userMetrics?: {[key: string]: string}} = {}): UserMetric {
+    if (!perfLogFeatures) {
+      perfLogFeatures =
+          new PerfLogFeatures({render: true, gc: true, frameCapture: true, userTiming: true});
+    }
+    if (!userMetrics) {
+      userMetrics = {};
+    }
+    wdAdapter = new MockDriverAdapter();
+    const providers: Provider[] = [
+      Options.DEFAULT_PROVIDERS, UserMetric.PROVIDERS,
+      {provide: Options.USER_METRICS, useValue: userMetrics},
+      {provide: WebDriverAdapter, useValue: wdAdapter}
+    ];
+    return ReflectiveInjector.resolveAndCreate(providers).get(UserMetric);
+  }
+
+  describe('user metric', () => {
+
+    it('should describe itself based on userMetrics', () => {
+      expect(createMetric([[]], new PerfLogFeatures(), {
+               userMetrics: {'loadTime': 'time to load'}
+             }).describe())
+          .toEqual({'loadTime': 'time to load'});
+    });
+
+    describe('endMeasure', () => {
+      it('should stop measuring when all properties have numeric values',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           const metric = createMetric(
+               [[]], new PerfLogFeatures(),
+               {userMetrics: {'loadTime': 'time to load', 'content': 'time to see content'}});
+           metric.beginMeasure()
+               .then((_) => metric.endMeasure(true))
+               .then((values: {[key: string]: string}) => {
+                 expect(values['loadTime']).toBe(25);
+                 expect(values['content']).toBe(250);
+                 async.done();
+               });
+
+           wdAdapter.data['loadTime'] = 25;
+           // Wait before setting 2nd property.
+           setTimeout(() => { wdAdapter.data['content'] = 250; }, 50);
+
+         }), 600);
+    });
+  });
+}
+
+class MockDriverAdapter extends WebDriverAdapter {
+  data: any = {};
+
+  executeScript(script: string): any {
+    // Just handles `return window.propName` ignores `delete window.propName`.
+    if (script.indexOf('return window.') == 0) {
+      const metricName = script.substring('return window.'.length);
+      return Promise.resolve(this.data[metricName]);
+    } else if (script.indexOf('delete window.') == 0) {
+      return Promise.resolve(null);
+    } else {
+      return Promise.reject(`Unexpected syntax: ${script}`);
+    }
+  }
+}

packages/benchpress/test/reporter/console_reporter_spec.ts

@@ -0,0 +1,93 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {Provider} from '@angular/core';
+import {describe, expect, it} from '@angular/core/testing/testing_internal';
+
+import {ConsoleReporter, MeasureValues, ReflectiveInjector, SampleDescription} from '../../index';
+
+export function main() {
+  describe('console reporter', () => {
+    let reporter: ConsoleReporter;
+    let log: string[];
+
+    function createReporter(
+        {columnWidth = null, sampleId = null, descriptions = null, metrics = null}: {
+          columnWidth?: number,
+          sampleId?: string,
+          descriptions?: {[key: string]: any}[],
+          metrics?: {[key: string]: any}
+        }) {
+      log = [];
+      if (!descriptions) {
+        descriptions = [];
+      }
+      if (sampleId == null) {
+        sampleId = 'null';
+      }
+      const providers: Provider[] = [
+        ConsoleReporter.PROVIDERS, {
+          provide: SampleDescription,
+          useValue: new SampleDescription(sampleId, descriptions, metrics)
+        },
+        {provide: ConsoleReporter.PRINT, useValue: (line: string) => log.push(line)}
+      ];
+      if (columnWidth != null) {
+        providers.push({provide: ConsoleReporter.COLUMN_WIDTH, useValue: columnWidth});
+      }
+      reporter = ReflectiveInjector.resolveAndCreate(providers).get(ConsoleReporter);
+    }
+
+    it('should print the sample id, description and table header', () => {
+      createReporter({
+        columnWidth: 8,
+        sampleId: 'someSample',
+        descriptions: [{'a': 1, 'b': 2}],
+        metrics: {'m1': 'some desc', 'm2': 'some other desc'}
+      });
+      expect(log).toEqual([
+        'BENCHMARK someSample',
+        'Description:',
+        '- a: 1',
+        '- b: 2',
+        'Metrics:',
+        '- m1: some desc',
+        '- m2: some other desc',
+        '',
+        '      m1 |       m2',
+        '-------- | --------',
+      ]);
+    });
+
+    it('should print a table row', () => {
+      createReporter({columnWidth: 8, metrics: {'a': '', 'b': ''}});
+      log = [];
+      reporter.reportMeasureValues(mv(0, 0, {'a': 1.23, 'b': 2}));
+      expect(log).toEqual(['    1.23 |     2.00']);
+    });
+
+    it('should print the table footer and stats when there is a valid sample', () => {
+      createReporter({columnWidth: 8, metrics: {'a': '', 'b': ''}});
+      log = [];
+      reporter.reportSample([], [mv(0, 0, {'a': 3, 'b': 6}), mv(1, 1, {'a': 5, 'b': 9})]);
+      expect(log).toEqual(['======== | ========', '4.00+-25% | 7.50+-20%']);
+    });
+
+    it('should print the coefficient of variation only when it is meaningful', () => {
+      createReporter({columnWidth: 8, metrics: {'a': '', 'b': ''}});
+      log = [];
+      reporter.reportSample([], [mv(0, 0, {'a': 3, 'b': 0}), mv(1, 1, {'a': 5, 'b': 0})]);
+      expect(log).toEqual(['======== | ========', '4.00+-25% |     0.00']);
+    });
+
+  });
+}
+
+function mv(runIndex: number, time: number, values: {[key: string]: number}) {
+  return new MeasureValues(runIndex, new Date(time), values);
+}

packages/benchpress/test/reporter/json_file_reporter_spec.ts

@@ -0,0 +1,80 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {AsyncTestCompleter, describe, expect, inject, it} from '@angular/core/testing/testing_internal';
+
+import {JsonFileReporter, MeasureValues, Options, ReflectiveInjector, SampleDescription} from '../../index';
+
+export function main() {
+  describe('file reporter', () => {
+    let loggedFile: any;
+
+    function createReporter({sampleId, descriptions, metrics, path}: {
+      sampleId: string,
+      descriptions: {[key: string]: any}[],
+      metrics: {[key: string]: string},
+      path: string
+    }) {
+      const providers = [
+        JsonFileReporter.PROVIDERS, {
+          provide: SampleDescription,
+          useValue: new SampleDescription(sampleId, descriptions, metrics)
+        },
+        {provide: JsonFileReporter.PATH, useValue: path},
+        {provide: Options.NOW, useValue: () => new Date(1234)}, {
+          provide: Options.WRITE_FILE,
+          useValue: (filename: string, content: string) => {
+            loggedFile = {'filename': filename, 'content': content};
+            return Promise.resolve(null);
+          }
+        }
+      ];
+      return ReflectiveInjector.resolveAndCreate(providers).get(JsonFileReporter);
+    }
+
+    it('should write all data into a file',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createReporter({
+           sampleId: 'someId',
+           descriptions: [{'a': 2}],
+           path: 'somePath',
+           metrics: {'a': 'script time', 'b': 'render time'}
+         })
+             .reportSample(
+                 [mv(0, 0, {'a': 3, 'b': 6})],
+                 [mv(0, 0, {'a': 3, 'b': 6}), mv(1, 1, {'a': 5, 'b': 9})]);
+         const regExp = /somePath\/someId_\d+\.json/;
+         expect(loggedFile['filename'].match(regExp) != null).toBe(true);
+         const parsedContent = JSON.parse(loggedFile['content']);
+         expect(parsedContent).toEqual({
+           'description': {
+             'id': 'someId',
+             'description': {'a': 2},
+             'metrics': {'a': 'script time', 'b': 'render time'}
+           },
+           'stats': {'a': '4.00+-25%', 'b': '7.50+-20%'},
+           'completeSample': [
+             {'timeStamp': '1970-01-01T00:00:00.000Z', 'runIndex': 0, 'values': {'a': 3, 'b': 6}}
+           ],
+           'validSample': [
+             {'timeStamp': '1970-01-01T00:00:00.000Z', 'runIndex': 0, 'values': {'a': 3, 'b': 6}}, {
+               'timeStamp': '1970-01-01T00:00:00.001Z',
+               'runIndex': 1,
+               'values': {'a': 5, 'b': 9}
+             }
+           ]
+         });
+         async.done();
+       }));
+
+  });
+}
+
+function mv(runIndex: number, time: number, values: {[key: string]: number}) {
+  return new MeasureValues(runIndex, new Date(time), values);
+}

packages/benchpress/test/reporter/multi_reporter_spec.ts

@@ -0,0 +1,68 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {AsyncTestCompleter, describe, expect, inject, it} from '@angular/core/testing/testing_internal';
+
+import {MeasureValues, MultiReporter, ReflectiveInjector, Reporter} from '../../index';
+
+export function main() {
+  function createReporters(ids: any[]) {
+    const r = ReflectiveInjector
+                  .resolveAndCreate([
+                    ids.map(id => ({provide: id, useValue: new MockReporter(id)})),
+                    MultiReporter.provideWith(ids)
+                  ])
+                  .get(MultiReporter);
+    return Promise.resolve(r);
+  }
+
+  describe('multi reporter', () => {
+
+    it('should reportMeasureValues to all',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         const mv = new MeasureValues(0, new Date(), {});
+         createReporters(['m1', 'm2']).then((r) => r.reportMeasureValues(mv)).then((values) => {
+
+           expect(values).toEqual([{'id': 'm1', 'values': mv}, {'id': 'm2', 'values': mv}]);
+           async.done();
+         });
+       }));
+
+    it('should reportSample to call', inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         const completeSample =
+             [new MeasureValues(0, new Date(), {}), new MeasureValues(1, new Date(), {})];
+         const validSample = [completeSample[1]];
+
+         createReporters(['m1', 'm2'])
+             .then((r) => r.reportSample(completeSample, validSample))
+             .then((values) => {
+
+               expect(values).toEqual([
+                 {'id': 'm1', 'completeSample': completeSample, 'validSample': validSample},
+                 {'id': 'm2', 'completeSample': completeSample, 'validSample': validSample}
+               ]);
+               async.done();
+             });
+       }));
+
+  });
+}
+
+class MockReporter extends Reporter {
+  constructor(private _id: string) { super(); }
+
+  reportMeasureValues(values: MeasureValues): Promise<{[key: string]: any}> {
+    return Promise.resolve({'id': this._id, 'values': values});
+  }
+
+  reportSample(completeSample: MeasureValues[], validSample: MeasureValues[]):
+      Promise<{[key: string]: any}> {
+    return Promise.resolve(
+        {'id': this._id, 'completeSample': completeSample, 'validSample': validSample});
+  }
+}

packages/benchpress/test/runner_spec.ts

@@ -0,0 +1,140 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {AsyncTestCompleter, describe, expect, inject, it} from '@angular/core/testing/testing_internal';
+
+import {Injector, Metric, Options, ReflectiveInjector, Runner, SampleDescription, SampleState, Sampler, Validator, WebDriverAdapter} from '../index';
+
+export function main() {
+  describe('runner', () => {
+    let injector: ReflectiveInjector;
+    let runner: Runner;
+
+    function createRunner(defaultProviders: any[] = null): Runner {
+      if (!defaultProviders) {
+        defaultProviders = [];
+      }
+      runner = new Runner([
+        defaultProviders, {
+          provide: Sampler,
+          useFactory: (_injector: ReflectiveInjector) => {
+            injector = _injector;
+            return new MockSampler();
+          },
+          deps: [Injector]
+        },
+        {provide: Metric, useFactory: () => new MockMetric(), deps: []},
+        {provide: Validator, useFactory: () => new MockValidator(), deps: []},
+        {provide: WebDriverAdapter, useFactory: () => new MockWebDriverAdapter(), deps: []}
+      ]);
+      return runner;
+    }
+
+    it('should set SampleDescription.id',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createRunner()
+             .sample({id: 'someId'})
+             .then((_) => injector.get(SampleDescription))
+             .then((desc) => {
+               expect(desc.id).toBe('someId');
+               async.done();
+             });
+       }));
+
+    it('should merge SampleDescription.description',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createRunner([{provide: Options.DEFAULT_DESCRIPTION, useValue: {'a': 1}}])
+             .sample({
+               id: 'someId',
+               providers: [{provide: Options.SAMPLE_DESCRIPTION, useValue: {'b': 2}}]
+             })
+             .then((_) => injector.get(SampleDescription))
+             .then((desc) => {
+               expect(desc.description)
+                   .toEqual(
+                       {'forceGc': false, 'userAgent': 'someUserAgent', 'a': 1, 'b': 2, 'v': 11});
+               async.done();
+             });
+       }));
+
+    it('should fill SampleDescription.metrics from the Metric',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createRunner()
+             .sample({id: 'someId'})
+             .then((_) => injector.get(SampleDescription))
+             .then((desc) => {
+
+               expect(desc.metrics).toEqual({'m1': 'some metric'});
+               async.done();
+             });
+       }));
+
+    it('should provide Options.EXECUTE',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         const execute = () => {};
+         createRunner().sample({id: 'someId', execute: execute}).then((_) => {
+           expect(injector.get(Options.EXECUTE)).toEqual(execute);
+           async.done();
+         });
+       }));
+
+    it('should provide Options.PREPARE',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         const prepare = () => {};
+         createRunner().sample({id: 'someId', prepare: prepare}).then((_) => {
+           expect(injector.get(Options.PREPARE)).toEqual(prepare);
+           async.done();
+         });
+       }));
+
+    it('should provide Options.MICRO_METRICS',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createRunner().sample({id: 'someId', microMetrics: {'a': 'b'}}).then((_) => {
+           expect(injector.get(Options.MICRO_METRICS)).toEqual({'a': 'b'});
+           async.done();
+         });
+       }));
+
+    it('should overwrite providers per sample call',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createRunner([{provide: Options.DEFAULT_DESCRIPTION, useValue: {'a': 1}}])
+             .sample({
+               id: 'someId',
+               providers: [{provide: Options.DEFAULT_DESCRIPTION, useValue: {'a': 2}}]
+             })
+             .then((_) => injector.get(SampleDescription))
+             .then((desc) => {
+
+               expect(desc.description['a']).toBe(2);
+               async.done();
+             });
+
+       }));
+
+  });
+}
+
+class MockWebDriverAdapter extends WebDriverAdapter {
+  executeScript(script: string): Promise<string> { return Promise.resolve('someUserAgent'); }
+  capabilities(): Promise<Map<string, any>> { return null; }
+}
+
+class MockValidator extends Validator {
+  constructor() { super(); }
+  describe() { return {'v': 11}; }
+}
+
+class MockMetric extends Metric {
+  constructor() { super(); }
+  describe() { return {'m1': 'some metric'}; }
+}
+
+class MockSampler extends Sampler {
+  constructor() { super(null, null, null, null, null, null, null); }
+  sample(): Promise<SampleState> { return Promise.resolve(new SampleState([], [])); }
+}

packages/benchpress/test/sampler_spec.ts

@@ -0,0 +1,270 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {AsyncTestCompleter, describe, expect, inject, it} from '@angular/core/testing/testing_internal';
+
+import {MeasureValues, Metric, Options, ReflectiveInjector, Reporter, Sampler, Validator, WebDriverAdapter} from '../index';
+
+export function main() {
+  const EMPTY_EXECUTE = () => {};
+
+  describe('sampler', () => {
+    let sampler: Sampler;
+
+    function createSampler({driver, metric, reporter, validator, prepare, execute}: {
+      driver?: any,
+      metric?: Metric,
+      reporter?: Reporter,
+      validator?: Validator,
+      prepare?: any,
+      execute?: any
+    } = {}) {
+      let time = 1000;
+      if (!metric) {
+        metric = new MockMetric([]);
+      }
+      if (!reporter) {
+        reporter = new MockReporter([]);
+      }
+      if (driver == null) {
+        driver = new MockDriverAdapter([]);
+      }
+      const providers = [
+        Options.DEFAULT_PROVIDERS, Sampler.PROVIDERS, {provide: Metric, useValue: metric},
+        {provide: Reporter, useValue: reporter}, {provide: WebDriverAdapter, useValue: driver},
+        {provide: Options.EXECUTE, useValue: execute}, {provide: Validator, useValue: validator},
+        {provide: Options.NOW, useValue: () => new Date(time++)}
+      ];
+      if (prepare != null) {
+        providers.push({provide: Options.PREPARE, useValue: prepare});
+      }
+
+      sampler = ReflectiveInjector.resolveAndCreate(providers).get(Sampler);
+    }
+
+    it('should call the prepare and execute callbacks using WebDriverAdapter.waitFor',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         const log: any[] = [];
+         let count = 0;
+         const driver = new MockDriverAdapter([], (callback: Function) => {
+           const result = callback();
+           log.push(result);
+           return Promise.resolve(result);
+         });
+         createSampler({
+           driver: driver,
+           validator: createCountingValidator(2),
+           prepare: () => count++,
+           execute: () => count++,
+         });
+         sampler.sample().then((_) => {
+           expect(count).toBe(4);
+           expect(log).toEqual([0, 1, 2, 3]);
+           async.done();
+         });
+
+       }));
+
+    it('should call prepare, beginMeasure, execute, endMeasure for every iteration',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         let workCount = 0;
+         const log: any[] = [];
+         createSampler({
+           metric: createCountingMetric(log),
+           validator: createCountingValidator(2),
+           prepare: () => { log.push(`p${workCount++}`); },
+           execute: () => { log.push(`w${workCount++}`); }
+         });
+         sampler.sample().then((_) => {
+           expect(log).toEqual([
+             'p0',
+             ['beginMeasure'],
+             'w1',
+             ['endMeasure', false, {'script': 0}],
+             'p2',
+             ['beginMeasure'],
+             'w3',
+             ['endMeasure', false, {'script': 1}],
+           ]);
+           async.done();
+         });
+       }));
+
+    it('should call execute, endMeasure for every iteration if there is no prepare callback',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         const log: any[] = [];
+         let workCount = 0;
+         createSampler({
+           metric: createCountingMetric(log),
+           validator: createCountingValidator(2),
+           execute: () => { log.push(`w${workCount++}`); },
+           prepare: null
+         });
+         sampler.sample().then((_) => {
+           expect(log).toEqual([
+             ['beginMeasure'],
+             'w0',
+             ['endMeasure', true, {'script': 0}],
+             'w1',
+             ['endMeasure', true, {'script': 1}],
+           ]);
+           async.done();
+         });
+       }));
+
+    it('should only collect metrics for execute and ignore metrics from prepare',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         let scriptTime = 0;
+         let iterationCount = 1;
+         createSampler({
+           validator: createCountingValidator(2),
+           metric: new MockMetric(
+               [],
+               () => {
+                 const result = Promise.resolve({'script': scriptTime});
+                 scriptTime = 0;
+                 return result;
+               }),
+           prepare: () => { scriptTime = 1 * iterationCount; },
+           execute: () => {
+             scriptTime = 10 * iterationCount;
+             iterationCount++;
+           }
+         });
+         sampler.sample().then((state) => {
+           expect(state.completeSample.length).toBe(2);
+           expect(state.completeSample[0]).toEqual(mv(0, 1000, {'script': 10}));
+           expect(state.completeSample[1]).toEqual(mv(1, 1001, {'script': 20}));
+           async.done();
+         });
+       }));
+
+    it('should call the validator for every execution and store the valid sample',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         const log: any[] = [];
+         const validSample = [mv(null, null, {})];
+
+         createSampler({
+           metric: createCountingMetric(),
+           validator: createCountingValidator(2, validSample, log),
+           execute: EMPTY_EXECUTE
+         });
+         sampler.sample().then((state) => {
+           expect(state.validSample).toBe(validSample);
+           // TODO(tbosch): Why does this fail??
+           // expect(log).toEqual([
+           //   ['validate', [{'script': 0}], null],
+           //   ['validate', [{'script': 0}, {'script': 1}], validSample]
+           // ]);
+
+           expect(log.length).toBe(2);
+           expect(log[0]).toEqual(['validate', [mv(0, 1000, {'script': 0})], null]);
+           expect(log[1]).toEqual(
+               ['validate', [mv(0, 1000, {'script': 0}), mv(1, 1001, {'script': 1})], validSample]);
+
+           async.done();
+         });
+       }));
+
+    it('should report the metric values',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         const log: any[] = [];
+         const validSample = [mv(null, null, {})];
+         createSampler({
+           validator: createCountingValidator(2, validSample),
+           metric: createCountingMetric(),
+           reporter: new MockReporter(log),
+           execute: EMPTY_EXECUTE
+         });
+         sampler.sample().then((_) => {
+           // TODO(tbosch): Why does this fail??
+           // expect(log).toEqual([
+           //   ['reportMeasureValues', 0, {'script': 0}],
+           //   ['reportMeasureValues', 1, {'script': 1}],
+           //   ['reportSample', [{'script': 0}, {'script': 1}], validSample]
+           // ]);
+           expect(log.length).toBe(3);
+           expect(log[0]).toEqual(['reportMeasureValues', mv(0, 1000, {'script': 0})]);
+           expect(log[1]).toEqual(['reportMeasureValues', mv(1, 1001, {'script': 1})]);
+           expect(log[2]).toEqual([
+             'reportSample', [mv(0, 1000, {'script': 0}), mv(1, 1001, {'script': 1})], validSample
+           ]);
+
+           async.done();
+         });
+       }));
+
+  });
+}
+
+function mv(runIndex: number, time: number, values: {[key: string]: number}) {
+  return new MeasureValues(runIndex, new Date(time), values);
+}
+
+function createCountingValidator(
+    count: number, validSample: MeasureValues[] = null, log: any[] = []) {
+  return new MockValidator(log, (completeSample: MeasureValues[]) => {
+    count--;
+    if (count === 0) {
+      return validSample || completeSample;
+    } else {
+      return null;
+    }
+  });
+}
+
+function createCountingMetric(log: any[] = []) {
+  let scriptTime = 0;
+  return new MockMetric(log, () => ({'script': scriptTime++}));
+}
+
+class MockDriverAdapter extends WebDriverAdapter {
+  constructor(private _log: any[] = [], private _waitFor: Function = null) { super(); }
+  waitFor(callback: Function): Promise<any> {
+    if (this._waitFor != null) {
+      return this._waitFor(callback);
+    } else {
+      return Promise.resolve(callback());
+    }
+  }
+}
+
+
+class MockValidator extends Validator {
+  constructor(private _log: any[] = [], private _validate: Function = null) { super(); }
+  validate(completeSample: MeasureValues[]): MeasureValues[] {
+    const stableSample = this._validate != null ? this._validate(completeSample) : completeSample;
+    this._log.push(['validate', completeSample, stableSample]);
+    return stableSample;
+  }
+}
+
+class MockMetric extends Metric {
+  constructor(private _log: any[] = [], private _endMeasure: Function = null) { super(); }
+  beginMeasure() {
+    this._log.push(['beginMeasure']);
+    return Promise.resolve(null);
+  }
+  endMeasure(restart: boolean) {
+    const measureValues = this._endMeasure != null ? this._endMeasure() : {};
+    this._log.push(['endMeasure', restart, measureValues]);
+    return Promise.resolve(measureValues);
+  }
+}
+
+class MockReporter extends Reporter {
+  constructor(private _log: any[] = []) { super(); }
+  reportMeasureValues(values: MeasureValues): Promise<any> {
+    this._log.push(['reportMeasureValues', values]);
+    return Promise.resolve(null);
+  }
+  reportSample(completeSample: MeasureValues[], validSample: MeasureValues[]): Promise<any> {
+    this._log.push(['reportSample', completeSample, validSample]);
+    return Promise.resolve(null);
+  }
+}

packages/benchpress/test/statistic_spec.ts

@@ -0,0 +1,39 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {describe, expect, it} from '@angular/core/testing/testing_internal';
+import {Statistic} from '../src/statistic';
+
+export function main() {
+  describe('statistic', () => {
+
+    it('should calculate the mean', () => {
+      expect(Statistic.calculateMean([])).toBeNaN();
+      expect(Statistic.calculateMean([1, 2, 3])).toBe(2.0);
+    });
+
+    it('should calculate the standard deviation', () => {
+      expect(Statistic.calculateStandardDeviation([], NaN)).toBeNaN();
+      expect(Statistic.calculateStandardDeviation([1], 1)).toBe(0.0);
+      expect(Statistic.calculateStandardDeviation([2, 4, 4, 4, 5, 5, 7, 9], 5)).toBe(2.0);
+    });
+
+    it('should calculate the coefficient of variation', () => {
+      expect(Statistic.calculateCoefficientOfVariation([], NaN)).toBeNaN();
+      expect(Statistic.calculateCoefficientOfVariation([1], 1)).toBe(0.0);
+      expect(Statistic.calculateCoefficientOfVariation([2, 4, 4, 4, 5, 5, 7, 9], 5)).toBe(40.0);
+    });
+
+    it('should calculate the regression slope', () => {
+      expect(Statistic.calculateRegressionSlope([], NaN, [], NaN)).toBeNaN();
+      expect(Statistic.calculateRegressionSlope([1], 1, [2], 2)).toBeNaN();
+      expect(Statistic.calculateRegressionSlope([1, 2], 1.5, [2, 4], 3)).toBe(2.0);
+    });
+
+  });
+}

packages/benchpress/test/trace_event_factory.ts

@@ -0,0 +1,40 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {PerfLogEvent} from '../index';
+
+export class TraceEventFactory {
+  constructor(private _cat: string, private _pid: string) {}
+
+  create(ph: any, name: string, time: number, args: any = null) {
+    const res:
+        PerfLogEvent = {'name': name, 'cat': this._cat, 'ph': ph, 'ts': time, 'pid': this._pid};
+    if (args != null) {
+      res['args'] = args;
+    }
+    return res;
+  }
+
+  markStart(name: string, time: number) { return this.create('B', name, time); }
+
+  markEnd(name: string, time: number) { return this.create('E', name, time); }
+
+  start(name: string, time: number, args: any = null) { return this.create('B', name, time, args); }
+
+  end(name: string, time: number, args: any = null) { return this.create('E', name, time, args); }
+
+  instant(name: string, time: number, args: any = null) {
+    return this.create('I', name, time, args);
+  }
+
+  complete(name: string, time: number, duration: number, args: any = null) {
+    const res = this.create('X', name, time, args);
+    res['dur'] = duration;
+    return res;
+  }
+}

packages/benchpress/test/validator/regression_slope_validator_spec.ts

@@ -0,0 +1,62 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {describe, expect, it} from '@angular/core/testing/testing_internal';
+
+import {MeasureValues, ReflectiveInjector, RegressionSlopeValidator} from '../../index';
+
+export function main() {
+  describe('regression slope validator', () => {
+    let validator: RegressionSlopeValidator;
+
+    function createValidator({size, metric}: {size: number, metric: string}) {
+      validator = ReflectiveInjector
+                      .resolveAndCreate([
+                        RegressionSlopeValidator.PROVIDERS,
+                        {provide: RegressionSlopeValidator.METRIC, useValue: metric},
+                        {provide: RegressionSlopeValidator.SAMPLE_SIZE, useValue: size}
+                      ])
+                      .get(RegressionSlopeValidator);
+    }
+
+    it('should return sampleSize and metric as description', () => {
+      createValidator({size: 2, metric: 'script'});
+      expect(validator.describe()).toEqual({'sampleSize': 2, 'regressionSlopeMetric': 'script'});
+    });
+
+    it('should return null while the completeSample is smaller than the given size', () => {
+      createValidator({size: 2, metric: 'script'});
+      expect(validator.validate([])).toBe(null);
+      expect(validator.validate([mv(0, 0, {})])).toBe(null);
+    });
+
+    it('should return null while the regression slope is < 0', () => {
+      createValidator({size: 2, metric: 'script'});
+      expect(validator.validate([mv(0, 0, {'script': 2}), mv(1, 1, {'script': 1})])).toBe(null);
+    });
+
+    it('should return the last sampleSize runs when the regression slope is ==0', () => {
+      createValidator({size: 2, metric: 'script'});
+      const sample = [mv(0, 0, {'script': 1}), mv(1, 1, {'script': 1}), mv(2, 2, {'script': 1})];
+      expect(validator.validate(sample.slice(0, 2))).toEqual(sample.slice(0, 2));
+      expect(validator.validate(sample)).toEqual(sample.slice(1, 3));
+    });
+
+    it('should return the last sampleSize runs when the regression slope is >0', () => {
+      createValidator({size: 2, metric: 'script'});
+      const sample = [mv(0, 0, {'script': 1}), mv(1, 1, {'script': 2}), mv(2, 2, {'script': 3})];
+      expect(validator.validate(sample.slice(0, 2))).toEqual(sample.slice(0, 2));
+      expect(validator.validate(sample)).toEqual(sample.slice(1, 3));
+    });
+
+  });
+}
+
+function mv(runIndex: number, time: number, values: {[key: string]: number}) {
+  return new MeasureValues(runIndex, new Date(time), values);
+}

packages/benchpress/test/validator/size_validator_spec.ts

@@ -0,0 +1,48 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {describe, expect, it} from '@angular/core/testing/testing_internal';
+
+import {MeasureValues, ReflectiveInjector, SizeValidator} from '../../index';
+
+export function main() {
+  describe('size validator', () => {
+    let validator: SizeValidator;
+
+    function createValidator(size: number) {
+      validator =
+          ReflectiveInjector
+              .resolveAndCreate(
+                  [SizeValidator.PROVIDERS, {provide: SizeValidator.SAMPLE_SIZE, useValue: size}])
+              .get(SizeValidator);
+    }
+
+    it('should return sampleSize as description', () => {
+      createValidator(2);
+      expect(validator.describe()).toEqual({'sampleSize': 2});
+    });
+
+    it('should return null while the completeSample is smaller than the given size', () => {
+      createValidator(2);
+      expect(validator.validate([])).toBe(null);
+      expect(validator.validate([mv(0, 0, {})])).toBe(null);
+    });
+
+    it('should return the last sampleSize runs when it has at least the given size', () => {
+      createValidator(2);
+      const sample = [mv(0, 0, {'a': 1}), mv(1, 1, {'b': 2}), mv(2, 2, {'c': 3})];
+      expect(validator.validate(sample.slice(0, 2))).toEqual(sample.slice(0, 2));
+      expect(validator.validate(sample)).toEqual(sample.slice(1, 3));
+    });
+
+  });
+}
+
+function mv(runIndex: number, time: number, values: {[key: string]: number}) {
+  return new MeasureValues(runIndex, new Date(time), values);
+}

packages/benchpress/test/web_driver_extension_spec.ts

@@ -0,0 +1,56 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {AsyncTestCompleter, describe, expect, inject, it} from '@angular/core/testing/testing_internal';
+
+import {Options, ReflectiveInjector, WebDriverExtension} from '../index';
+
+export function main() {
+  function createExtension(ids: any[], caps: any) {
+    return new Promise<any>((res, rej) => {
+      try {
+        res(ReflectiveInjector
+                .resolveAndCreate([
+                  ids.map((id) => ({provide: id, useValue: new MockExtension(id)})),
+                  {provide: Options.CAPABILITIES, useValue: caps},
+                  WebDriverExtension.provideFirstSupported(ids)
+                ])
+                .get(WebDriverExtension));
+      } catch (e) {
+        rej(e);
+      }
+    });
+  }
+
+  describe('WebDriverExtension.provideFirstSupported', () => {
+
+    it('should provide the extension that matches the capabilities',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createExtension(['m1', 'm2', 'm3'], {'browser': 'm2'}).then((m) => {
+           expect(m.id).toEqual('m2');
+           async.done();
+         });
+       }));
+
+    it('should throw if there is no match',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createExtension(['m1'], {'browser': 'm2'}).catch((err) => {
+           expect(err != null).toBe(true);
+           async.done();
+         });
+       }));
+  });
+}
+
+class MockExtension extends WebDriverExtension {
+  constructor(public id: string) { super(); }
+
+  supports(capabilities: {[key: string]: any}): boolean {
+    return capabilities['browser'] === this.id;
+  }
+}

packages/benchpress/test/webdriver/chrome_driver_extension_spec.ts

@@ -0,0 +1,407 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {AsyncTestCompleter, describe, expect, inject, it} from '@angular/core/testing/testing_internal';
+
+import {ChromeDriverExtension, Options, ReflectiveInjector, WebDriverAdapter, WebDriverExtension} from '../../index';
+import {TraceEventFactory} from '../trace_event_factory';
+
+export function main() {
+  describe('chrome driver extension', () => {
+    const CHROME45_USER_AGENT =
+        '"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2499.0 Safari/537.36"';
+
+    let log: any[];
+    let extension: ChromeDriverExtension;
+
+    const blinkEvents = new TraceEventFactory('blink.console', 'pid0');
+    const v8Events = new TraceEventFactory('v8', 'pid0');
+    const v8EventsOtherProcess = new TraceEventFactory('v8', 'pid1');
+    const chromeTimelineEvents =
+        new TraceEventFactory('disabled-by-default-devtools.timeline', 'pid0');
+    const chrome45TimelineEvents = new TraceEventFactory('devtools.timeline', 'pid0');
+    const chromeTimelineV8Events = new TraceEventFactory('devtools.timeline,v8', 'pid0');
+    const chromeBlinkTimelineEvents = new TraceEventFactory('blink,devtools.timeline', 'pid0');
+    const chromeBlinkUserTimingEvents = new TraceEventFactory('blink.user_timing', 'pid0');
+    const benchmarkEvents = new TraceEventFactory('benchmark', 'pid0');
+    const normEvents = new TraceEventFactory('timeline', 'pid0');
+
+    function createExtension(
+        perfRecords: any[] = null, userAgent: string = null,
+        messageMethod = 'Tracing.dataCollected'): WebDriverExtension {
+      if (!perfRecords) {
+        perfRecords = [];
+      }
+      if (userAgent == null) {
+        userAgent = CHROME45_USER_AGENT;
+      }
+      log = [];
+      extension = ReflectiveInjector
+                      .resolveAndCreate([
+                        ChromeDriverExtension.PROVIDERS, {
+                          provide: WebDriverAdapter,
+                          useValue: new MockDriverAdapter(log, perfRecords, messageMethod)
+                        },
+                        {provide: Options.USER_AGENT, useValue: userAgent}
+                      ])
+                      .get(ChromeDriverExtension);
+      return extension;
+    }
+
+    it('should force gc via window.gc()',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createExtension().gc().then((_) => {
+           expect(log).toEqual([['executeScript', 'window.gc()']]);
+           async.done();
+         });
+       }));
+
+    it('should mark the timeline via console.time()',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createExtension().timeBegin('someName').then((_) => {
+           expect(log).toEqual([['executeScript', `console.time('someName');`]]);
+           async.done();
+         });
+       }));
+
+    it('should mark the timeline via console.timeEnd()',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createExtension().timeEnd('someName', null).then((_) => {
+           expect(log).toEqual([['executeScript', `console.timeEnd('someName');`]]);
+           async.done();
+         });
+       }));
+
+    it('should mark the timeline via console.time() and console.timeEnd()',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createExtension().timeEnd('name1', 'name2').then((_) => {
+           expect(log).toEqual(
+               [['executeScript', `console.timeEnd('name1');console.time('name2');`]]);
+           async.done();
+         });
+       }));
+
+    it('should normalize times to ms and forward ph and pid event properties',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createExtension([chromeTimelineV8Events.complete('FunctionCall', 1100, 5500, null)])
+             .readPerfLog()
+             .then((events) => {
+               expect(events).toEqual([
+                 normEvents.complete('script', 1.1, 5.5, null),
+               ]);
+               async.done();
+             });
+       }));
+
+    it('should normalize "tdur" to "dur"',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         const event: any = chromeTimelineV8Events.create('X', 'FunctionCall', 1100, null);
+         event['tdur'] = 5500;
+         createExtension([event]).readPerfLog().then((events) => {
+           expect(events).toEqual([
+             normEvents.complete('script', 1.1, 5.5, null),
+           ]);
+           async.done();
+         });
+       }));
+
+    it('should report FunctionCall events as "script"',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createExtension([chromeTimelineV8Events.start('FunctionCall', 0)])
+             .readPerfLog()
+             .then((events) => {
+               expect(events).toEqual([
+                 normEvents.start('script', 0),
+               ]);
+               async.done();
+             });
+       }));
+
+    it('should report EvaluateScript events as "script"',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createExtension([chromeTimelineV8Events.start('EvaluateScript', 0)])
+             .readPerfLog()
+             .then((events) => {
+               expect(events).toEqual([
+                 normEvents.start('script', 0),
+               ]);
+               async.done();
+             });
+       }));
+
+    it('should report minor gc', inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createExtension([
+           chromeTimelineV8Events.start('MinorGC', 1000, {'usedHeapSizeBefore': 1000}),
+           chromeTimelineV8Events.end('MinorGC', 2000, {'usedHeapSizeAfter': 0}),
+         ])
+             .readPerfLog()
+             .then((events) => {
+               expect(events.length).toEqual(2);
+               expect(events[0]).toEqual(
+                   normEvents.start('gc', 1.0, {'usedHeapSize': 1000, 'majorGc': false}));
+               expect(events[1]).toEqual(
+                   normEvents.end('gc', 2.0, {'usedHeapSize': 0, 'majorGc': false}));
+               async.done();
+             });
+       }));
+
+    it('should report major gc', inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createExtension(
+             [
+               chromeTimelineV8Events.start('MajorGC', 1000, {'usedHeapSizeBefore': 1000}),
+               chromeTimelineV8Events.end('MajorGC', 2000, {'usedHeapSizeAfter': 0}),
+             ], )
+             .readPerfLog()
+             .then((events) => {
+               expect(events.length).toEqual(2);
+               expect(events[0]).toEqual(
+                   normEvents.start('gc', 1.0, {'usedHeapSize': 1000, 'majorGc': true}));
+               expect(events[1]).toEqual(
+                   normEvents.end('gc', 2.0, {'usedHeapSize': 0, 'majorGc': true}));
+               async.done();
+             });
+       }));
+
+    ['Layout', 'UpdateLayerTree', 'Paint'].forEach((recordType) => {
+      it(`should report ${recordType} as "render"`,
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           createExtension(
+               [
+                 chrome45TimelineEvents.start(recordType, 1234),
+                 chrome45TimelineEvents.end(recordType, 2345)
+               ], )
+               .readPerfLog()
+               .then((events) => {
+                 expect(events).toEqual([
+                   normEvents.start('render', 1.234),
+                   normEvents.end('render', 2.345),
+                 ]);
+                 async.done();
+               });
+         }));
+    });
+
+    it(`should report UpdateLayoutTree as "render"`,
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createExtension(
+             [
+               chromeBlinkTimelineEvents.start('UpdateLayoutTree', 1234),
+               chromeBlinkTimelineEvents.end('UpdateLayoutTree', 2345)
+             ], )
+             .readPerfLog()
+             .then((events) => {
+               expect(events).toEqual([
+                 normEvents.start('render', 1.234),
+                 normEvents.end('render', 2.345),
+               ]);
+               async.done();
+             });
+       }));
+
+    it('should ignore FunctionCalls from webdriver',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createExtension([chromeTimelineV8Events.start(
+                             'FunctionCall', 0, {'data': {'scriptName': 'InjectedScript'}})])
+             .readPerfLog()
+             .then((events) => {
+               expect(events).toEqual([]);
+               async.done();
+             });
+       }));
+
+    it('should ignore FunctionCalls with empty scriptName',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createExtension(
+             [chromeTimelineV8Events.start('FunctionCall', 0, {'data': {'scriptName': ''}})])
+             .readPerfLog()
+             .then((events) => {
+               expect(events).toEqual([]);
+               async.done();
+             });
+       }));
+
+    it('should report navigationStart',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createExtension([chromeBlinkUserTimingEvents.instant('navigationStart', 1234)])
+             .readPerfLog()
+             .then((events) => {
+               expect(events).toEqual([normEvents.instant('navigationStart', 1.234)]);
+               async.done();
+             });
+       }));
+
+    it('should report receivedData', inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createExtension([chrome45TimelineEvents.instant(
+                             'ResourceReceivedData', 1234, {'data': {'encodedDataLength': 987}})], )
+             .readPerfLog()
+             .then((events) => {
+               expect(events).toEqual(
+                   [normEvents.instant('receivedData', 1.234, {'encodedDataLength': 987})]);
+               async.done();
+             });
+       }));
+
+    it('should report sendRequest', inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createExtension([chrome45TimelineEvents.instant(
+                             'ResourceSendRequest', 1234,
+                             {'data': {'url': 'http://here', 'requestMethod': 'GET'}})], )
+             .readPerfLog()
+             .then((events) => {
+               expect(events).toEqual([normEvents.instant(
+                   'sendRequest', 1.234, {'url': 'http://here', 'method': 'GET'})]);
+               async.done();
+             });
+       }));
+
+    describe('readPerfLog (common)', () => {
+
+      it('should execute a dummy script before reading them',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           // TODO(tbosch): This seems to be a bug in ChromeDriver:
+           // Sometimes it does not report the newest events of the performance log
+           // to the WebDriver client unless a script is executed...
+           createExtension([]).readPerfLog().then((_) => {
+             expect(log).toEqual([['executeScript', '1+1'], ['logs', 'performance']]);
+             async.done();
+           });
+         }));
+
+      ['Rasterize', 'CompositeLayers'].forEach((recordType) => {
+        it(`should report ${recordType} as "render"`,
+           inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+             createExtension(
+                 [
+                   chromeTimelineEvents.start(recordType, 1234),
+                   chromeTimelineEvents.end(recordType, 2345)
+                 ], )
+                 .readPerfLog()
+                 .then((events) => {
+                   expect(events).toEqual([
+                     normEvents.start('render', 1.234),
+                     normEvents.end('render', 2.345),
+                   ]);
+                   async.done();
+                 });
+           }));
+      });
+
+      describe('frame metrics', () => {
+        it('should report ImplThreadRenderingStats as frame event',
+           inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+             createExtension([benchmarkEvents.instant(
+                                 'BenchmarkInstrumentation::ImplThreadRenderingStats', 1100,
+                                 {'data': {'frame_count': 1}})])
+                 .readPerfLog()
+                 .then((events) => {
+                   expect(events).toEqual([
+                     normEvents.instant('frame', 1.1),
+                   ]);
+                   async.done();
+                 });
+           }));
+
+        it('should not report ImplThreadRenderingStats with zero frames',
+           inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+             createExtension([benchmarkEvents.instant(
+                                 'BenchmarkInstrumentation::ImplThreadRenderingStats', 1100,
+                                 {'data': {'frame_count': 0}})])
+                 .readPerfLog()
+                 .then((events) => {
+                   expect(events).toEqual([]);
+                   async.done();
+                 });
+           }));
+
+        it('should throw when ImplThreadRenderingStats contains more than one frame',
+           inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+
+             createExtension([benchmarkEvents.instant(
+                                 'BenchmarkInstrumentation::ImplThreadRenderingStats', 1100,
+                                 {'data': {'frame_count': 2}})])
+                 .readPerfLog()
+                 .catch((err): any => {
+                   expect(() => {
+                     throw err;
+                   }).toThrowError('multi-frame render stats not supported');
+                   async.done();
+                 });
+           }));
+
+      });
+
+      it('should report begin timestamps',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           createExtension([blinkEvents.create('S', 'someName', 1000)])
+               .readPerfLog()
+               .then((events) => {
+                 expect(events).toEqual([normEvents.markStart('someName', 1.0)]);
+                 async.done();
+               });
+         }));
+
+      it('should report end timestamps',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           createExtension([blinkEvents.create('F', 'someName', 1000)])
+               .readPerfLog()
+               .then((events) => {
+                 expect(events).toEqual([normEvents.markEnd('someName', 1.0)]);
+                 async.done();
+               });
+         }));
+
+      it('should throw an error on buffer overflow',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+
+           createExtension(
+               [
+                 chromeTimelineEvents.start('FunctionCall', 1234),
+               ],
+               CHROME45_USER_AGENT, 'Tracing.bufferUsage')
+               .readPerfLog()
+               .catch((err): any => {
+                 expect(() => {
+                   throw err;
+                 }).toThrowError('The DevTools trace buffer filled during the test!');
+                 async.done();
+               });
+         }));
+
+      it('should match chrome browsers', () => {
+        expect(createExtension().supports({'browserName': 'chrome'})).toBe(true);
+
+        expect(createExtension().supports({'browserName': 'Chrome'})).toBe(true);
+      });
+
+    });
+
+  });
+}
+
+class MockDriverAdapter extends WebDriverAdapter {
+  constructor(private _log: any[], private _events: any[], private _messageMethod: string) {
+    super();
+  }
+
+  executeScript(script: string) {
+    this._log.push(['executeScript', script]);
+    return Promise.resolve(null);
+  }
+
+  logs(type: string) {
+    this._log.push(['logs', type]);
+    if (type === 'performance') {
+      return Promise.resolve(this._events.map(
+          (event) => ({
+            'message': JSON.stringify(
+                {'message': {'method': this._messageMethod, 'params': event}}, null, 2)
+          })));
+    } else {
+      return null;
+    }
+  }
+}

packages/benchpress/test/webdriver/ios_driver_extension_spec.ts

@@ -0,0 +1,195 @@
+/**
+ * @license
+ * Copyright Google Inc. All Rights Reserved.
+ *
+ * Use of this source code is governed by an MIT-style license that can be
+ * found in the LICENSE file at https://angular.io/license
+ */
+
+import {AsyncTestCompleter, describe, expect, inject, it} from '@angular/core/testing/testing_internal';
+
+import {IOsDriverExtension, ReflectiveInjector, WebDriverAdapter, WebDriverExtension} from '../../index';
+import {TraceEventFactory} from '../trace_event_factory';
+
+export function main() {
+  describe('ios driver extension', () => {
+    let log: any[];
+    let extension: IOsDriverExtension;
+
+    const normEvents = new TraceEventFactory('timeline', 'pid0');
+
+    function createExtension(perfRecords: any[] = null): WebDriverExtension {
+      if (!perfRecords) {
+        perfRecords = [];
+      }
+      log = [];
+      extension =
+          ReflectiveInjector
+              .resolveAndCreate([
+                IOsDriverExtension.PROVIDERS,
+                {provide: WebDriverAdapter, useValue: new MockDriverAdapter(log, perfRecords)}
+              ])
+              .get(IOsDriverExtension);
+      return extension;
+    }
+
+    it('should throw on forcing gc', () => {
+      expect(() => createExtension().gc()).toThrowError('Force GC is not supported on iOS');
+    });
+
+    it('should mark the timeline via console.time()',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createExtension().timeBegin('someName').then((_) => {
+           expect(log).toEqual([['executeScript', `console.time('someName');`]]);
+           async.done();
+         });
+       }));
+
+    it('should mark the timeline via console.timeEnd()',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createExtension().timeEnd('someName', null).then((_) => {
+           expect(log).toEqual([['executeScript', `console.timeEnd('someName');`]]);
+           async.done();
+         });
+       }));
+
+    it('should mark the timeline via console.time() and console.timeEnd()',
+       inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+         createExtension().timeEnd('name1', 'name2').then((_) => {
+           expect(log).toEqual(
+               [['executeScript', `console.timeEnd('name1');console.time('name2');`]]);
+           async.done();
+         });
+       }));
+
+    describe('readPerfLog', () => {
+
+      it('should execute a dummy script before reading them',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           // TODO(tbosch): This seems to be a bug in ChromeDriver:
+           // Sometimes it does not report the newest events of the performance log
+           // to the WebDriver client unless a script is executed...
+           createExtension([]).readPerfLog().then((_) => {
+             expect(log).toEqual([['executeScript', '1+1'], ['logs', 'performance']]);
+             async.done();
+           });
+         }));
+
+      it('should report FunctionCall records as "script"',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           createExtension([durationRecord('FunctionCall', 1, 5)]).readPerfLog().then((events) => {
+             expect(events).toEqual([normEvents.start('script', 1), normEvents.end('script', 5)]);
+             async.done();
+           });
+         }));
+
+      it('should ignore FunctionCalls from webdriver',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           createExtension([internalScriptRecord(1, 5)]).readPerfLog().then((events) => {
+             expect(events).toEqual([]);
+             async.done();
+           });
+         }));
+
+      it('should report begin time', inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           createExtension([timeBeginRecord('someName', 12)]).readPerfLog().then((events) => {
+             expect(events).toEqual([normEvents.markStart('someName', 12)]);
+             async.done();
+           });
+         }));
+
+      it('should report end timestamps',
+         inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           createExtension([timeEndRecord('someName', 12)]).readPerfLog().then((events) => {
+             expect(events).toEqual([normEvents.markEnd('someName', 12)]);
+             async.done();
+           });
+         }));
+
+      ['RecalculateStyles', 'Layout', 'UpdateLayerTree', 'Paint', 'Rasterize', 'CompositeLayers']
+          .forEach((recordType) => {
+            it(`should report ${recordType}`,
+               inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+                 createExtension([durationRecord(recordType, 0, 1)])
+                     .readPerfLog()
+                     .then((events) => {
+                       expect(events).toEqual([
+                         normEvents.start('render', 0),
+                         normEvents.end('render', 1),
+                       ]);
+                       async.done();
+                     });
+               }));
+          });
+
+
+      it('should walk children', inject([AsyncTestCompleter], (async: AsyncTestCompleter) => {
+           createExtension([durationRecord('FunctionCall', 1, 5, [timeBeginRecord('someName', 2)])])
+               .readPerfLog()
+               .then((events) => {
+                 expect(events).toEqual([
+                   normEvents.start('script', 1), normEvents.markStart('someName', 2),
+                   normEvents.end('script', 5)
+                 ]);
+                 async.done();
+               });
+         }));
+
+      it('should match safari browsers', () => {
+        expect(createExtension().supports({'browserName': 'safari'})).toBe(true);
+
+        expect(createExtension().supports({'browserName': 'Safari'})).toBe(true);
+      });
+
+    });
+
+  });
+}
+
+function timeBeginRecord(name: string, time: number) {
+  return {'type': 'Time', 'startTime': time, 'data': {'message': name}};
+}
+
+function timeEndRecord(name: string, time: number) {
+  return {'type': 'TimeEnd', 'startTime': time, 'data': {'message': name}};
+}
+
+function durationRecord(type: string, startTime: number, endTime: number, children: any[] = null) {
+  if (!children) {
+    children = [];
+  }
+  return {'type': type, 'startTime': startTime, 'endTime': endTime, 'children': children};
+}
+
+function internalScriptRecord(startTime: number, endTime: number) {
+  return {
+    'type': 'FunctionCall',
+    'startTime': startTime,
+    'endTime': endTime,
+    'data': {'scriptName': 'InjectedScript'}
+  };
+}
+
+class MockDriverAdapter extends WebDriverAdapter {
+  constructor(private _log: any[], private _perfRecords: any[]) { super(); }
+
+  executeScript(script: string) {
+    this._log.push(['executeScript', script]);
+    return Promise.resolve(null);
+  }
+
+  logs(type: string) {
+    this._log.push(['logs', type]);
+    if (type === 'performance') {
+      return Promise.resolve(this._perfRecords.map(function(record) {
+        return {
+          'message': JSON.stringify(
+              {'message': {'method': 'Timeline.eventRecorded', 'params': {'record': record}}}, null,
+              2)
+        };
+      }));
+    } else {
+      return null;
+    }
+  }
+}

packages/benchpress/tsconfig-build.json

@@ -0,0 +1,27 @@
+{
+  "compilerOptions": {
+    "module": "commonjs",
+    "target": "es2015",
+    "lib": ["es6", "dom"],
+    "noImplicitAny": true,
+    "sourceMap": true,
+    "baseUrl": ".",
+    "outDir": "../../../dist/packages-dist/benchpress",
+    "paths": {
+      "@angular/core": ["../../../dist/packages-dist/core"]
+    },
+    "experimentalDecorators": true,
+    "rootDir": ".",
+    "sourceRoot": ".",
+    "declaration": true,
+    "skipLibCheck": true,
+    // don't auto-discover @types/node, it results in a ///<reference in the .d.ts output
+    "types": []
+  },
+  "files": [
+    "index.ts",
+    "../../../node_modules/@types/node/index.d.ts",
+    "../../../node_modules/@types/jasmine/index.d.ts",
+    "../../../node_modules/zone.js/dist/zone.js.d.ts"
+  ]
+}