cargdev/angular
1
0
Fork 0
Code Issues 11 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(aio): update metatags to control search engine crawling (#21665)

Browse Source 
The `<meta name="robots" content="noindex">` tag is used
to indicate to search engine crawlers that they should not index
the current page. This is set dynamically by the the document
viewer component to ensure that 404 and other erroring pages
are not added to the search index.

This relies upon the idea that the crawling bot will run the JS
and wait to see if this meta tag has been added or not.

Since we believe that the `googebot` will do this, we also
pre-emptively add a hard-coded noindex tag specifically for
this bot, so that if anything else fails in bootstrapping the app,
the failed page will not be added to the index.

Closes #21317

PR Close #21665
This commit is contained in:
Pete Bacon Darwin
2018-01-19 14:58:23 +00:00
committed by Misko Hevery
parent 0b38a039d0
commit 88045a5050
5 changed files with 77 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
aio/e2e/app.e2e-spec.ts
View File

@ -128,6 +128,20 @@ describe('site App', function() {
});
describe('404 page', () => {
it('should add or remove the "noindex" meta tag depending upon the validity of the page', () => {
page.navigateTo('');
expect(element(by.css('meta[name="googlebot"]')).isPresent()).toBeFalsy();
expect(element(by.css('meta[name="robots"]')).isPresent()).toBeFalsy();
page.navigateTo('does/not/exist');
expect(element(by.css('meta[name="googlebot"][content="noindex"]')).isPresent()).toBeTruthy();
expect(element(by.css('meta[name="robots"][content="noindex"]')).isPresent()).toBeTruthy();
page.getTopMenuLink('features').click();
expect(element(by.css('meta[name="googlebot"]')).isPresent()).toBeFalsy();
expect(element(by.css('meta[name="robots"]')).isPresent()).toBeFalsy();
});
it('should search the index for words found in the url', () => {
page.navigateTo('http/router');
const results = page.getSearchResults();