Search engine available for Ajax content

In 2009, Google proposed a standard allowing search engines to access web page content dynamically generated by Ajax, as web applications are expected to grow exponentially.

In 2015, the search engine changed its mind and explained this in an article titled: Deprecating our AJAX crawling schemes. (The real name is Ajax, not AJAX, but there are some preconceived notions that cannot be eradicated.)

As of 2015, the article states that it is enough for a crawler to have access to JavaScript and CSS files in order to be able to index dynamic content (this depends heavily on the script). If the site follows the principle of progressive improvement, meaning that content is added little by little through JavaScript, the crawler can access the data and take it into account when indexing.

The 2009 method is here for the record and is no longer relevant except to know what it was. Use pushState; this is now the recommended method.

Using pushState

If your audience uses a recent browser other than IE 9, you can skip all of the above and simply use the pushState method of the history object. Its role is to artificially create a URL for the new page state.

The parameters are an object that stores data related to the page state, a temporary title, and a temporary URL. Example of use:

var state1 = { ref : "info" };
history.pushState(state1, "Titre de page", "nouvelle-page.html");

When a user launches an action representing a new page state, the URL is changed and can be indexed separately. All that remains is to write code that will make this URL directly available!

Recall the 2009 segment method

When an Ajax page changes content using JavaScript code, it creates a new state, and Google seeks to index all possible states of the dynamic page, thus creating links in its index pointing to the corresponding page states.

For this to be possible, each state must correspond to a fragment of the form:

http://www.example.com/mapage#fragment
This is what is automatically generated using the Ajax or HTML 5 framework.
However, the snippet does not return HTTP code; to compensate, Google must artificially convert it into a query form. Queries return code.
http://www.example.com/mapage#_escaped_fragment_?fragment

The fragment text now becomes the value assigned to the generic parameter. This is something invisible to the user.

System Overview

  1. The user creates a static start page. It can include dynamic links and bindings.
    http://www.example.com/index.php?requete#ancre
  2. We know that Google can currently index internal links, see Google Algorithm Evolution (September 25, 2009). This is provided that there is a link inside the page, which is not the case with dynamic content.
  3. Because dynamic content is also available to the user in this format. 
    http://www.example.com/index.php?requete#etat
  4. In addition, JavaScript can change the content of the page either initially or at the request of the user .
  5. However, search engine robots only see this code:
    <script src='showcase.js'></script>
  6. If the search engine tries to interpret JavaScript code, the result will be random.
  7. Google initially proposed that the server interpret the JavaScript code and create a URL corresponding to the state of the page modified by the JavaScript code.
    But it seems that the conversion is actually performed by the crawler himself.
    8. URL
  8. is generated for the corresponding page state as:
    http://www.example.com/index.php?_escaped_fragment_=state
  9. But it would be displayed in such a more readable form:
    http://www.example.com/index.php#!state
  10. The crawler creates a hard-to-read URL. Robots and browsers will be able to convert a readable URL into a hard-to-read one.
  11. Only the readable URL is available to the user and appears in the Google index.

This standard format is now recognized by web executors for server conversion and management.

Resume Summary

http://www.example.com/stock.html#Google

You can easily change (with the added exclamation mark) to:

http://www.example.com/stock.html#!Google

This will be converted by search engines to:

http://www.example.com/stock.html?_escaped_fragment_=Google

But it appears in the results as:

http://www.example.com/stock.html#!Google

How do I make an Ajax page navigable?

With the established principle, what should the webmaster do to get their Ajax page into the system and be recognized by Google as navigable?

The page itself must have this meta tag:

<meta name="fragment" content="!">

When an Ajax script creates a new state, that is, when it changes the content of the page, it must create a fragment that adds to the URL of the page (split #) and replaces it. This allows you to still use the Back button.
The method of creating this fragment depends on your Ajax framework .

If a page contains links to page states generated by Ajax code, these links must contain a fragment and an exclamation mark:

<a href="http://www.example.com/showcase.html#!checkbox