Search engine indexes old versions

I've noticed often that when I google for CIF documentation (I find it faster than using the Table of Contents), it links to a specific version page.

For example when I google ESCET CIF documentation, the first hit links to https://www.eclipse.org/escet/v0.1/cifc. Ideally, we want only links to the non-version pages.

I've read that this can be accomplished by creating a robots.txt. See E.g., this issue or this issue, and this example file.

added RelEng/DevOps TypeEnhancement labels

For me the first result is actually good:

However, I do see your point. I'm fine with adding a robots.txt to exclude v*/* or so.

Are you sure the first result is good? I see the same, the first hit links to v0.1, the second indented hit is good.

Dutch Google seems to give the desired result, English one doesn't.

It seems we can't use robots.txt, as it must be in the root, i.e., eclipse.org/robots.txt not eclipse.org/escet/robots.txt. See e.g. here and here.

The alternative seems to be meta tags, but that would require changing already deployed websites for older releases, as well as removing such tags when making them the default website.

but that would require changing already deployed websites for older releases

That should be doable, I think.

as well as removing such tags when making them the default website.

That we could automate in misc/website/switch-standard-visible-website.bash.

Indeed. And we need to generate the meta tags for new websites then as well (single page HTML and multi-page HTML).

I created !462 (closed). Here the meta tag is added during splitting of the HTML pages. It is a simple solution but I'm not sure whether this functionality should be a part of the splitter.

I also found: https://docs.asciidoctor.org/asciidoctorj/latest/extensions/docinfo-processor/ but I'm not sure how that works and if that is a potential solution.

If we can do more with AsciiDoc, that would also be my preference. How about using this: https://docs.asciidoctor.org/asciidoc/latest/docinfo/. Should be enough to add a line to a docinfo file. We have such files already.

assigned to @freijnen

created branch 488-search-engine-indexes-old-versions to address this issue

mentioned in commit 87e673bb

mentioned in merge request !462 (closed)

created branch 488-search-engine-indexes-old-versions-2 to address this issue

mentioned in commit a21a0848

mentioned in merge request !469 (merged)

changed milestone to %v0.9

mentioned in commit c9263a68

How should I commit the changes to the website repository? At https://gitlab.eclipse.org/eclipse/escet/escet-website, I don't see an option for merge request. I'd rather not commit on the master branch directly.

I've enabled merge requests for the website repo.

I created https://gitlab.eclipse.org/eclipse/escet/escet-website/-/tree/488-Search-engine-indexes-old-versions. I cannot create a merge request for it. I'm still working on it, so it is not urgent.

I enabled it again, this time with actually pressing the save button...

mentioned in merge request escet-website!1 (merged)

I created a merge request that adds the meta tags to the version websites, see: escet-website!1 (merged).

I did it via search and replace (Searched for <meta name="viewport" content="width=device-width, initial-scale=1.0"> and replaced with <meta name="robots" content="noindex"> \n <meta name="viewport" content="width=device-width, initial-scale=1.0">. Then I checked which .html files were not edited and changes these by hand.

Not sure if there is anything else to do. We could request a recrawl of our URLs from Google: https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl

Or just wait and then it should change automatically at some point.

It seems you must put the sitemap on the website. Would be some work. Maybe just wait a bit and see? Maybe put an entry in the v0.9-RC1 issue, to check again?

mentioned in issue #480 (closed)

unassigned @freijnen

mentioned in issue #512 (closed)

I think this issue can be closed. ESCET CIF Documentation now links to the newest docs. While you can still find old pages on Google, e.g., ESCET CIF Documentation enumeration, they'll disappear slowly over time.

Indeed, seems to work, but just takes some time.

closed