The three front-end nodes serving this content have high usage spikes at times. What causes this is still unknown, but it's been happening on-and-off for years.
Note that from bug 564113, use are actually seeing 502 Bad Gateway response from the wiki, so it's more than just general slowness, it causes the reverse proxy to timeout.
There are many requests on Marketplace which take the webheads > 1s to respond. Although the responses are cacheable, the request string contains so many parameters about the client that it makes all of these requests quite unique.
If you fetch that same request again, you get a cached response which is great. But change one of java.version, product.version, etc. and it's another 3s of processing time to resolve that request.
There are many requests on Marketplace which take the webheads > 1s to
respond. Although the responses are cacheable, the request string contains
so many parameters about the client that it makes all of these requests
quite unique.
Witness:
$ time wget -S https://marketplace.eclipse.org/popular/top/api/p?client=org.eclipse.epp.mpc.\
core&client.version=1.8.1.v20191106-
1317&os=win32&ws=win32&nl=en_GB&java.version=1.8.0_251&product=org.
eclipse.epp.package.java.product&product.version=4.14.0.I20191210-
0610&runtime.version=3.17.0.v20191122-2104&platform.version=4.14.0.
v20191210-0612
If you fetch that same request again, you get a cached response which is
great. But change one of java.version, product.version, etc. and it's
another 3s of processing time to resolve that request.
This one takes 5s: https://marketplace.eclipse.org/category/free-tagging/fileExtension_accounts.\
json%2CfileExtension_json/api/p?client=org.eclipse.epp.mpc.core&client.
version=1.8.0.v20190725-1807&os=win32&ws=win32&nl=en_US&java.version=1.8.
0_201&product=org.eclipse.epp.package.java.product&product.version=4.13.0.
I20190916-1045&runtime.version=3.16.0.v20190823-1314&platform.version=4.13.0.
v20190916-1045
Eventually, more requests come in that can be resolved in a timely fashion,
and the server CPUs get overloaded.
I believe MP API calls are the cause of the issues.
The marketplace REST API server is designed to return listings based off these variables.
For example, if you are using Eclipse 2020-03, we will only return listings that are compatible with your Eclipse version.
I do agree that we don't need all these parameters. For example, we are not using &nl=en_US. The problem is that a similar environment might not share a cached response since a value might be slightly different.
We are currently working with MPC to improve these requests for version 2 of the API.
@Denis, would it help if we give you a full list of params that we need for the request so that we can configure Nginx to ignore the others?
$ time wget -S https://marketplace.eclipse.org/popular/top/api/p?client=org.eclipse.epp.mpc.\
core&client.version=1.8.1.v20191106-
1317&os=win32&ws=win32&nl=en_GB&java.version=1.8.0_251&product=org.
eclipse.epp.package.java.product&product.version=4.14.0.I20191210-
0610&runtime.version=3.17.0.v20191122-2104&platform.version=4.14.0.
v20191210-0612
I think one of the problems that we have with the hosting of Eclipse Marketplace right now is: A request today is more expensive than a request from last year.
I took a quick look at our stats from May 2019 and compared them with those from last month.
Marketplace served 19,777,485 API requests in May 2020 and 20,264,673 in 2019.
The difference from last year is that we have more 4 extra Eclipse Releases to support:
2020-03
2019-12
2019-09
2019-06
This is causing more requests to hit our php-nodes instead of being cached by our reverse proxy.