[Bug 564056] projects, wiki, marketplace extremely slow to respond
| Bugzilla Link | 564056 |
| Status | NEW |
| Importance | P3 normal |
| Reported | Jun 08, 2020 03:05 EDT |
| Modified | Jun 23, 2020 06:14 EDT |
| Depends on | 564580 |
Description
Created attachment 283192
slow respond
The following page https://projects.eclipse.org/create/project-proposal takes ~50 seconds to respond.
I'm located in France with a fiber connection. Subsequent requests are also very slow.
ttfb.png
Activity
Created attachment 283193 load waterfall
46-second-load.png*** Bug 564076 has been marked as a duplicate of this bug. ***
*** Bug 564113 has been marked as a duplicate of this bug. ***
I've created and incident on https://status.eclipse.org (https://status.eclipse.org/incidents/v7397n30gm2d) to let people now that we are aware and avoid new dups to be created.
Note that from bug 564113, use are actually seeing 502 Bad Gateway response from the wiki, so it's more than just general slowness, it causes the reverse proxy to timeout.
*** Bug 564112 has been marked as a duplicate of this bug. ***
There are many requests on Marketplace which take the webheads > 1s to respond. Although the responses are cacheable, the request string contains so many parameters about the client that it makes all of these requests quite unique.
Witness:
Resolving marketplace.eclipse.org (marketplace.eclipse.org)... 198.41.30.195
Connecting to marketplace.eclipse.org (marketplace.eclipse.org)|198.41.30.195|:443... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 09 Jun 2020 13:21:05 GMT
Content-Type: application/xml; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Expires: Tue, 09 Jun 2020 13:36:09 GMT
Cache-Control: max-age=900
Pragma: cache
X-NodeID: php-vm2
Strict-Transport-Security: max-age=15552000; includeSubDomains; preload
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Proxy-Cache: MISSreal 0m2.863s
If you fetch that same request again, you get a cached response which is great. But change one of java.version, product.version, etc. and it's another 3s of processing time to resolve that request.
Eventually, more requests come in that can be resolved in a timely fashion, and the server CPUs get overloaded.
I believe MP API calls are the cause of the issues.
-
(In reply to Denis Roy from comment #10)
There are many requests on Marketplace which take the webheads > 1s to
respond. Although the responses are cacheable, the request string contains
so many parameters about the client that it makes all of these requests
quite unique.
Witness:
$ time wget -S
https://marketplace.eclipse.org/popular/top/api/p?client=org.eclipse.epp.mpc.\ core&client.version=1.8.1.v20191106-
1317&os=win32&ws=win32&nl=en_GB&java.version=1.8.0_251&product=org.
eclipse.epp.package.java.product&product.version=4.14.0.I20191210-
0610&runtime.version=3.17.0.v20191122-2104&platform.version=4.14.0.
v20191210-0612
Resolving marketplace.eclipse.org (marketplace.eclipse.org)... 198.41.30.195
Connecting to marketplace.eclipse.org
(marketplace.eclipse.org)|198.41.30.195|:443... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 09 Jun 2020 13:21:05 GMT
Content-Type: application/xml; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Expires: Tue, 09 Jun 2020 13:36:09 GMT
Cache-Control: max-age=900
Pragma: cache
X-NodeID: php-vm2
Strict-Transport-Security: max-age=15552000; includeSubDomains; preload
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Proxy-Cache: MISS
real 0m2.863s
If you fetch that same request again, you get a cached response which is
great. But change one of java.version, product.version, etc. and it's
another 3s of processing time to resolve that request.
This one takes 5s:
https://marketplace.eclipse.org/category/free-tagging/fileExtension_accounts.\ json%2CfileExtension_json/api/p?client=org.eclipse.epp.mpc.core&client.
version=1.8.0.v20190725-1807&os=win32&ws=win32&nl=en_US&java.version=1.8.
0_201&product=org.eclipse.epp.package.java.product&product.version=4.13.0.
I20190916-1045&runtime.version=3.16.0.v20190823-1314&platform.version=4.13.0.
v20190916-1045
Eventually, more requests come in that can be resolved in a timely fashion,
and the server CPUs get overloaded.
I believe MP API calls are the cause of the issues.The marketplace REST API server is designed to return listings based off these variables.
For example, if you are using Eclipse 2020-03, we will only return listings that are compatible with your Eclipse version.
I do agree that we don't need all these parameters. For example, we are not using &nl=en_US. The problem is that a similar environment might not share a cached response since a value might be slightly different.
We are currently working with MPC to improve these requests for version 2 of the API.
@Denis, would it help if we give you a full list of params that we need for the request so that we can configure Nginx to ignore the others?
-
(In reply to Christopher Guindon from comment #13)
(In reply to Denis Roy from comment #11)\
I've pinned MP to servers 1 and 2
I've pinned wiki and projects to server 3
We'll provision a couple of additional servers for marketplace while the api
calls can be examined.
We used to host marketplace with 5 nodes and now we are down to 2?Sorry, miss your point that you are planning to scale up Marketplace :)
-
(In reply to Denis Roy from comment #10)
$ time wget -S
https://marketplace.eclipse.org/popular/top/api/p?client=org.eclipse.epp.mpc.\ core&client.version=1.8.1.v20191106-
1317&os=win32&ws=win32&nl=en_GB&java.version=1.8.0_251&product=org.
eclipse.epp.package.java.product&product.version=4.14.0.I20191210-
0610&runtime.version=3.17.0.v20191122-2104&platform.version=4.14.0.
v20191210-0612I think one of the problems that we have with the hosting of Eclipse Marketplace right now is: A request today is more expensive than a request from last year.
I took a quick look at our stats from May 2019 and compared them with those from last month.
Marketplace served 19,777,485 API requests in May 2020 and 20,264,673 in 2019.
The difference from last year is that we have more 4 extra Eclipse Releases to support:
2020-03
2019-12
2019-09
2019-06This is causing more requests to hit our php-nodes instead of being cached by our reverse proxy.
Regardless, a call like this almost takes 6s. I think we should investigate what is taking all that time, and optimize it.
$ time wget -S https://marketplace.eclipse.org/category/free-tagging/fileExtension_accounts.json%2CfileExtension_json/api/p?client=org.eclipse.epp.mpc.core\&client.version=1.8.0.v20190725-1807\&os=win32\&ws=win32\&nl=en_US\&java.version=1.8.0_201\&product=org.eclipse.epp.package.java.product\&product.version=4.13.0.I20190916-1045\&runtime.version=3.16.0.v20190823-1314\&platform.version=4.13.0.v20190916-1011\ Resolving marketplace.eclipse.org (marketplace.eclipse.org)... 198.41.30.195
Connecting to marketplace.eclipse.org (marketplace.eclipse.org)|198.41.30.195|:443... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 09 Jun 2020 15:45:54 GMT
Content-Type: application/xml; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Expires: Tue, 09 Jun 2020 15:59:39 GMT
Cache-Control: max-age=900
Pragma: cache
X-NodeID: php-vm5
Strict-Transport-Security: max-age=15552000; includeSubDomains; preload
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Proxy-Cache: MISSreal 0m5.856s
-
(In reply to Denis Roy from comment #16)
Regardless, a call like this almost takes 6s. I think we should investigate
what is taking all that time, and optimize it.
$ time wget -S
https://marketplace.eclipse.org/category/free-tagging/fileExtension_accounts.\ json%2CfileExtension_json/api/p?client=org.eclipse.epp.mpc.core&client.
version=1.8.0.v20190725-1807&os=win32&ws=win32&nl=en_US&java.version=1.8.
0_201&product=org.eclipse.epp.package.java.product&product.version=4.13.0.
I20190916-1045&runtime.version=3.16.0.v20190823-1314&platform.version=4.13.
0.v20190916-1011
Resolving marketplace.eclipse.org (marketplace.eclipse.org)... 198.41.30.195
Connecting to marketplace.eclipse.org
(marketplace.eclipse.org)|198.41.30.195|:443... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 09 Jun 2020 15:45:54 GMT
Content-Type: application/xml; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Expires: Tue, 09 Jun 2020 15:59:39 GMT
Cache-Control: max-age=900
Pragma: cache
X-NodeID: php-vm5
Strict-Transport-Security: max-age=15552000; includeSubDomains; preload
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Proxy-Cache: MISS
real 0m5.856s5 seconds is a problem.
I am going to start by pruning data that we don't use anymore on Eclipse Marketplace.
This will help Drupal load entities faster such as listings, tags, and solutions versions but I don't expect that to fix the problem right away.
I might also need to set some new restrictions such as the number of tags a listing can have.
From your example, impulse 2 has an excessive number of tags which increase the amount of data that drupal must load before responding to the request.
I will try to free up my afternoon so that I can focus on this problem.
Could bug 564308 be a duplicate, or a symptom of the same issue?
-
*** Bug 564308 has been marked as a duplicate of this bug. ***
