Would like to get a sizeable sample of the open-vsx.org access logs
Summary
Our largest DB table is the file_resource table. We're not sure we actually need this table. Ref: https://github.com/EclipseFdn/open-vsx.org/issues/2711. Can I get a sizeable sample of the access logs. I'm looking for references to any of these API calls:
/api/{namespace}/logo/{fileName}
/api/{namespace}/{extension}/{targetPlatform}/{version}/file/**
/api/{namespace}/{extension}/{version}/file/**If we can determine we don't actually need this table, it could significantly reduce our DB migration time.
Priority
- Urgent
- High
- Medium
- Low
Severity
- Blocker
- Major
- Normal
- Low
Activity
added team:infra label
@droy Thanks! I had a chance to crawl through the data. There is clearly a good deal of use of the
/fileAPIs. Here is a breakdown of the file types requested from this set of entries:type count avif 3 gif 1146 ico 16 jpeg 74 jpg 228 json 533 md 1335 png 68546 sigzip 358 vsix 11490 webp 88 I do have a couple of questions about the entries. Take this one for example:
[08/Jul/2024:00:07:30 -0400] "GET /api/rust-lang/rust-analyzer/alpine-x64/0.4.2028/file/icon.png HTTP/1.1" "https/TLSv1.2/ECDHE-RSA-AES256-GCM-SHA384" 302 0 "https://open-vsx.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" "163.***.*.81" - HIT - -I can see it's for an icon file. I'm assuming this is coming from a browser running the browser side of open-vsx.
- Why the
302redirect HTTP code? - Is the
"https://open-vsx.org/"reference the URL the user hit to resulted in this API call? I also see entries likehttps://open-vsx.org/?category=Programming%20Languages&sortBy=relevance&sortOrder=desc. - What is the
"163.***.*.81"reference (my redacting)? In most of the log entries this is not defined, though I do see about 70 different IPs listed. Is this the origin IP of the request that only gets set some of the time?
- Why the
Why the
302redirect HTTP code?The upstream server (open-vsx server) is returning a 302 to https://openvsxorg.blob.core.windows.net/resources/rust-lang/rust-analyzer/alpine-x64/0.4.2028/icon.png
In the specific case you've outlined, the presence of "HIT" means our gateway served a cache response for that request, and the open-vsx server did not see that specific request at all.
Is the "https://open-vsx.org/" reference the URL the user hit to resulted in this API call? I also see entries like
Yep, it's the referrer page.
What is the "163.**..81" reference (my redacting)?
That's a proxy server used by the client, and the client's proxy server added itself to the X-Forwarded-For. If multiple proxy servers are encountered on the client side, they are comma-separated.
Edited by Denis Roy@droy Thanks, this is really helpful (at least for my education). The vast majority of these log entries had the 302 code and "HIT". That means our gateway is handling all of them and they're not hitting the open-vsx server?
mentioned in issue #4941 (closed)
@droy Reopening this one. This information was helpful in analyzing the traffic that came in through the official API. As it turns out there is a compatibility layer that mimics the VS Marketplace server interface so that Open VSX can work with e.g. VS Codium. These resources are also stored in the
file_resourcetable.So, part II of this request. Can I get a sizeable sample of the access logs with all accesses to
/vscode/*URLs, say over a 30 day period. I don't recall from our last results what all was included. Do I get all of the HTTP header info?mentioned in issue #5107 (closed)
