java.lang.OutOfMemoryError: Java heap space
Created by: mbarbero
Since last deployment (July 29: https://foundation.eclipse.org/ci/webdev/job/geoip-rest-api/job/master/136/) geoip containers are being restarted on a regular basis. They are being killed with the following error message
Terminating due to java.lang.OutOfMemoryError: Java heap space It also show some activity spikes following a pattern:
There has been no code change since June 1st thought: https://github.com/EclipseFdn/geoip-rest-api
Both previous image and current image run the same java version
❯ docker run -it --rm --entrypoint /bin/sh eclipsefdn/geoip-rest-api:production-b51c72c-135 -c "java -version"
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (IcedTea 3.15.0) (Alpine 8.242.08-r0)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)
❯ docker run -it --rm --entrypoint /bin/sh eclipsefdn/geoip-rest-api:production-b51c72c-136 -c "java -version"
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (IcedTea 3.15.0) (Alpine 8.242.08-r0)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)It does not seem to be a DoS or a service abuse. When I look at service logs on nginx for today and yesterday (excluding request from 52.157.153.85, as it is the statuspage monitor):
nginx1# cat /var/log/nginx/account.eclipse.org/access.log /var/log/nginx/account.eclipse.org/access.log.1 | grep geoip | grep -v "52.157.153.85" | cut -d ' ' -f1 | sort | uniq -c
136 172.30.206.92
136 198.41.30.195all traffic is coming from internal network. It seems that we now self DoS our service as not a single request completes
I'm able to reproduce with requests like curl http://localhost:8080/geoip/subnets/ipv4/sy
My guess is that maxmind DB has changed in some way and it breaks some assumptions on the app.