Wednesday, 3 September 2014

Insufficient ulimit -u (NPROC) Value Contributes to Native OutOfMemory

I've hit this problem several times in the past few days, mainly on Red Hat Enterprise Linux 6.3 platforms: -

...
An out of memory may be observed on a system running WebSphere Application Server on Linux. Further investigation may reveal a "Failed to create a thread:" message within the generated javacore which would indicate a native out of memory issue has been encountered. The cause of the problem may be an insufficient ulimit setting. While this type of issue can occur on any level of Linux, more recent releases, such as Redhat Enterprise Linux 6, are more likely to encounter this issue due to a decreased default value for certain ulimits. The following will outline how to identify if a process ulimit is the culprit and what WebSphere Application Server Support recommends to fix the case.
...


The trick was to realise that it's not enough to merely extend the ulimits in /etc/security/limits.conf but that, with RHEL 6.X, it's also necessary to amend /etc/security/limits.d/90-nproc.conf as well.

Simples ....

No comments:

Note to self - use kubectl to query images in a pod or deployment

In both cases, we use JSON ... For a deployment, we can do this: - kubectl get deployment foobar --namespace snafu --output jsonpath="{...