grails - Grails堆空间间歇性溢出

Question

我遇到的那些欢乐的问题之一是断断续续，难以复制的。

经过几分钟或几天的运行后，我在测试服务器上的grails应用失败了。我无法在Dev或本地复制。

这里的stacktrace模式示例从中间截去了它的1000行(这将一直这样转储，直到我重新启动Tomcat为止):

at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
    at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
    at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:117)
    at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:231)
    at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
Caused by: javax.servlet.ServletException: javax.servlet.ServletException: org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
    at net.sf.ehcache.constructs.web.filter.Filter.logThrowable(Filter.java:143)
    at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:91)
    at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:117)
    at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:231)
    at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
......
    at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
    at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
    at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
Caused by: javax.servlet.ServletException: org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
    at grails.plugin.cache.web.filter.AbstractFilter.logThrowable(AbstractFilter.java:116)
    at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:70)
    at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
    at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
    at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:117)
    at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:231)
    at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
    at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
    at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)

..................
    at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
    at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:117)
    at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:231)
    at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
    at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
    at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
    at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:117)
    at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:231)
    at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
    at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
    at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
    at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:117)
    at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:231)
    at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
    at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
    at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
    at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:117)
    at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:231)
    at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
    at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
    at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
    at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:117)
    at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:231)
    at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
    at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
    at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
    at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:117)
    at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:231)
    at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
    at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
    at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
    at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:117)
    at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:231)
    at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
    at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
Caused by: org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
    at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
    at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    ... 132 more
Caused by: java.lang.OutOfMemoryError: Java heap space

看来可能发生了应用程序错误，调用errorHandler并不断调用自身... kaboom!

环境:

Grails:2.0.4

Tomcat:apache-tomcat-7.0.28

操作系统:Red Hat Enterprise Linux Server 5.8版(Tikanga)

Java :

java version "1.6.0_22"
OpenJDK Runtime Environment (IcedTea6 1.10.8) (rhel-1.27.1.10.8.el5_8-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode)

在开发Java中(这可能是我试图让SA在测试系统中更改的主要问题)

java version "1.6.0_30"
Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)

该服务器当前只有0-5个用户登录，但是此过程在24小时内会消退。

分配的堆空间为5g，通常使用<20％。

在Test中发生了另外2件事，而在Dev中没有发生:

日志，最初几乎没有日志记录，但是由于问题，已将其打开，直到DEBUG和RollingFile 100Meg文件在第50个滚动。日志很好，可以记录应用程序特定的内容，直到堆栈崩溃，然后其中唯一的内容就是堆栈，tomcat日志记录也是如此。即使在关闭第50条日志之前就关闭服务器也是如此。因此，我不知道是否发生了某些特定于应用程序的不祥之兆，从而触发了这种情况。

JavaMelody插件:图上的标签有点像我3岁的 child 写的那样不可读。我相信所有OS字体都已加载，但可以指向Java版本。

计划程序:
是的，有2个正在运行，已经运行它们没有问题

数据库配置:
遗留数据库有8个数据源，例如，每个当前可能都设置得有点饿:

def connectionPropertiesMedium = [
        maxActive: 100,
        maxIdle: 30,
        minIdle: 5,
        initialSize: 30,
        testOnBorrow: true,
        testWhileIdle: false,
        testOnReturn: false,
        validationQuery: "SELECT 1",
        minEvictableIdleTimeMillis: 600000,
        timeBetweenEvictionRunsMillis: 600000,
        numTestsPerEvictionRun: 3,
        maxWait: 10000,
        defaultTransactionIsolation: java.sql.Connection.TRANSACTION_READ_UNCOMMITTED
]

HeapDump:
使用 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=heapdump.hprof运行
用VisualVM进行阅读，几乎看不出(无论如何对我来说)以上堆栈跟踪中的char []充满了70％以上。 Eclipse MAT不会加载它。

JVM args:

CATALINA_OPTS="-server -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=heapdump.hprof -XX:MaxPermSize=1024m -XX:MaxNewSize=256m -XX:NewSize=256m -Xms768m -Xmx1024m -XX:SurvivorRatio=128 -XX:MaxTenuringThreshold=0 -XX:+UseTLAB -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSIncrementalMode -XX:-UseGCOverheadLimit -XX:+ExplicitGCInvokesConcurrent

压力测试:
该应用程序可以运行几个激烈的查询，多达200个模拟JMeter用户多次遇到这些查询而没有问题……实际上，数据库确实有点慢了；)但是内存问题并未在上面复制很多次。

因此，如果有人读过这么远，我有2条线索(可以打开更多线索):

Java版本-旨在退出OpenJDK并转为标准发行版

我的Grails设置中发生错误时导致无限/递归处理的某件事。

直到我找到能使这些机器中的鬼魂破坏的质子包，该应用程序才能上线。

有任何想法吗？

干杯...

更新＃1

安装了Java 1.7并删除了OpenJDK，这肯定解决了JavaMelody显示问题。现在，它是一个等待的游戏，以查看这是否解决了主要问题。有趣的是，该网站似乎速度更快。

Answer 1

如果用户多次重新提交登录表单(至j_spring_security_check)，则导致在自定义UserDetailsS​​ervice中调用了User.save()(例如，更新了上次登录时间)，则在每个用户的后续调用中都关闭了Hibernate session 。

将User.save()移到AuthenticationEventListener.onApplicationEvent(AuthenticationSuccessEvent事件)，它可以正常工作。

基本设置来自Spring security core plugin - events上的文档，我的类似于:

在您的配置中添加:grails.plugin.springsecurity.useSecurityEventListener = true
在resources.groovy中:

import security.AuthenticationEventListener
....
authenticationEventListener(AuthenticationEventListener)

AuthenticationEventListener.groovy:

class AuthenticationEventListener implements ApplicationListener<AuthenticationSuccessEvent> {

    void onApplicationEvent(AuthenticationSuccessEvent event) {
        if (event instanceof AuthenticationSuccessEvent) {
            UserDetails userDetails = (UserDetails) event.getAuthentication().getPrincipal()
            def httpSession = SecurityRequestHolder.request.session

            if (!httpSession.loggedIn) {
                httpSession.loggedIn = true
                synchronized (httpSession.loggedIn) {
                    httpSession.timeZone = userDetails.timeZone
                    User.withSession { session ->
                        if (session.isOpen()) {
                            User user = User.findByUsername(userDetails.username, [fetch: [roles: 'eager']])
                            user.lastLoggedIn = new Date()
                            user.save(flush: true)
                        }
                    }
                }
            }
        }
    }
}