↓ Skip to main content
JDK Tough Way - 6. Practical Guide and Underlying Principles of Tracking Java Heap OOM with JFR
  1. Posts/

JDK Tough Way - 6. Practical Guide and Underlying Principles of Tracking Java Heap OOM with JFR

·4214 words·20 mins
NeatGuyCoding
Author
NeatGuyCoding
Table of Contents

0. Important Notice
#

In the previous article, we analyzed Heap dump and error handling diagnostic evolution and best practices, where we mentioned that we do not recommend enabling the -XX:+HeapDumpOnOutOfMemoryError option, but instead use JFR to quickly locate Java heap OOM. This article will provide a detailed explanation of how to quickly locate Java heap OOM using JFR with practical methods and underlying principles.

First, prepare the following tools or environment:

1. What Scenarios Do We Need to Simulate?
#

Here we simulate scenarios where Java heap OOM would trigger Heap Dump when the -XX:+HeapDumpOnOutOfMemoryError option is enabled:

  • java.lang.OutOfMemoryError: Java heap space: Java heap space exhausted (Java heap OOM) and java.lang.OutOfMemoryError: GC overhead limit exceeded: GC overhead limit exceeded (Java heap-related OOM)
    • Scenario 1: A request has a bug that causes a full table scan, overwhelming the Java heap memory. An OutOfMemoryError is thrown, but this is an exceptional situation that may not output stack trace logs, making it difficult to find this request among the vast number of requests.
    • Scenario 2: As your system matures, users’ cumulative order volumes increase, and there are more and more users with large historical order volumes. Previous code had a bug where the user order list actually pulls all orders for each user for in-memory pagination. When two users with large historical order volumes query simultaneously, it may throw OutOfMemoryError, and even if it doesn’t, frequent GC will affect performance.
    • Scenario 3: A request triggers allocation of a small object into a cache-like location, but this small object is never reclaimed, accumulating over time and causing FullGC to become more frequent, eventually leading to OutOfMemoryError.
  • java.lang.OutOfMemoryError: Metaspace / Compressed class space: Metaspace exhausted (will trigger Java heap dump because loaded classes have Class objects on the Java heap)
    • Scenario 4: Too many dynamically generated classes (such as CGLIB, Javassist, etc.) eventually lead to metaspace exhaustion: This will have more direct exception display and won’t be buried. So JFR is not needed for localization.
    • Scenario 5: Insufficient metaspace itself, dynamic class loading fails: This also won’t be buried, JFR is not needed for localization.
  • java.lang.OutOfMemoryError: Requested array size exceeds VM limit: Array size exceeds VM limit (will trigger Java heap dump because it usually indicates an oversized collection)
    • Scenario 6: A request has a bug that causes allocation of an oversized array, eventually leading to array size limit exceeded: This also won’t be buried, JFR is not needed for localization.

We will analyze scenarios 1-3 that require JFR for localization.

2. Scenario 1: A Request Has a Bug That Causes a Full Table Scan, Overwhelming the Java Heap Memory
#

After git cloning the code repository, we use maven to build. Open the pom.xml in the project root directory and ensure the Java language level is version 21 or above:

img_15.png

Modify application.properties to enable virtual threads (spring.threads.virtual.enabled=true)

img_3.png

Add a query that returns a large amount of data to simulate the problem:

public interface OwnerRepository extends JpaRepository<Owner, Integer> {
    // ... omit other existing code ...
	
	@Query(
		value = "SELECT REPEAT('A', :count) as repeated_string",
		nativeQuery = true
	)
	String repeatString(@Param("count") int count);
}

Add a Controller method to call this query:

@Controller
class CrashController {

    // ... omit other existing code ...

	@Autowired
	private OwnerRepository repository;

	@ResponseBody
	@PostMapping("/large-oom-query")
	public String triggerLargeOomQuery() {
		return repository.repeatString(1024 * 1024 * 1024);
	}
}

Then find the jmeter script located at src/test/jmeter/petclinic.jmx:

img.png

We change the thread count to 50 and set the loop count to infinite to simulate a continuously accessed environment, similar to an online environment that is constantly being accessed:

img_1.png

Add a jfc configuration file to enable some JFR events, named test-oom.jfc:

<?xml version="1.0" encoding="UTF-8"?>
<configuration version="2.0">
	<event name="jdk.AllocationRequiringGC">
		<setting name="enabled" control="gc-enabled-high">true</setting>
		<setting name="stackTrace">true</setting>
	</event>
	<event name="jdk.ZAllocationStall">
		<setting name="enabled">true</setting>
		<setting name="stackTrace">true</setting>
		<setting name="threshold">0 ms</setting>
	</event>
</configuration>

Start the Spring Boot application with JVM parameters:

-XX:+UseG1GC
-Xmx256m 
-XX:StartFlightRecording=disk=true,maxsize=5000m,maxage=2d,settings=./test-oom.jfc 
-XX:FlightRecorderOptions=maxchunksize=128m,repository=./,stackdepth=256
  • -XX:+UseG1GC: Use G1 garbage collector, I’m currently using JDK 25, where the default garbage collector is G1
  • -Xmx256m: Limit maximum heap memory to 256MB to easily trigger Java heap OOM
  • -XX:StartFlightRecording=disk=true,maxsize=5000m,maxage=2d,settings=./test-oom.jfc: Enable JFR, set maximum size to 5000MB, maximum retention time to 2 days, use our custom configuration file test-oom.jfc, and start JFR event temporary file chunks
  • -XX:FlightRecorderOptions=maxchunksize=128m,repository=./,stackdepth=256,preserve-repository=true: Set JFR chunk maximum size to 128MB, JFR event temporary file storage location to current directory, stack depth to 256

Start the jmeter script to simulate continuous requests.

Then, send a request: POST http://localhost:8080/large-oom-query to trigger Java heap OOM. You can see a large number of exception outputs in the logs, and OutOfMemoryError: Java heap space has been buried:

2025-11-09T10:47:01.322+08:00  INFO 6708 --- [at-handler-2731] [                                                 ] o.h.e.internal.DefaultLoadEventListener  : HHH000327: Error performing load command

org.hibernate.exception.JDBCConnectionException: JDBC exception executing SQL [The database has been closed [90098-232]] [select o1_0.id,o1_0.address,o1_0.city,o1_0.first_name,o1_0.last_name,o1_0.telephone,p1_0.owner_id,p1_0.id,p1_0.birth_date,p1_0.name,t1_0.id,t1_0.name from owners o1_0 left join pets p1_0 on o1_0.id=p1_0.owner_id left join types t1_0 on t1_0.id=p1_0.type_id where o1_0.id=? order by p1_0.name]
	at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:49) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:34) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:115) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.sql.results.jdbc.internal.DeferredResultSetAccess.executeQuery(DeferredResultSetAccess.java:304) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.sql.results.jdbc.internal.DeferredResultSetAccess.getResultSet(DeferredResultSetAccess.java:200) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.sql.results.jdbc.internal.JdbcValuesResultSetImpl.<init>(JdbcValuesResultSetImpl.java:72) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.sql.exec.internal.JdbcSelectExecutorStandardImpl.resolveJdbcValues(JdbcSelectExecutorStandardImpl.java:372) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.sql.exec.internal.JdbcSelectExecutorStandardImpl.resolveJdbcValuesSource(JdbcSelectExecutorStandardImpl.java:332) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.sql.exec.internal.JdbcSelectExecutorStandardImpl.doExecuteQuery(JdbcSelectExecutorStandardImpl.java:135) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.sql.exec.internal.JdbcSelectExecutorStandardImpl.executeQuery(JdbcSelectExecutorStandardImpl.java:100) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.sql.exec.spi.JdbcSelectExecutor.executeQuery(JdbcSelectExecutor.java:64) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.sql.exec.spi.JdbcSelectExecutor.list(JdbcSelectExecutor.java:138) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.loader.ast.internal.SingleIdLoadPlan.load(SingleIdLoadPlan.java:143) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.loader.ast.internal.SingleIdLoadPlan.load(SingleIdLoadPlan.java:115) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.loader.ast.internal.SingleIdEntityLoaderStandardImpl.load(SingleIdEntityLoaderStandardImpl.java:66) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.persister.entity.AbstractEntityPersister.doLoad(AbstractEntityPersister.java:3532) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3521) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.event.internal.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:596) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.event.internal.DefaultLoadEventListener.loadFromCacheOrDatasource(DefaultLoadEventListener.java:570) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:554) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:538) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:204) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.event.internal.DefaultLoadEventListener.loadWithRegularProxy(DefaultLoadEventListener.java:284) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:236) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.event.internal.DefaultLoadEventListener.doOnLoad(DefaultLoadEventListener.java:109) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:68) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:151) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.internal.SessionImpl.fireLoadNoChecks(SessionImpl.java:1287) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1274) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.internal.SessionImpl.load(SessionImpl.java:1256) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.loader.internal.IdentifierLoadAccessImpl.doLoad(IdentifierLoadAccessImpl.java:181) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.loader.internal.IdentifierLoadAccessImpl.lambda$load$1(IdentifierLoadAccessImpl.java:167) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.loader.internal.IdentifierLoadAccessImpl.perform(IdentifierLoadAccessImpl.java:134) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.loader.internal.IdentifierLoadAccessImpl.load(IdentifierLoadAccessImpl.java:167) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.internal.SessionImpl.find(SessionImpl.java:2403) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at org.hibernate.internal.SessionImpl.find(SessionImpl.java:2377) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:565) ~[na:na]
	at org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:317) ~[spring-orm-7.0.0-M9.jar:7.0.0-M9]
	at jdk.proxy2/jdk.proxy2.$Proxy166.find(Unknown Source) ~[na:na]
	at org.springframework.data.jpa.repository.support.SimpleJpaRepository.findById(SimpleJpaRepository.java:344) ~[spring-data-jpa-4.0.0-M6.jar:4.0.0-M6]
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:565) ~[na:na]
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359) ~[spring-aop-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.data.repository.core.support.RepositoryMethodInvoker$RepositoryFragmentMethodInvoker.lambda$new$0(RepositoryMethodInvoker.java:278) ~[spring-data-commons-4.0.0-M6.jar:4.0.0-M6]
	at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:169) ~[spring-data-commons-4.0.0-M6.jar:4.0.0-M6]
	at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:158) ~[spring-data-commons-4.0.0-M6.jar:4.0.0-M6]
	at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:545) ~[spring-data-commons-4.0.0-M6.jar:4.0.0-M6]
	at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:290) ~[spring-data-commons-4.0.0-M6.jar:4.0.0-M6]
	at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:708) ~[spring-data-commons-4.0.0-M6.jar:4.0.0-M6]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:171) ~[spring-data-commons-4.0.0-M6.jar:4.0.0-M6]
	at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:146) ~[spring-data-commons-4.0.0-M6.jar:4.0.0-M6]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:69) ~[spring-data-commons-4.0.0-M6.jar:4.0.0-M6]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:369) ~[spring-tx-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118) ~[spring-tx-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:135) ~[spring-tx-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:167) ~[spring-data-jpa-4.0.0-M6.jar:4.0.0-M6]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.data.util.NullnessMethodInvocationValidator.invoke(NullnessMethodInvocationValidator.java:99) ~[spring-data-commons-4.0.0-M6.jar:4.0.0-M6]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:222) ~[spring-aop-7.0.0-M9.jar:7.0.0-M9]
	at jdk.proxy2/jdk.proxy2.$Proxy176.findById(Unknown Source) ~[na:na]
	at org.springframework.samples.petclinic.owner.PetController.findOwner(PetController.java:69) ~[classes/:na]
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:565) ~[na:na]
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:256) ~[spring-web-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:188) ~[spring-web-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.web.method.annotation.ModelFactory.invokeModelAttributeMethods(ModelFactory.java:142) ~[spring-webmvc-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.web.method.annotation.ModelFactory.initModel(ModelFactory.java:111) ~[spring-webmvc-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:918) ~[spring-webmvc-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:852) ~[spring-webmvc-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:86) ~[spring-webmvc-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) ~[spring-webmvc-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:866) ~[spring-webmvc-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1003) ~[spring-webmvc-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:892) ~[spring-webmvc-7.0.0-M9.jar:7.0.0-M9]
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:622) ~[tomcat-embed-core-11.0.11.jar:6.1]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:874) ~[spring-webmvc-7.0.0-M9.jar:7.0.0-M9]
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:710) ~[tomcat-embed-core-11.0.11.jar:6.1]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:130) ~[tomcat-embed-core-11.0.11.jar:11.0.11]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-11.0.11.jar:11.0.11]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:109) ~[tomcat-embed-core-11.0.11.jar:11.0.11]
	at org.springframework.web.servlet.resource.ResourceUrlEncodingFilter.doFilter(ResourceUrlEncodingFilter.java:66) ~[spring-webmvc-7.0.0-M9.jar:7.0.0-M9]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:109) ~[tomcat-embed-core-11.0.11.jar:11.0.11]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-7.0.0-M9.jar:7.0.0-M9]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:109) ~[tomcat-embed-core-11.0.11.jar:11.0.11]
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-webmvc-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-7.0.0-M9.jar:7.0.0-M9]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:109) ~[tomcat-embed-core-11.0.11.jar:11.0.11]
	at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:110) ~[spring-webmvc-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-7.0.0-M9.jar:7.0.0-M9]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:109) ~[tomcat-embed-core-11.0.11.jar:11.0.11]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:199) ~[spring-webmvc-7.0.0-M9.jar:7.0.0-M9]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-7.0.0-M9.jar:7.0.0-M9]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:109) ~[tomcat-embed-core-11.0.11.jar:11.0.11]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) ~[tomcat-embed-core-11.0.11.jar:11.0.11]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:79) ~[tomcat-embed-core-11.0.11.jar:11.0.11]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483) ~[tomcat-embed-core-11.0.11.jar:11.0.11]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:116) ~[tomcat-embed-core-11.0.11.jar:11.0.11]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) ~[tomcat-embed-core-11.0.11.jar:11.0.11]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-11.0.11.jar:11.0.11]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-11.0.11.jar:11.0.11]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:396) ~[tomcat-embed-core-11.0.11.jar:11.0.11]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) ~[tomcat-embed-core-11.0.11.jar:11.0.11]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:903) ~[tomcat-embed-core-11.0.11.jar:11.0.11]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1780) ~[tomcat-embed-core-11.0.11.jar:11.0.11]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) ~[tomcat-embed-core-11.0.11.jar:11.0.11]
	at java.base/java.lang.VirtualThread.run(VirtualThread.java:456) ~[na:na]
Caused by: org.h2.jdbc.JdbcSQLNonTransientConnectionException: The database has been closed [90098-232]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:690) ~[h2-2.3.232.jar:2.3.232]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:489) ~[h2-2.3.232.jar:2.3.232]
	at org.h2.message.DbException.get(DbException.java:212) ~[h2-2.3.232.jar:2.3.232]
	at org.h2.engine.SessionLocal.getTransaction(SessionLocal.java:1616) ~[h2-2.3.232.jar:2.3.232]
	at org.h2.engine.SessionLocal.startStatementWithinTransaction(SessionLocal.java:1637) ~[h2-2.3.232.jar:2.3.232]
	at org.h2.command.Command.executeQuery(Command.java:190) ~[h2-2.3.232.jar:2.3.232]
	at org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:130) ~[h2-2.3.232.jar:2.3.232]
	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52) ~[HikariCP-7.0.2.jar:na]
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java) ~[HikariCP-7.0.2.jar:na]
	at org.hibernate.sql.results.jdbc.internal.DeferredResultSetAccess.executeQuery(DeferredResultSetAccess.java:285) ~[hibernate-core-7.1.1.Final.jar:7.1.1.Final]
	... 110 common frames omitted

...... Large amount of logs omitted

Do not stop the process yet. Copy out the JFR event local chunk temporary files. The directory structure is similar to:

 - 2025_11_09_10_54_11_7246
   ｜ 2025_11_09_10_54_11.jfr

Open this jfr file using JMC, view the Allocation Requiring GC event, sorted by size in descending order. You can find the largest object allocation request (this event is generated before allocation, regardless of whether the allocation succeeds):

img_4.png

Let’s try switching to ZGC:

-XX:+UseZGC
-Xmx256m
-XX:StartFlightRecording=disk=true,maxsize=5000m,maxage=2d,settings=./test-oom.jfc
-XX:FlightRecorderOptions=maxchunksize=128m,repository=./,stackdepth=256,preserve-repository=true

Since ZGC doesn’t have Allocation Requiring GC events, we can view the ZAllocationStall event (this event mechanism is similar, generated before allocation regardless of whether allocation succeeds), sorted by size in descending order. Similarly, we can find the largest object allocation request:

img_5.png

It can be seen that for scenario 1, we can quickly locate the request that triggered Java heap OOM through JFR, and then locate the code position for fixing, without relying on Heap Dump files generated by the -XX:+HeapDumpOnOutOfMemoryError option for offline analysis.

3. Scenario 2: Users’ Cumulative Order Volumes Increase as Your System Matures, More and More Users Have Large Historical Order Volumes
#

Add a new Controller method to simulate user queries with large historical order volumes:

@ResponseBody
@GetMapping("/large-user-query")
public String largeUserQuery() {
    return repository.repeatString(32 * 1024 * 1024);
}

Then modify the jmeter script to add a new request simulating many users accessing this interface:

img_6.png

Start the Spring Boot application with the same JVM parameters as above:

-XX:+UseG1GC
-Xmx256m
-XX:StartFlightRecording=disk=true,maxsize=5000m,maxage=2d,settings=./test-oom.jfc
-XX:FlightRecorderOptions=maxchunksize=128m,repository=./,stackdepth=256,preserve-repository=true

Start the jmeter script to simulate continuous requests. After a while, you will see Java heap OOM, and you can see a large number of exception outputs in the logs. OutOfMemoryError: Java heap space has been buried.

Similarly, copy out the JFR event local chunk temporary files, open this jfr file using JMC, and view the Allocation Requiring GC event sorted by size in descending order. For this scenario, you can find many Allocation Requiring GC events with object sizes around 32MB, corresponding to our simulated request:

img_7.png

Let’s try switching to ZGC again:

-XX:+UseZGC
-Xmx256m
-XX:StartFlightRecording=disk=true,maxsize=5000m,maxage=2,settings=./test-oom.jfc
-XX:FlightRecorderOptions=maxchunksize=128m,repository=./,stackdepth=256,preserve-repository=true

Since ZGC doesn’t have Allocation Requiring GC events, we can view the ZAllocationStall event sorted by size in descending order. Similarly, we can find many ZAllocationStall events with object sizes around 32MB, corresponding to our simulated request:

img_8.png

It can be seen that for scenario 2, we can also quickly locate the request that triggered Java heap OOM through JFR, and then locate the code position for fixing, without relying on Heap Dump files generated by the -XX:+HeapDumpOnOutOfMemoryError option for offline analysis.

4. Scenario 3: A Request Triggers Allocation of a Small Object into a Cache-Like Location, but This Small Object Is Never Reclaimed
#

This cache is generally a container, such as Map, List, Queue, etc. The built-in implementations in JDK all involve arrays at the bottom layer and are dynamically expandable. Moreover, when the container is already relatively large, this expansion generally expands to 1.5 to 2 times the original size each time.

4.1. List Container Expansion Implementation
#

4.1.1. ArrayList
#

JDK 11: jdk-jdk-11-28/src/java.base/share/classes/java/util/ArrayList.java

    private int newCapacity(int minCapacity) {
        // overflow-conscious code
        int oldCapacity = elementData.length;
        int newCapacity = oldCapacity + (oldCapacity >> 1);
        if (newCapacity - minCapacity <= 0) {
            if (elementData == DEFAULTCAPACITY_EMPTY_ELEMENTDATA)
                return Math.max(DEFAULT_CAPACITY, minCapacity);
            if (minCapacity < 0) // overflow
                throw new OutOfMemoryError();
            return minCapacity;
        }
        return (newCapacity - MAX_ARRAY_SIZE <= 0)
            ? newCapacity
            : hugeCapacity(minCapacity);
    }

JDK 17+: jdk-jdk-17-35/src/java.base/share/classes/java/util/ArrayList.java

    private Object[] grow(int minCapacity) {
        int oldCapacity = elementData.length;
        if (oldCapacity > 0 || elementData != DEFAULTCAPACITY_EMPTY_ELEMENTDATA) {
            int newCapacity = ArraysSupport.newLength(oldCapacity,
                    minCapacity - oldCapacity, /* minimum growth */
                    oldCapacity >> 1           /* preferred growth */);
            return elementData = Arrays.copyOf(elementData, newCapacity);
        } else {
            return elementData = new Object[Math.max(DEFAULT_CAPACITY, minCapacity)];
        }
    }

Expansion Mechanism:

  • Expansion Factor: 1.5x (oldCapacity + (oldCapacity >> 1))
  • Default Initial Capacity: 10
  • Version Differences:
    • JDK 11: Direct capacity calculation
    • JDK 17+: Uses ArraysSupport.newLength for unified processing, same logic but more unified code

4.1.2. Vector
#

JDK 11: jdk-jdk-11-28/src/java.base/share/classes/java/util/Vector.java

    private int newCapacity(int minCapacity) {
        // overflow-conscious code
        int oldCapacity = elementData.length;
        int newCapacity = oldCapacity + ((capacityIncrement > 0) ?
                                         capacityIncrement : oldCapacity);
        if (newCapacity - minCapacity <= 0) {
            if (minCapacity < 0) // overflow
                throw new OutOfMemoryError();
            return minCapacity;
        }
        return (newCapacity - MAX_ARRAY_SIZE <= 0)
            ? newCapacity
            : hugeCapacity(minCapacity);
    }

Expansion Mechanism:

  • Expansion Strategy:
    • If capacityIncrement > 0: Expand by increment (oldCapacity + capacityIncrement)
    • If capacityIncrement <= 0: 2x expansion (oldCapacity + oldCapacity)
  • Default Initial Capacity: 10
  • Default capacityIncrement: 0 (therefore default is 2x expansion)

4.1.3. ArrayDeque
#

JDK 11: jdk-jdk-11-28/src/java.base/share/classes/java/util/ArrayDeque.java

    private void grow(int needed) {
        // overflow-conscious code
        final int oldCapacity = elements.length;
        int newCapacity;
        // Double capacity if small; else grow by 50%
        int jump = (oldCapacity < 64) ? (oldCapacity + 2) : (oldCapacity >> 1);
        if (jump < needed
            || (newCapacity = (oldCapacity + jump)) - MAX_ARRAY_SIZE > 0)
            newCapacity = newCapacity(needed, jump);
        final Object[] es = elements = Arrays.copyOf(elements, newCapacity);
        // Exceptionally, here tail == head needs to be disambiguated
        if (tail < head || (tail == head && es[head] != null)) {
            // wrap around; slide first leg forward to end of array
            int newSpace = newCapacity - oldCapacity;
            System.arraycopy(es, head,
                             es, head + newSpace,
                             oldCapacity - head);
            for (int i = head, to = (head += newSpace); i < to; i++)
                es[i] = null;
        }
    }

Expansion Mechanism:

  • Expansion Strategy:
    • Capacity < 64: Add 2 (oldCapacity + 2)
    • Capacity >= 64: 1.5x expansion (oldCapacity + (oldCapacity >> 1))
  • Default Initial Capacity: 16

4.1.4. CopyOnWriteArrayList
#

JDK 11-25: jdk-jdk-11-28/src/java.base/share/classes/java/util/concurrent/CopyOnWriteArrayList.java

    public boolean add(E e) {
        synchronized (lock) {
            Object[] es = getArray();
            int len = es.length;
            es = Arrays.copyOf(es, len + 1);
            es[len] = e;
            setArray(es);
            return true;
        }
    }

Expansion Mechanism:

  • Expansion Strategy: Exact expansion, only increases by 1 each time (len + 1)
  • Characteristics: Copy-On-Write, each modification operation creates a new array
  • Initial Capacity: 0 (empty array)
  • Memory Impact: Since each write operation copies the entire array, frequent writes will cause significant memory allocation and GC pressure

4.2. Queue Container Expansion Implementation
#

4.2.1. PriorityQueue
#

JDK 11: jdk-jdk-11-28/src/java.base/share/classes/java/util/PriorityQueue.java

    private void grow(int minCapacity) {
        int oldCapacity = queue.length;
        // Double size if small; else grow by 50%
        int newCapacity = oldCapacity + ((oldCapacity < 64) ?
                                         (oldCapacity + 2) :
                                         (oldCapacity >> 1));
        // overflow-conscious code
        if (newCapacity - MAX_ARRAY_SIZE > 0)
            newCapacity = hugeCapacity(minCapacity);
        queue = Arrays.copyOf(queue, newCapacity);
    }

JDK 17+: jdk-jdk-17-35/src/java.base/share/classes/java/util/PriorityQueue.java

    private void grow(int minCapacity) {
        int oldCapacity = queue.length;
        // Double size if small; else grow by 50%
        int newCapacity = ArraysSupport.newLength(oldCapacity,
                minCapacity - oldCapacity, /* minimum growth */
                oldCapacity < 64 ? oldCapacity + 2 : oldCapacity >> 1
                                           /* preferred growth */);
        queue = Arrays.copyOf(queue, newCapacity);
    }

Expansion Mechanism:

  • Expansion Strategy:
    • Capacity < 64: Add oldCapacity + 2 (close to 2x)
    • Capacity >= 64: 1.5x expansion (oldCapacity + (oldCapacity >> 1))
  • Default Initial Capacity: 11
  • Version Differences:
    • JDK 11: Direct capacity calculation
    • JDK 17+: Uses ArraysSupport.newLength for unified processing

4.2.2. PriorityBlockingQueue
#

JDK 11: jdk-jdk-11-28/src/java.base/share/classes/java/util/concurrent/PriorityBlockingQueue.java

    private void tryGrow(Object[] array, int oldCap) {
        lock.unlock(); // must release and then re-acquire main lock
        Object[] newArray = null;
        if (allocationSpinLock == 0 &&
            ALLOCATIONSPINLOCK.compareAndSet(this, 0, 1)) {
            try {
                int newCap = oldCap + ((oldCap < 64) ?
                                       (oldCap + 2) : // grow faster if small
                                       (oldCap >> 1));
                if (newCap - MAX_ARRAY_SIZE > 0) {    // possible overflow
                    int minCap = oldCap + 1;
                    if (minCap < 0 || minCap > MAX_ARRAY_SIZE)
                        throw new OutOfMemoryError();
                    newCap = MAX_ARRAY_SIZE;
                }
                if (newCap > oldCap && queue == array)
                    newArray = new Object[newCap];
            } finally {
                allocationSpinLock = 0;
            }
        }

JDK 17+: jdk-jdk-17-35/src/java.base/share/classes/java/util/concurrent/PriorityBlockingQueue.java

    private void tryGrow(Object[] array, int oldCap) {
        lock.unlock(); // must release and then re-acquire main lock
        Object[] newArray = null;
        if (allocationSpinLock == 0 &&
            ALLOCATIONSPINLOCK.compareAndSet(this, 0, 1)) {
            try {
                int growth = (oldCap < 64)
                    ? (oldCap + 2) // grow faster if small
                    : (oldCap >> 1);
                int newCap = ArraysSupport.newLength(oldCap, 1, growth);
                if (queue == array)
                    newArray = new Object[newCap];
            } finally {
                allocationSpinLock = 0;
            }
        }
        if (newArray == null) // back off if another thread is allocating
            Thread.yield();
        lock.lock();
        if (newArray != null && queue == array) {
            queue = newArray;
            System.arraycopy(array, 0, newArray, 0, oldCap);
        }
    }

Expansion Mechanism:

  • Expansion Strategy: Same as PriorityQueue
    • Capacity < 64: Add oldCapacity + 2
    • Capacity >= 64: 1.5x expansion
  • Default Initial Capacity: 11
  • Thread Safety: Uses CAS and lock mechanisms to ensure concurrency safety
  • Version Differences:
    • JDK 11: Direct capacity calculation
    • JDK 17+: Uses ArraysSupport.newLength for unified processing

4.2.3. ArrayBlockingQueue
#

JDK 11-25: jdk-jdk-11-28/src/java.base/share/classes/java/util/concurrent/ArrayBlockingQueue.java

Expansion Mechanism:

  • Expansion Strategy: Fixed capacity, does not support expansion
  • Characteristics: Capacity specified at construction, capacity is immutable afterward
  • Purpose: Bounded queue for producer-consumer patterns, preventing unlimited queue growth

4.2.4. LinkedBlockingQueue
#

JDK 11-25: jdk-jdk-11-28/src/java.base/share/classes/java/util/concurrent/LinkedBlockingQueue.java

Expansion Mechanism:

  • Expansion Strategy: Linked list implementation, does not involve array expansion
  • Characteristics: Each element is an independent Node object, dynamically created and destroyed
  • Capacity Limit: Can specify maximum capacity, default Integer.MAX_VALUE (unbounded)
  • Memory Impact: Each node has additional object header overhead, memory utilization is lower than array implementation

4.2.5. ConcurrentLinkedQueue
#

Expansion Mechanism:

  • Expansion Strategy: Linked list implementation, does not involve array expansion
  • Characteristics: Lock-free concurrent queue, uses CAS operations
  • Capacity Limit: Unbounded queue

4.3. Map Container Expansion Implementation
#

4.3.1. HashMap
#

JDK 11-25: jdk-jdk-11-28/src/java.base/share/classes/java/util/HashMap.java

    final Node<K,V>[] resize() {
        Node<K,V>[] oldTab = table;
        int oldCap = (oldTab == null) ? 0 : oldTab.length;
        int oldThr = threshold;
        int newCap, newThr = 0;
        if (oldCap > 0) {
            if (oldCap >= MAXIMUM_CAPACITY) {
                threshold = Integer.MAX_VALUE;
                return oldTab;
            }
            else if ((newCap = oldCap << 1) < MAXIMUM_CAPACITY &&
                     oldCap >= DEFAULT_INITIAL_CAPACITY)
                newThr = oldThr << 1; // double threshold
        }
        else if (oldThr > 0) // initial capacity was placed in threshold
            newCap = oldThr;
        else {               // zero initial threshold signifies using defaults
            newCap = DEFAULT_INITIAL_CAPACITY;
            newThr = (int)(DEFAULT_LOAD_FACTOR * DEFAULT_INITIAL_CAPACITY);
        }
        if (newThr == 0) {
            float ft = (float)newCap * loadFactor;
            newThr = (newCap < MAXIMUM_CAPACITY && ft < (float)MAXIMUM_CAPACITY ?
                      (int)ft : Integer.MAX_VALUE);
        }
        threshold = newThr;
        @SuppressWarnings({"rawtypes","unchecked"})
        Node<K,V>[] newTab = (Node<K,V>[])new Node[newCap];
        table = newTab;
        if (oldTab != null) {
            for (int j = 0; j < oldCap; ++j) {
                Node<K,V> e;
                if ((e = oldTab[j]) != null) {
                    oldTab[j] = null;
                    if (e.next == null)
                        newTab[e.hash & (newCap - 1)] = e;
                    else if (e instanceof TreeNode)
                        ((TreeNode<K,V>)e).split(this, newTab, j, oldCap);
                    else { // preserve order
                        Node<K,V> loHead = null, loTail = null;
                        Node<K,V> hiHead = null, hiTail = null;
                        Node<K,V> next;
                        do {
                            next = e.next;
                            if ((e.hash & oldCap) == 0) {
                                if (loTail == null)
                                    loHead = e;
                                else
                                    loTail.next = e;
                                loTail = e;
                            }
                            else {
                                if (hiTail == null)
                                    hiHead = e;
                                else
                                    hiTail.next = e;
                                hiTail = e;
                            }
                        } while ((e = next) != null);
                        if (loTail != null) {
                            loTail.next = null;
                            newTab[j] = loHead;
                        }
                        if (hiTail != null) {
                            hiTail.next = null;
                            newTab[j + oldCap] = hiHead;
                        }
                    }
                }
            }
        }
        return newTab;
    }

Expansion Mechanism:

  • Expansion Factor: 2x (newCap = oldCap << 1)
  • Default Initial Capacity: 16
  • Load Factor: 0.75
  • Expansion Trigger Condition: size > threshold (threshold = capacity * loadFactor)

4.3.2. LinkedHashMap
#

LinkedHashMap inherits from HashMap, and its expansion mechanism is the same as HashMap, both are 2x expansion.

4.3.3. Hashtable
#

Hashtable’s expansion mechanism is similar to HashMap, also 2x expansion, but it is thread-safe.

4.3.4. ConcurrentHashMap
#

ConcurrentHashMap’s expansion mechanism is also 2x expansion, but the implementation is more complex and supports multi-threaded concurrent expansion.

4.4. Version Comparison Summary
#

Container TypeJDK 11JDK 17+Expansion StrategyDefault Initial CapacityNotes
List Containers
ArrayListDirect calculationArraysSupport.newLength1.5x10
VectorDirect calculationArraysSupport.newLength2x (default) or by increment10
ArrayDequeDirect calculationDirect calculation< 64: +2, >= 64: 1.5x16
CopyOnWriteArrayListExact expansionExact expansion+1 each time0Copy-On-Write, each write operation creates new array
LinkedListLinked list implementationLinked list implementationNo array expansion-Based on linked list, dynamic node creation
Queue Containers
PriorityQueueDirect calculationArraysSupport.newLength< 64: +oldCap+2, >= 64: 1.5x11
PriorityBlockingQueueDirect calculationArraysSupport.newLength< 64: +oldCap+2, >= 64: 1.5x11Thread-safe version
ArrayBlockingQueueFixed capacityFixed capacityNo expansionConstructor specifiedBounded queue
LinkedBlockingQueueLinked list implementationLinked list implementationNo array expansionInteger.MAX_VALUEBased on linked list
ConcurrentLinkedQueueLinked list implementationLinked list implementationNo array expansionUnboundedLock-free concurrent queue
Map Containers
HashMapDirect calculationDirect calculation2x16
LinkedHashMapInherits HashMapInherits HashMap2x16
HashtableDirect calculationDirect calculation2x11
ConcurrentHashMapDirect calculationDirect calculation2x16Supports concurrent expansion

Version Evolution Characteristics:

  • JDK 17+: ArrayList, Vector, PriorityQueue, PriorityBlockingQueue use ArraysSupport.newLength for unified expansion calculation, improving code reusability and maintainability
  • Expansion Factors:
    • Most containers use 2x expansion (HashMap, Hashtable, ConcurrentHashMap, etc.)
    • ArrayList and ArrayDeque (large capacity), PriorityQueue (large capacity) use 1.5x expansion
    • CopyOnWriteArrayList uses exact expansion (+1 each time), but each write operation copies the entire array
  • Special Containers:
    • ArrayBlockingQueue: Fixed capacity, does not support expansion
    • LinkedList, LinkedBlockingQueue, ConcurrentLinkedQueue: Based on linked lists, no array expansion involved
  • Expansion Strategy: Multiplicative expansion can reduce expansion frequency but leads to memory waste, especially in memory leak scenarios

Most commonly used cache containers have array-based dynamic expansion mechanisms at the bottom layer. In memory leak scenarios, these containers will continuously expand, and there will definitely be relatively large array expansion requests recorded in JFR events, helping us quickly locate problems.

4.5. Locating Container Expansion-Triggered OOM Through JFR
#

Continue using the Spring Boot application above, add a new Controller method to simulate a cache container continuously expanding:

private ConcurrentHashMap cache = new ConcurrentHashMap();

@ResponseBody
@GetMapping("/small-cumulative-task")
public String smallCumulativeTask() {
   // Simulate a request that puts many small objects into the cache
   for (int i = 0; i < 100_000; i++) {
      cache.put(new Object(), new Object());
   }
   return "OK";
}

Then modify the jmeter script to add a new request simulating many users accessing this interface:

img_9.png

This time, we need to modify the JFR event collection configuration to add the Allocation Outside TLAB event (since this is a small memory continuous leak with many other types of concurrent requests, the array expansion request may not trigger Allocation Requiring GC or ZAllocationStall events, but the expansion process will definitely be reflected in Allocation Outside TLAB):

<?xml version="1.0" encoding="UTF-8"?>
<configuration version="2.0">
	<event name="jdk.ObjectAllocationOutsideTLAB">
		<setting name="enabled">true</setting>
		<setting name="stackTrace">true</setting>
	</event>
	<event name="jdk.AllocationRequiringGC">
		<setting name="enabled" control="gc-enabled-high">true</setting>
		<setting name="stackTrace">true</setting>
	</event>
	<event name="jdk.ZAllocationStall">
		<setting name="enabled">true</setting>
		<setting name="stackTrace">true</setting>
		<setting name="threshold">0 ms</setting>
	</event>
</configuration>

Start the Spring Boot application with the same JVM parameters as above:

-XX:+UseG1GC
-Xmx256m
-XX:StartFlightRecording=disk=true,maxsize=5000m,maxage=2d,settings=./test-oom.jfc
-XX:FlightRecorderOptions=maxchunksize=128m,repository=./,stackdepth=256,preserve-repository=true

Start the jmeter script to simulate continuous requests. After a while, you will see Java heap OOM, and there will be no stack trace information because this container expansion-triggered OutOfMemoryError is more severe and cannot be recovered. The first two scenarios generally trigger OutOfMemoryError exceptions with large allocation requests. After throwing OutOfMemoryError, the allocation is abandoned and memory is released. However, this scenario triggers OutOfMemoryError exceptions with many small objects continuously allocated, causing containers to continuously expand and eventually trigger OutOfMemoryError. In this case, memory cannot be released, and the console you see will be similar to the following:

img_10.png

Similarly, copy out the JFR event local chunk temporary files, open this jfr file using JMC, and first view the Allocation Requiring GC event sorted by size in descending order. You may not find obvious clues:

img_11.png

Then view the Allocation Outside TLAB event sorted by size in descending order. You can see obvious expansion request clues:

img_12.png

Next, let’s try switching to ZGC again:

-XX:+UseZGC
-Xmx256m
-XX:StartFlightRecording=disk=true,maxsize=5000m,maxage=2,settings=./test-oom.jfc
-XX:FlightRecorderOptions=maxchunksize=128m,repository=./,stackdepth=256,preserve-repository=true

Since ZGC doesn’t have Allocation Requiring GC events, we can view the ZAllocationStall event and find it’s also not obvious:

img_13.png

But viewing the Allocation Outside TLAB event sorted by size in descending order, we can also see obvious expansion request clues:

img_14.png

It can be seen that for scenario 3, we can also quickly locate the request that triggered Java heap OOM through JFR, and then locate the code position for fixing, without relying on Heap Dump files generated by the -XX:+HeapDumpOnOutOfMemoryError option for offline analysis.

5. Summary
#

Below are the collection timings for these three JFR events:

  • jdk.ObjectAllocationOutsideTLAB: Triggered when an object allocation request cannot be satisfied in TLAB (Thread-Local Allocation Buffer). TLAB is a memory area dedicated to each thread for fast allocation of small objects. When an object is too large or the current TLAB space is insufficient to accommodate the object, the JVM will try to allocate memory in other areas of the heap and trigger this event. For TLAB principles in JVM, you can refer to my other article: Complete TLAB Analysis. The difference from the other two events is that only successful allocations are recorded in this event. Through this event, you can clearly see the expansion trend of JDK internal containers, thereby locating the code position of memory leaks.
  • jdk.AllocationRequiringGC: Triggered when an object allocation request cannot be satisfied in the heap, usually because the heap memory is insufficient to accommodate the object. The JVM will try to trigger garbage collection (GC) to free memory, then retry allocation. If the allocation request still cannot be satisfied after GC, the JVM will throw an OutOfMemoryError exception. This event is recorded regardless of whether allocation succeeds. Through this event, you can most likely see one-time large memory request anomalies causing Java heap OOM in non-ZGC, non-ShenandoahGC GC situations.
  • jdk.ZAllocationStall: When using ZGC, if an object allocation request cannot be satisfied in the heap, this usually occurs when GC reclamation speed cannot meet allocation speed. This event is recorded regardless of whether allocation succeeds. Through this event, you can most likely see one-time large memory request anomalies causing Java heap OOM in ZGC GC situations.

Through these three events, we can cover most Java heap OOM scenarios, quickly locate the code position that triggered OOM, and then fix it, without relying on Heap Dump files generated by the -XX:+HeapDumpOnOutOfMemoryError option for offline analysis, greatly improving troubleshooting efficiency.

Related

Troubleshooting a SSL Performance Bottleneck Using JFR

Troubleshooting a SSL Performance Bottleneck Using JFR

·395 words·2 mins
In-depth analysis of a microservice performance issue with CPU spikes and database connection anomalies. Through JFR profiling, we discovered the root cause was Java SecureRandom blocking on /dev/random and provide solutions using /dev/urandom.
JDK Tough Way - 1. A Comprehensive Guide to Thread Local Allocation Buffers

JDK Tough Way - 1. A Comprehensive Guide to Thread Local Allocation Buffers

·9380 words·45 mins
A deep dive into JVM’s Thread Local Allocation Buffer (TLAB) mechanism, covering design principles, implementation details, performance optimization, and source code analysis. Learn how TLAB improves memory allocation efficiency in multi-threaded environments and master TLAB tuning techniques.