Spark Profiler — Debugging & Performance Analysis Guide

A practical reference for using spark to investigate performance issues on Minecraft servers, including CPU usage, memory allocations, lag spikes, garbage collection, and Netty thread bottlenecks.

This guide focuses on useful commands, when to use them, and how to interpret the results.

Quick Troubleshooting Guide

Not sure which profiling mode to use? Find your issue below and jump to the relevant section.

General recommendation: Start by identifying the symptoms you're experiencing rather than running every profiling mode. Make sure to collect profiles while the issue is occurring, and compare different profiling modes when necessary.

If you want a fairly comprehensive overview of your server's performance, I recommend collecting these three profiles:

1. Main-thread overview

   /spark profiler start --timeout 300

   Provides a general overview of what the main server thread is doing, helping you understand where tick time is being spent and spot potential bottlenecks.
2. CPU usage (all threads)

   /spark profiler start --thread * --ignore-sleeping --timeout 300

   Helps identify CPU-intensive operations across all threads, while excluding sleeping time.
3. Memory allocations (all threads)

   /spark profiler start --thread * --alloc --timeout 300

   Identifies methods responsible for memory allocations, helping detect excessive object creation and potential GC pressure.

Together, these profiles provide a solid starting point for investigating most server performance issues. For more specific problems, such as intermittent lag spikes, thread blocking, or Netty-related latency, refer to the dedicated sections below.

1. CPU Usage Profiling

To investigate CPU usage across all threads, excluding sleeping time:

/spark profiler start --thread * --ignore-sleeping --timeout 300

When to use:

• Investigating high CPU usage.
• Finding expensive methods or code paths.
• Identifying overloaded asynchronous threads.
• Understanding where CPU time is being spent.

Important: --ignore-sleeping

This option excludes sleeping/waiting time from the profiling results, helping you focus on active execution.

However, this can hide problematic behavior if your code sleeps, waits, or blocks in places where it shouldn't.

For this reason, I recommend also collecting a profile without --ignore-sleeping:

/spark profiler start --thread * --timeout 300

Comparing both reports can help distinguish expensive computations from excessive waiting, blocking, or synchronization issues.

Tip: Use Profiler Utils (userscript) or Spark Profile Tools (website) to analyze these reports more effectively.

2. Memory Allocation Profiling

To identify excessive memory allocations across all threads:

/spark profiler start --thread * --alloc --timeout 300

When to use:

• Investigating excessive garbage collection activity.
• Finding methods responsible for high allocation rates.
• Identifying unnecessary object creation.
• Diagnosing memory-related performance issues.

Notes:

• --alloc enables allocation profiling instead of CPU profiling.
• --thread * includes all threads.
• --timeout 300 automatically stops profiling after 300 seconds.

Important: High allocation rates do not necessarily indicate a memory leak. Short-lived objects can generate substantial GC pressure without increasing retained heap usage.

If you suspect a memory leak, investigate retained objects rather than relying exclusively on allocation profiling.

3. Main-Thread Lag Spikes

To investigate slow server ticks:

/spark profiler start --only-ticks-over 200 --timeout 300

This captures profiling samples associated with ticks exceeding 200 milliseconds.

When to use:

• Investigating intermittent TPS drops.
• Finding the cause of occasional server freezes.
• Diagnosing expensive operations occurring during specific ticks.
• Investigating lag spikes that don't appear in average performance metrics.

Adjusting the threshold

The --only-ticks-over value is expressed in milliseconds.

For particularly severe lag spikes, such as individual ticks taking more than one second:

/spark profiler start --only-ticks-over 1000 --timeout 300

Choose a threshold appropriate for the issue you're investigating.

A lower threshold captures more slow ticks, while a higher threshold focuses on particularly severe spikes.

Tip: If the server generally performs well but occasionally freezes, tick-based profiling is often more useful than collecting a general CPU profile.

4. Netty Thread Profiling (High Ping / Network Issues)

When players experience unusually high ping or network-related delays, profiling Netty threads can help identify expensive packet processing or overloaded network workers.

Separate Netty threads

/spark profiler start --ignore-sleeping --not-combined --thread Netty.* --regex --timeout 300

Recommended when:

• Investigating individual Netty threads.
• Checking whether a particular Netty worker is overloaded.
• Comparing workloads across network threads.
• Identifying uneven distribution of packet processing.

Why --not-combined is especially useful for Netty

Netty uses multiple event-loop threads, with each thread handling network operations for a subset of player connections.

In some cases, a single Netty thread may become overloaded while the others remain relatively idle. This can result in high ping or delayed packet processing only for players whose connections are handled by that particular thread, while other players experience no issues.

This is why --not-combined is particularly important when investigating Netty-related problems:

• Without --not-combined: Samples from multiple Netty threads are merged, potentially hiding an individual overloaded thread.
• With --not-combined: Each Netty thread can be analyzed separately, making it easier to identify uneven workloads and isolated bottlenecks.

Recommendation: When investigating high ping, especially if only some players are affected, start with --not-combined. A combined profile can still be useful afterward for understanding overall Netty resource usage.

Combined Netty threads

/spark profiler start --ignore-sleeping --thread Netty.* --regex --timeout 300

Recommended when:

• Investigating overall Netty CPU usage.
• Finding expensive operations across all Netty workers.
• Getting a general overview of network processing overhead.

Notes:

• --thread Netty.* selects matching Netty threads.
• --regex enables regular-expression matching for thread names.
• --ignore-sleeping excludes sleeping/waiting time.

Important: Netty resource attribution

Netty uses shared event loops and asynchronous packet processing, which can make it difficult to determine which plugin is responsible for resource consumption from a standard flame graph.

I recommend using Spark Profile Tools, which provides plugin-level resource attribution and can make Netty profiling results significantly easier to interpret.

Also remember that high ping doesn't necessarily indicate a Netty bottleneck. Main-thread stalls, network conditions, and other factors can contribute to latency.

5. Memory Usage and Potential Memory Leaks

"My server is using too much RAM!" — Is it a memory leak?

High RAM usage in a hosting panel such as Pterodactyl does not necessarily mean your server has a memory leak. The panel generally reports memory consumed by the server process or container, which is not the same as memory occupied by live Java objects. The JVM can retain committed heap memory for reuse and also consumes memory outside the heap.

To investigate, open a spark report → Memory tab and check:

• Heap — G1 Old Gen (at last GC): Usually a more reliable indicator of actual RAM usage, as it shows old-generation occupancy after the last relevant garbage collection.

Important: High Old Gen usage in a single snapshot is not proof of a leak. Watch whether post-GC Old Gen occupancy keeps increasing across multiple comparable GC cycles under similar workloads. If it does, investigate the objects being retained with the tools below. A high panel reading alone is not enough to diagnose a leak.

Allocation profiling shows where objects are created, but it doesn't necessarily tell you which objects remain in memory.

For retained-memory investigations, spark provides additional tools.

Heap summary

/spark heapsummary

Provides an overview of objects currently occupying heap memory.

Heap dump

/spark heapdump

Creates a heap dump for more detailed memory analysis.

When to use:

• Investigating steadily increasing heap usage.
• Looking for objects that remain reachable unexpectedly.
• Diagnosing potential memory leaks.
• Understanding what is retaining large amounts of memory.

Warning: Heap dumps can be expensive to generate, potentially causing pauses and requiring substantial disk space. Avoid generating them during peak production load unless necessary.

6. Profiling Duration and Best Practices

I generally recommend 300 seconds (5 minutes) as the default profiling duration.

However, shorter or longer sessions may be more appropriate depending on the problem.

General recommendations

• Profile while the issue is happening. A report collected under normal conditions may not reveal the actual bottleneck.
• Use representative workloads. Test with realistic player counts, entities, plugins, and server activity.
• Compare different profiling modes. In particular, compare reports with and without --ignore-sleeping.
• Avoid drawing conclusions from a single sample. Profilers are statistical, and short-lived operations may not always appear.
• Measure before and after optimizations. A change that looks beneficial in code may not produce a meaningful performance improvement.

Inclusive vs self time

• Inclusive time: Time spent in a method and its callees.
• Self time: Time spent directly in the method itself.

A method appearing at the top of a flame graph doesn't necessarily mean its own implementation is expensive. The actual bottleneck may be deeper in the call stack.

Allocation rate vs retained memory

• Allocation rate: How much memory is allocated over time.
• Retained memory: How much memory remains reachable and occupied.

High allocation rates can increase garbage collection pressure without necessarily indicating a memory leak.

Combined vs individual threads

Combining profiling results can make it easier to identify expensive code across a group of threads.

However, it can also hide situations where one thread is significantly more overloaded than others.

For Netty profiling, consider collecting both combined and non-combined reports.

7. Quick Command Reference

8. Additional Tools for Spark Analysis

Besides spark's built-in viewer, I recommend the following tools for analyzing profiling results.

Disclaimer: I developed both tools, so I'm obviously biased. Suggestions and feedback are welcome. AI-generated interpretations of spark reports, however, are not.

Profiler Utils — Userscript

A userscript that improves the visualization and analysis of spark profiling reports.

It helps make CPU usage, memory allocations, and thread activity easier to inspect and understand.

Spark Profile Tools — Web Analyzer

A fully client-side website for analyzing spark profiling reports.

It provides the same functionality as the userscript, plus additional analysis features:

Plugin resource attribution

Attributes resource usage to individual plugins, making it easier to identify which plugins are responsible for CPU consumption.

This is especially useful when investigating Netty threads, since shared event loops and asynchronous packet processing can make it difficult to identify the responsible plugin directly from a flame graph.

Sleeping time cleanup

Attempts to clean up sleeping/waiting time in all-thread (--thread *) spark reports collected with --ignore-sleeping.

This is not 100% accurate, but provides a useful approximation that can make the results easier to interpret.

Client-side processing

Analysis is performed directly in your browser, without requiring server-side processing.

For more advanced investigations, particularly Netty profiling and per-plugin resource attribution, I recommend using the website.

Useful Links

• spark — Official Website
• spark — GitHub Repository
• Profiler Utils — Userscript
• Spark Profile Tools — Web Analyzer