Skywalking-Logger Plugin: Fix No Memory Errors Under Load

Aug 14, 2025 by RICHARD 58 views

Help! Skywalking-Logger Plugin Throwing "No Memory" Errors Under High Concurrency

Hey everyone,

We've got a tricky situation on our hands and could really use your expertise! We're encountering "no memory" errors with the skywalking-logger plugin in our Apache APISIX setup under high concurrency. Let's dive into the details and see if we can figure this out together.

The Issue: "No Memory" Errors with Skywalking-Logger

So, the main problem we are facing is that the skywalking-logger plugin is throwing these frustrating "no memory" errors. This is happening specifically when the system is under heavy load. It’s like the plugin is getting overwhelmed and running out of resources. This is a critical issue because it impacts our ability to monitor and trace our services effectively.

The error messages look something like this:

[Include the image of the error message here]

This screenshot clearly shows the "No memory" error, which indicates that the plugin is failing to allocate memory, likely due to the high volume of requests. This is a common issue in high-throughput systems, but we need to pinpoint the exact cause in our setup.

Our APISIX Configuration

To give you a clearer picture, here’s our APISIX configuration. We want to make sure everything is set up correctly and that there aren’t any obvious misconfigurations causing this issue.

[Include the image of the APISIX config here]

This config outlines the core settings for our APISIX instance, including the plugins enabled, logging configurations, and other important parameters. It’s crucial to review this to ensure that the resources allocated are sufficient for the expected load. We've tried to optimize the settings, but the errors persist under high concurrency.

Route Configuration Details

Here’s a look at our route configuration. This is where we define how traffic is handled and which plugins are applied to specific routes.

[Include the image of the route config here]

The route configuration is where we specify which services are being proxied and the plugins applied to them. In this case, we’re using the skywalking-logger plugin, and it seems the issue might be related to how this plugin interacts with the traffic on these routes. We need to examine this configuration closely to identify any potential bottlenecks or misconfigurations.

Environment Information

To help with troubleshooting, here’s some information about our environment:

APISIX version: 3.13
Operating system: Linux d0ea4fda6c64 5.15.0-151-generic #161-Ubuntu SMP Tue Jul 22 14:25:40 UTC 2025 x86_64 GNU/Linux
OpenResty / Nginx version: 1.27.1.2
etcd version: 3.5.11
APISIX Dashboard version: (Not specified)
Plugin runner version: (Not specified)
LuaRocks version: (Not specified)

Knowing the versions of these components is vital for identifying compatibility issues or known bugs. For instance, there might be specific issues with the skywalking-logger plugin in APISIX 3.13 or with the combination of OpenResty 1.27.1.2 and etcd 3.5.11. This information helps narrow down the potential causes and find relevant solutions or workarounds.

Possible Causes and Solutions

1. Memory Limits and Concurrency

One potential cause of the "no memory" errors could be that the memory limits set for the APISIX worker processes are insufficient to handle the high concurrency. When a large number of requests come in simultaneously, the skywalking-logger plugin might be trying to allocate more memory than is available, leading to these errors. Think of it like trying to fit too many people into a small room – eventually, you run out of space.

To address this, we can try increasing the memory limits for the worker processes. This can often be done by adjusting the worker_processes and worker_connections directives in the Nginx configuration. However, simply increasing these limits without proper monitoring can lead to other issues, such as excessive memory consumption and system instability. It’s crucial to monitor the system’s memory usage after making these changes to ensure that the adjustments are effective and not causing new problems.

2. Plugin Configuration Issues

Another possibility is that there might be some misconfigurations within the skywalking-logger plugin itself. Perhaps there are settings that are not optimized for high concurrency, or there could be resource leaks within the plugin’s code. We need to carefully review the plugin’s configuration to ensure that it's set up correctly.

This involves checking the plugin's specific settings, such as buffer sizes, logging levels, and sampling rates. Incorrect settings can lead to excessive memory usage or processing overhead. Additionally, if there are known issues or bugs in the plugin version being used, upgrading to a more stable version might resolve the problem. Reviewing the plugin's documentation and release notes can provide valuable insights into potential configuration issues and fixes.

3. Underlying System Limitations

Sometimes, the issue might not be directly related to APISIX or the plugin but could stem from the underlying system limitations. For instance, if the server has limited RAM or if there are other processes consuming significant memory, it could impact APISIX’s ability to allocate memory.

To diagnose this, we need to monitor system-level metrics such as CPU usage, memory consumption, and disk I/O. Tools like top, htop, and vmstat can provide valuable information about resource utilization. Identifying other processes that are consuming significant resources can help pinpoint potential conflicts. If the system is indeed resource-constrained, options include upgrading the hardware, optimizing other processes, or distributing the load across multiple servers.

4. Code-Level Issues in the Plugin

In more complex scenarios, there might be actual code-level issues within the skywalking-logger plugin. This could include memory leaks, inefficient algorithms, or other bugs that cause the plugin to consume excessive resources under high load. This is less common but still a possibility.

To investigate this, we might need to delve into the plugin’s source code and perform a code review. Tools like memory profilers and debuggers can help identify memory leaks and performance bottlenecks. If a bug is found, it might require patching the plugin or reporting the issue to the plugin developers. This often involves a deeper level of technical expertise and collaboration with the plugin's maintainers.

Seeking Advice and Solutions

So, guys, we’re really scratching our heads here. We’ve checked our configs, looked at the error messages, and explored some potential causes, but we’re still not 100% sure what’s going on. That’s why we’re reaching out to you, the community, for some help.

Here are some specific questions we have:

Has anyone else encountered similar "no memory" errors with the skywalking-logger plugin under high concurrency?
Are there any specific configuration tweaks or optimizations that we should consider?
Could there be compatibility issues between our APISIX version, OpenResty, and the plugin?
What are the best practices for debugging memory-related issues in APISIX plugins?

If you’ve faced this before or have any insights, please share your thoughts and suggestions. We’re open to any advice that can help us resolve this issue and ensure our system runs smoothly. Let’s work together to get this sorted out!

We really appreciate any help you can offer! Thanks in advance for your time and expertise.