Fixing High CPU Usage In Kubernetes Pods

Aug 13, 2025 by RICHARD 41 views

Analyzing and Resolving High CPU Usage in Kubernetes Pod 'test-app:8001'

CPU Usage Analysis

Pod Information

Pod Name: test-app:8001
Namespace: default

Analysis

In our scenario, the test-app:8001 pod was experiencing high CPU usage, leading to frequent restarts. After a thorough investigation, the logs revealed normal application behavior, but the persistent high CPU consumption pointed towards an underlying issue. The primary suspect was identified as the cpu_intensive_task() function. This function was designed to simulate CPU-intensive operations by running an unoptimized brute-force shortest path algorithm on large graphs. However, it lacked proper rate limiting and resource constraints, causing it to overwhelm the system. The function's design involved creating multiple CPU-intensive threads, which, without any controls, could easily lead to CPU spikes and the observed pod restarts.

To understand this better, let's break down why this cpu_intensive_task() function was causing so much trouble. The brute-force shortest path algorithm, by its very nature, is computationally expensive. It explores every possible path in a graph to find the shortest one, which means the time it takes to complete grows exponentially with the size of the graph. When the function was running on large graphs, it was essentially trying out a vast number of possibilities, each requiring CPU cycles. The lack of rate limiting meant that these calculations were happening continuously, without any pauses to allow the system to catch its breath. Moreover, the creation of multiple threads compounded the problem. Each thread was independently performing these intensive calculations, all competing for CPU resources simultaneously. This perfect storm of factors resulted in the CPU usage spiking to unsustainable levels, ultimately causing the pod to become unresponsive and restart. In essence, the application was trying to do too much, too quickly, without any mechanism to throttle its resource consumption.

Proposed Fix

To address the high CPU usage issue, a series of optimizations were proposed for the cpu_intensive_task() function. These changes aimed to reduce the computational load and introduce mechanisms to prevent CPU spikes. The proposed fix includes four key strategies:

Reducing the Graph Size: The initial graph size of 20 nodes was deemed too large for the simulation's needs. Reducing it to 10 nodes significantly decreases the computational complexity of the shortest path algorithm. This is because the number of possible paths in a graph increases dramatically with each additional node. By halving the graph size, the number of paths to explore is reduced exponentially, leading to a substantial decrease in CPU usage. Think of it like searching for a needle in a haystack; a smaller haystack means less searching. This simple adjustment can have a profound impact on the overall CPU load.
Adding Rate Limiting: To prevent the function from overwhelming the system, a rate-limiting sleep of 0.1 seconds was introduced between iterations. This small pause allows the CPU to recover and prevents the task from consuming all available resources continuously. Rate limiting is a common technique in software development to control the frequency of operations. In this context, it acts like a traffic light, pausing the execution briefly to prevent congestion. This ensures that the CPU has time to process other tasks, preventing a complete resource hogging situation.
Adding Maximum Execution Time Check: A maximum execution time check of 5 seconds per iteration was implemented. If an iteration takes longer than 5 seconds, it is terminated. This prevents the algorithm from running indefinitely on particularly complex graphs. This is a crucial safeguard against scenarios where the algorithm might get stuck in a particularly challenging part of the graph, consuming CPU resources for an extended period. By setting a time limit, we ensure that the algorithm doesn't monopolize the CPU, and the system remains responsive.
Reducing Maximum Path Depth: The maximum path depth in the shortest path algorithm was reduced from 10 to 5. This limits the search space and further reduces the computational complexity. Similar to reducing the graph size, limiting the path depth reduces the number of possibilities the algorithm needs to explore. This is like narrowing down your search area, making it easier to find what you're looking for. By reducing the path depth, we significantly decrease the amount of computation required for each iteration.

By implementing these changes, we aim to strike a balance between allowing the simulation functionality to work effectively and preventing CPU spikes. These modifications ensure that the cpu_intensive_task() function remains within acceptable resource limits, preventing the pod from becoming unresponsive and restarting.

Code Change

The following code snippet demonstrates the implemented changes to the cpu_intensive_task() function:

def cpu_intensive_task():
    print(f"[CPU Task] Starting CPU-intensive graph algorithm task")
    iteration = 0
    while cpu_spike_active:
        iteration += 1
        # Reduced graph size and added rate limiting
        graph_size = 10
        graph = generate_large_graph(graph_size)
        
        start_node = random.randint(0, graph_size-1)
        end_node = random.randint(0, graph_size-1)
        while end_node == start_node:
            end_node = random.randint(0, graph_size-1)
        
        print(f"[CPU Task] Iteration {iteration}: Running optimized shortest path algorithm on graph with {graph_size} nodes from node {start_node} to {end_node}")
        
        start_time = time.time()
        path, distance = brute_force_shortest_path(graph, start_node, end_node, max_depth=5)
        elapsed = time.time() - start_time
        
        if path:
            print(f"[CPU Task] Found path with {len(path)} nodes and distance {distance} in {elapsed:.2f} seconds")
        else:
            print(f"[CPU Task] No path found after {elapsed:.2f} seconds")
            
        # Add rate limiting sleep
        time.sleep(0.1)
        
        # Break if taking too long
        if elapsed > 5:
            print(f"[CPU Task] Task taking too long, breaking iteration")
            break

Let's break down the key changes in this code snippet. First, the graph_size has been reduced from its original value to 10. This directly addresses the issue of excessive computational complexity by decreasing the number of nodes in the graph, thereby reducing the number of possible paths to explore. Next, we've introduced a time.sleep(0.1) call within the loop. This is the rate-limiting mechanism, which pauses the execution for 0.1 seconds after each iteration, preventing the function from consuming CPU resources continuously. This small pause allows the system to catch its breath and process other tasks, avoiding CPU spikes. Additionally, a check for maximum execution time has been implemented. The start_time is recorded before the pathfinding algorithm is executed, and the elapsed time is calculated afterward. If elapsed exceeds 5 seconds, the iteration is terminated with a message, preventing the algorithm from running indefinitely on complex graphs. Finally, the max_depth parameter in the brute_force_shortest_path function call has been reduced to 5. This limits the depth of the search, further reducing the computational complexity of the algorithm. These changes collectively ensure that the cpu_intensive_task function operates within reasonable resource constraints, preventing high CPU usage and pod restarts.

File to Modify

main.py

Next Steps

A pull request will be created with the proposed fix. This allows for a collaborative review process where other developers can examine the changes, provide feedback, and ensure the fix is implemented correctly. Once the pull request is approved, the changes will be merged into the main codebase, and the updated application will be deployed. This will effectively address the high CPU usage issue and prevent future pod restarts. By following this process, we ensure that code changes are thoroughly vetted and integrated smoothly into the application.