Struggling with high request rate

[rldleblanc]

I'm struggling to get Goose to exceed 269,000 requests per second on a single machine even though there is plenty of capacity left.

I have 2 systems with 64 cores and 1TiB of RAM with 100 Gbps Nics. My application is very high transaction (100-200 microseconds response time on average). I've been able to test ActiX on a single machine up to 1.1 million requests per second with Locust (8 servers and 384 cores), so I know what the server is capable and it consumed 55 cores on the application machine, so there was still some head room.

I'm struggling to get Goose to use more than 24 users as the application just becomes unresponsive and I don't get any more output from the test after the last launched user message. Goose is consuming about 10 cores and my application is consuming about 10 cores and I'm getting about 269k RPS.

Steps to reproduce:

1. Stand up a simple ActiX server and in the route return a success response.
2. Create a simple client that just connects and downloads the file.

use goose::prelude::*;
use rand::prelude::*;

#[tokio::main]
async fn main() -> Result<(), GooseError> {
    GooseAttack::initialize()?
        .register_scenario(
            scenario!("LoadTest")
                .register_transaction(transaction!(simple_test))
        )
        .execute()
        .await?;

    Ok(())
}

async fn simple_test(user: &mut GooseUser) -> TransactionResult {
    const DIR_WIDTH: usize = 2;
    const DIR_DEPTH: usize = 1;
    const FILE_WIDTH: usize = 2;
    const CHARS: &[char] = &['a', 'b', 'c'];
    
    // Generate path in a separate block so rng is dropped before await
    let path = {
        let mut rng = rand::rng();
        let mut path = String::from("/pub-test/testfiles2");
        
        for _ in 0..DIR_DEPTH {
            path.push('/');
            for _ in 0..DIR_WIDTH {
                path.push(*CHARS.choose(&mut rng).unwrap());
            }
        }
        
        path.push('/');
        for _ in 0..FILE_WIDTH {
            path.push(*CHARS.choose(&mut rng).unwrap());
        }
        
        path
    }; // rng is dropped here, before the await
    
    let _goose = user.get(&path).await?;
    
    Ok(())
}

Goose is giving me better performance overall than Locust which has cost me a couple of days chasing a ghost in my application. I see the issue about incorrect latency numbers in the graph and I can confirm that it gets very wonky as time passes (saying that the latency was 15 nanoseconds, man I sure wish that was true). Keep up the good work! Luckily I don't need this exact workload to see the absolute max may app and hardware