FlaskApp significantly slow compared to AsyncApp or Flask with gevent

[pradhan-v]

When running a Connexion application with FlaskApp, the server becomes very slow or even blocked under moderate user load. However, the same application performs significantly better when ported to AsyncApp, when run as a pure Flask application with gevent.

This suggests there may be an issue in FlaskApp. This performance gap makes it difficult to rely on FlaskApp in production.

I’ve created a minimal reproducible example here:
👉 https://github.com/pradhan-v/connexion-slow-flaskapp

Environment:
Connexion version: 3.2.0
Python version: 3.12

Related?: #2045

[chrisinmtown]

What times do you see when you run your curl script? Do you have any idea of where the slowdown might be? Did you by any chance gather any performance/profile data with something like cProfile? From what I understand about Connexion, on receipt of a request it has to validate the payload and find the app endpoint that handles it, then it's done. Could Connexion possibly be doing some kind of expensive-in-time computation for those initial steps?

Anyhow, for now I'm trying to reproduce your result of varying response times in the Connexion v3 FlaskApp. I cloned your repo and for a first attempt I thought maybe I would avoid the complicating factor of Docker and just run the app locally (I'm using an old mac system for what that's worth). I created a local venv, installed the requirements then launched the FlaskApp like this in a window:
./entrypoint.sh flaskapp
Then in another window, after I changed the port in the script bcos I only have one server running, I ran the curl script. So far, the 2 UvicornWorker instances always process every batch of 200 FlaskApp requests in exactly 22 seconds. Here's the output:

% ./run_curl.sh
Invoking connexion_flaskapp on port 8080
All requests to connexion_flaskapp on 8080 completed in 22 seconds.

Please let me know if I have faithfully reproduced your results, if I made any dumb mistakes, etc.

[pradhan-v]

Hi @chrisinmtown , Thank you for looking in to this.

What times do you see when you run your curl script?

A sample output from my laptop is shown in the Readme file of that project. Pasting it here.

❯ ./run_curl.sh
Invoking connexion_asyncapp on port 8001
All requests to connexion_asyncapp on 8001 completed in 3 seconds.
Invoking connexion_flaskapp on port 8002
All requests to connexion_flaskapp on 8002 completed in 24 seconds.
Invoking flask on port 8003
All requests to flask on 8003 completed in 3 seconds.

The FlaskApp takes 24 seconds while the Connexion AsyncApp and pure Flask application with gevent takes only 3 seconds. All that the server does is to sleep for 2 seconds, for every request. We could replace this sleep with any other I/O operation.

Do you have any idea of where the slowdown might be? Did you by any chance gather any performance/profile data...

I have not profiled this program using any performance tools. Im dont think that this problem is related to the initial steps, I have my doubts on how the Connexion FlaskApp is handling the threads within the ASGI runtime. I could be completely wrong as I consider myself to be a noob in the Python ecosystem.

_Please let me know if I have faithfully reproduced your results, ... _

You are on the right track. If you run the AsyncApp and the pure Flask application in a similar way, you will see that they are much faster compared to the connexion FlaskApp.

Here is a simplified version bash script with curl invocation to simulate 200 parallel requests.

for i in {1..200}
do
  curl -s -o /dev/null localhost:8080/greeting/hello$i &
done
wait # optional

As you already did, run the program using the ./entrypoint.sh flaskapp command, and in the other terminal run the above script to make the requests.

FlaskApp takes more time to complete all the requests. I also observed that the requests are handled in "bursts" of 20 for the first few, and then 10 for the last few requests. The server hangs for a second or so between bursts.

When the same steps are followed for ./entrypoint.sh flask(Pure Flask, no Connexion) or ./entrypoint.sh asyncapp(Connexion Async), the requests are completed within 3 seconds.
(Note the 2 seconds of sleep to simulate I/O)

[pradhan-v]

When we get 1000s of request, the server is unable to handle the load and are blocked on I/O. Our service mainly makes DB, Cache and upstream API requests.
Flask has blocking I/O and this behavior may be expected. If this applies to Connexion FlaskApp too then, is there an option in Connexion to handle this scenario, similar to using gevent workers in Flask?

To make matters worse, we have introduced a streaming API which takes an additional 5 to 50 seconds of I/O, and I believe that this is going to slow down the server further. I will update my sample repo and add a /stream too.

Please let me know if Im missing something obvious or not using this the right way.

[chrisinmtown]

I'm trying to learn about the pieces in play here, forgive if I'm slow to catch on. Guessing at a lower limit, curl launches 200 requests and each takes 2 seconds, the total time spent sleeping is 400 seconds. This example launches two Gunicorn workers and each worker defaults to one thread. If each worker processes half the requests in a synchronous way, the elapsed time could be as high as 200 seconds. With that said, elapsed time is always far below that, so requests are obviously being processed in parallel by Flask.

Trying to get more information, I changed the entry point to use just 1 worker, and modified the setup_logging function to report the thread doing the work:

    logging_format = '%(levelname)s %(threadName)s %(name)s.%(funcName)s %(message)s'
    logging.basicConfig(format=logging_format, level=logging.INFO)

Then I ran the connexion+flask app and saw batches of lines like this in the output:

INFO WSGI_0 app.some_io_operation_service.perform_io_operation Blocking I/O operation for 2 seconds. flask62
INFO WSGI_7 app.some_io_operation_service.perform_io_operation Blocking I/O operation for 2 seconds. flask30
INFO WSGI_5 app.some_io_operation_service.perform_io_operation Blocking I/O operation for 2 seconds. flask4
INFO WSGI_6 app.some_io_operation_service.perform_io_operation Blocking I/O operation for 2 seconds. flask32
INFO WSGI_3 app.some_io_operation_service.perform_io_operation Blocking I/O operation for 2 seconds. flask65
INFO WSGI_9 app.some_io_operation_service.perform_io_operation Blocking I/O operation for 2 seconds. flask51
INFO WSGI_8 app.some_io_operation_service.perform_io_operation Blocking I/O operation for 2 seconds. flask35
INFO WSGI_1 app.some_io_operation_service.perform_io_operation Blocking I/O operation for 2 seconds. flask71
INFO WSGI_2 app.some_io_operation_service.perform_io_operation Blocking I/O operation for 2 seconds. flask78
INFO WSGI_4 app.some_io_operation_service.perform_io_operation Blocking I/O operation for 2 seconds. flask50

And the total run time is 41 seconds. This says to me, Gunicorn + Connexion + Flask use 10 threads, each thread synchronously processes 1/10 of the requests, about 40 seconds total.

I think the direct comparison to make is Connexion + Flask against pure Flask. In your example for plain Flask, the entrypoint.sh script sets Gunicorn worker class gevent which introduces async processing of requests. The plain connexion+flask example does not have this advantage. I tried dropping that special worker class for the Flask app, trying to make the comparison more fair, but that yielded an error that I have not figured out (TypeError: Flask.__call__() missing 1 required positional argument: 'start_response'). I kind of stalled out at this point in the comparison.

tl;dr - based on the number of threads I see in use, the Connexion + FlaskApp response time seems reasonable to me right now.

[pradhan-v]

for plain Flask, the entrypoint.sh script sets Gunicorn worker class gevent which introduces async processing of requests. The plain connexion+flask example does not have this advantage.

This is precisely the problem

Can Connexion provide this flexibility for FlaskApp? Are there any settings we can apply to reduce this blocking behavior?

[chrisinmtown]

To restate the obvious, Connexion V3 is focused on async / ASGI, a big change from the old V2. I believe the support for FlaskApp in Connexion V3 is primarily intended to give people a reasonable migration path, and includes some compromise. I sure hope an expert or maintainer will jump in here to provide more information about whether (or not) a gevent worker can be used for a FlaskApp in Connexion V3.

[pradhan-v]

I believe the support for FlaskApp in Connexion V3 is primarily intended to give people a reasonable migration path, and includes some compromise.

If there is no fix for this and FlaskApp is supposed to be used in projects that intend to migrate only, then this must be made evident in the documentation. I see that this is mentioned in the docs but the line also gives an impression that Flask is fully supported.

The FlaskApp leverages the Flask framework, which is useful if you're migrating from connexion 2.X or you want to leverage the Flask ecosystem.

It must be made clear that the FlaskApp is not ready for production use, especially with services receiving loads of requests. I would also recommend to put a warning message in the logs when the FlaskApp starts.

[RobertBlackhart]

@pradhan-v - we recently migrated our application from to Connexion 3.x and we make use of gevent as well. We did not transition to using async controller functions though (other dependencies in the way of that).

Initially, we were using the FlaskApp because that was the most seamless way forward. We had found that we were limited to only having 10 requests be processed in parallel. While trying to track down why, it seemed to essentially boil down to how Connexion allows for both sync and async controller functions. This decorator gets used:

connexion/connexion/decorators/main.py

Lines 167 to 179 in 1d4bb81

	@property
	def _sync_async_decorator(self) -> t.Callable[[t.Callable], t.Callable]:
	def decorator(function: t.Callable) -> t.Callable:
	@functools.wraps(function)
	async def wrapper(*args, **kwargs):
	if asyncio.iscoroutinefunction(function):
	return await function(*args, **kwargs)
	else:
	return await run_in_threadpool(function, *args, **kwargs)
	return wrapper
	return decorator

This basically says to me that if the thing being called is async, then put it on the even loop to be executed, else run it in a thread pool. The run_in_threadpool function comes from from starlette.concurrency import run_in_threadpool. That's a pretty thin wrapper over the anyio library: https://github.com/encode/starlette/blob/eb6cebbeecbbc44f87acb310330b37813d06598d/starlette/concurrency.py#L35-L37.

Looking around the starlette documentation and threads, I found this: Kludex/starlette#1724 (comment). It seems like the default concurrency level is 40, but we only seem to be getting 10. I'm not sure how to explain that difference. Regardless, here's what we did to add that change into our app:

@contextlib.asynccontextmanager
async def startup_shutdown(app):
    limiter = anyio.to_thread.current_default_thread_limiter()
    limiter.total_tokens = 300  # or whatever value you want for your app
    try:
        yield
    except asyncio.CancelledError:
        logging.info("Shutting down the application")

app = connexion.AsyncApp(__name__, lifespan=startup_shutdown)