Concurrent scrapyrt requests

[stefanobalzan]

Hello, I'm using scrapyrt to provide an HTTP interface to a big Scrapy project. I'm running a single scrapyrt instance in a Docker container, some spiders require ~60-120 seconds to complete and I've noticed that requests are handled sequentially, causing substantial delays.

Is that the expected behavior? I know scrapyrt is not suitable for long running spiders, but I'm wondering if there exist a quick fix, for example running multiple workers/threads. Asking here cause I'm not really familiar with the twisted framework.
Another solution would be running multiple scrapyrt instances behind a load balancer, but I'd rather not go down that path.

[doverradio]

I'm facing this same issue right now.

Previously, I had solved it by just creating more instances of the scrapy scripts on various ports.

Did you solve it?

[AlejoGarat]

Hi @pawelmhm, I’m a bit confused about this issue. I’m currently using ScrapyRT in a project, and while I do notice that more active users increase the crawl time, the delays are not as huge as described by @BalzySte in this post.

I’d really appreciate if you could help me understand how ScrapyRT manages requests internally. Specifically, at what point requests get enqueued and cause delays?

Thanks in advance!