fix "Missing lock for job" errors and better concurrency handling for self host mode

[ciekawy]

I did that for my personal needs related to #1327, hope others can benefit from this

[FelixGYT]

First of all, thank you for your generosity. He has helped me a lot.
I deployed your code and it looks much better. There won't be any errors during 20 concurrent attempts.
But after about 20 concurrent requests on my end for a period of time, it still reports the error Job wait.
I checked the logs and found that the work log should have been successful, but there is a job wait issue in the API log.
I don't know if you have encountered the same problem, I'm curious why there is such a problem

[ciekawy]

unfortunately I have very little time and have no issues, while also it is working for my local instance and I use it daily (though not super intensively) - this should be debugged.

IMO extending lock itself is not an issue - same ID ls logged few lines below as done. May be worth checking logs from other containers (e.g. docker compose logs --follow - without container name shows from all and observe - they will be more in sync ) - I'd guess some jobs might possibly get stuck in playwright or something, but definitly timeout should be better handled. So the error might actually be real.

One thing is that all containers, playwright specifically may need more RAM (may require changing either global docker, or whats assigned to specific container).

BTW PLEASE always copy text, not screen :)

[ciekawy]

Also both Error in scrapeController and Job wait are logged from https://github.com/mendableai/firecrawl/blob/e799cf206b5d357158d242f4dcec76943f47194d/apps/api/src/controllers/v1/scrape.ts - not even subject to my PR - this would need further investigation. You could use LLM (preferably Claude Sonnet 3.7 thinking for this), sharing also relevant source files and relevant logs from all containers around the time the error occur