We have a Python crawler that uses Celery Workers and feeds the urls from Redis.
It starts off fine - then , without any reason. It Gradually slows down to a halt.
Some of the workers are getting stuck - even after a reboot they remain stuck!
We need an URGENT fix as I am on a very short deadline.
PLEASE DO NOT APPLY IF YOU DO NOT HAVE EXPERIENCE.