Celery Resources
Resource collection:
https://www.fullstackpython.com/task-queues.html
https://www.caktusgroup.com/blog/2014/09/29/celery-production/
Beat - Celery - periodic tasks --> use to check health of workers
https://www.vinta.com.br/blog/2017/celery-overview-archtecture-and-how-it-works/
Periodic tasks example:
https://www.twilio.com/blog/2015/11/international-space-station-notifications-with-python-redis-queue-and-twilio-copilot.html
- make it with beat.
https://medium.com/@channeng/setting-up-a-task-scheduler-application-with-celery-flask-part-1-8652265050dc
https://github.com/channeng/celery-scheduler
and docker
- eventlets - download apod for a month and store - also classify as mobile or desktop wallpaper
- web-scraping: download episodes of katekyo hitman reborn
- notification for when pratham will be overhead - 1 hour before
- https://www.distributedpython.com/2018/11/15/celery-docker/
- https://www.revsys.com/tidbits/celery-and-django-and-docker-oh-my/
celery + docker -->
Full demo:
https://medium.com/@tonywangcn/how-to-build-docker-cluster-with-celery-and-rabbitmq-in-10-minutes-13fc74d21730
similar to:
https://github.com/docker-library/docs/blob/d328e02359c6fc9a7f1f3c59efa2893f63e667e4/sentry/README.md#how-to-setup-a-full-sentry-instance
celery + sentry
routing - linux, windows
- https://www.slideshare.net/mahendram/advanced-task-management-with-celery
- slide 8 and 9
restart process after x number of tasks . - useful in memory leaks..
revoking tasks
task expiry
celery task tree
celery batches - reduce network and / or db load
monitoring:
- flower
- jobstatic
- django admin monitor
- celery events, celerymon
- celery inspect and celery control
- rabbitMQ monitoring plugin
logging:
- https://docs.celeryproject.org/en/latest/userguide/tasks.html#logging
- each task gets a logger.
Uses python logging as base:
https://docs.python.org/dev/library/logging.html#module-logging
logging exceptions:
kwargs which are inspected: exc_info, stack_info, stacklevel and extra
Logging patters:
- good, bad, ugly: https://www.loggly.com/blog/exceptional-logging-of-exceptions-in-python/https://www.loggly.com/ultimate-guide/python-logging-basics/
https://www.distributedpython.com/2018/11/06/celery-task-logger-format/
https://www.distributedpython.com/2018/08/28/celery-logging/
https://www.dimagi.com/blog/logging-inside-a-celery-task/
RabbitMQ - config:
https://www.rabbitmq.com/networking.html#dealing-with-high-connection-churn
https://vincent.bernat.ch/en/blog/2014-tcp-time-wait-state-linux
Stale connections:
https://www.rabbitmq.com/networking.html#heartbeats
Auto Scaling with AWS:
https://docs.aws.amazon.com/autoscaling/ec2/userguide/what-is-amazon-ec2-auto-scaling.html
https://docs.aws.amazon.com/autoscaling/ec2/userguide/scaling_plan.html#scaling_typesof
https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-scale-based-on-demand.html
May be some useful examples:
https://buildmedia.readthedocs.org/media/pdf/celery/stable/celery.pdf
Good resource for pool config description:
https://www.distributedpython.com/2018/10/26/celery-execution-pool/
-x-x-x-
small tips for VM setup -- convert it into a blog post
- nat
- second adapter --> host only adapter --> sort of VPN mimic
- mostly enable dhcp server for vboxnet0
- login once
- run dhclient -r --> remove RTNETLINK answers: File exists
- run dhclient --> generate
- ifconfig to get ip, subnet, broadcast
- /etc/netplan/01-hostonly-init.yaml
- test with try --debug flag; generate with --debug flag (netplan --debug generate)
- once ready, apply
x-x-x-
shared folder with OS: (https://askubuntu.com/a/965197)
- prepare VM for guest additions (install dependancies)
- https://www.tecmint.com/install-virtualbox-guest-additions-in-ubuntu/
- install host additions
- insert host additions cd
- mount /dev/cdrom /media/
- run the installer: sudo sh
https://askubuntu.com/a/22745 (complete for guest additions)
when guest additions installed
- https://help.ubuntu.com/community/VirtualBox/SharedFolders
-x-x-x-
make shared folder persistent:
https://gist.github.com/estorgio/1d679f962e8209f8a9232f7593683265#make-the-mount-folder-persistent
- fstab entry
@SHARED_FOLDER_NAME /host/mount/folder/name vboxsf default 0 0
- module entry (load kernel module at boot)
vboxsf
-x-x-x-
making a celery worker
- make a working dir --> for env (guest cannot use hosts env)
- symlink shared folder into working dir
- create virtual env
- install from requirements.txt in the shared folder !!
-x-x-x-
sudo apt-get install build-essential autoconf libtool pkg-config python-opengl python-imaging python-pyrex python-pyside.qtopengl idle-python2.7 qt4-dev-tools qt4-designer libqtgui4 libqtcore4 libqt4-xml libqt4-test libqt4-script libqt4-network libqt4-dbus python-qt4 python-qt4-gl libgle3 python-dev
Page faults:
https://scoutapm.com/blog/understanding-page-faults-and-memory-swap-in-outs-when-should-you-worry
Dashboard example:
https://medium.com/meilleursagents-engineering/how-we-monitor-asynchronous-tasks-da25728173d6
Complete practical tutorial:
https://tests4geeks.com/python-celery-rabbitmq-tutorial/
https://fernandofreitasalves.com/using-celery-with-multiple-queues-retries-and-scheduled-tasks/
https://www.vinta.com.br/blog/2018/celery-wild-tips-and-tricks-run-async-tasks-real-world/
https://medium.com/@taylorhughes/three-quick-tips-from-two-years-with-celery-c05ff9d7f9ebf
-x-x-x-
http://docs.celeryproject.org/en/master/getting-started/brokers/rabbitmq.html#setting-up-rabbitmq
enable management plugin:
https://www.rabbitmq.com/management.html#getting-started
give rabbitmq user admin priviledges:
-x-x-x-
My slides:
https://docs.google.com/presentation/d/1ugCgfW7yxpSN25Nngl2tySV9mUGSLsrbYgv6NH-ck1A/edit?usp=sharing
https://docs.google.com/presentation/d/1K8sHTHPgttYW1qsuAE1CzfsTTHPoUztwnsaBNdtPdtY/edit?usp=sharing
https://www.fullstackpython.com/task-queues.html
https://www.caktusgroup.com/blog/2014/09/29/celery-production/
Beat - Celery - periodic tasks --> use to check health of workers
https://www.vinta.com.br/blog/2017/celery-overview-archtecture-and-how-it-works/
Periodic tasks example:
https://www.twilio.com/blog/2015/11/international-space-station-notifications-with-python-redis-queue-and-twilio-copilot.html
- make it with beat.
https://medium.com/@channeng/setting-up-a-task-scheduler-application-with-celery-flask-part-1-8652265050dc
https://github.com/channeng/celery-scheduler
and docker
- eventlets - download apod for a month and store - also classify as mobile or desktop wallpaper
- web-scraping: download episodes of katekyo hitman reborn
- notification for when pratham will be overhead - 1 hour before
- https://www.distributedpython.com/2018/11/15/celery-docker/
- https://www.revsys.com/tidbits/celery-and-django-and-docker-oh-my/
celery + docker -->
Full demo:
https://medium.com/@tonywangcn/how-to-build-docker-cluster-with-celery-and-rabbitmq-in-10-minutes-13fc74d21730
similar to:
https://github.com/docker-library/docs/blob/d328e02359c6fc9a7f1f3c59efa2893f63e667e4/sentry/README.md#how-to-setup-a-full-sentry-instance
celery + sentry
routing - linux, windows
- https://www.slideshare.net/mahendram/advanced-task-management-with-celery
- slide 8 and 9
restart process after x number of tasks . - useful in memory leaks..
revoking tasks
task expiry
celery task tree
celery batches - reduce network and / or db load
monitoring:
- flower
- jobstatic
- django admin monitor
- celery events, celerymon
- celery inspect and celery control
- rabbitMQ monitoring plugin
logging:
- https://docs.celeryproject.org/en/latest/userguide/tasks.html#logging
- each task gets a logger.
Uses python logging as base:
https://docs.python.org/dev/library/logging.html#module-logging
logging exceptions:
kwargs which are inspected: exc_info, stack_info, stacklevel and extra
Logging patters:
- good, bad, ugly: https://www.loggly.com/blog/exceptional-logging-of-exceptions-in-python/https://www.loggly.com/ultimate-guide/python-logging-basics/
https://www.distributedpython.com/2018/11/06/celery-task-logger-format/
https://www.distributedpython.com/2018/08/28/celery-logging/
https://www.dimagi.com/blog/logging-inside-a-celery-task/
RabbitMQ - config:
https://www.rabbitmq.com/networking.html#dealing-with-high-connection-churn
https://vincent.bernat.ch/en/blog/2014-tcp-time-wait-state-linux
Stale connections:
https://www.rabbitmq.com/networking.html#heartbeats
Auto Scaling with AWS:
https://docs.aws.amazon.com/autoscaling/ec2/userguide/what-is-amazon-ec2-auto-scaling.html
https://docs.aws.amazon.com/autoscaling/ec2/userguide/scaling_plan.html#scaling_typesof
https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-scale-based-on-demand.html
May be some useful examples:
https://buildmedia.readthedocs.org/media/pdf/celery/stable/celery.pdf
Good resource for pool config description:
https://www.distributedpython.com/2018/10/26/celery-execution-pool/
-x-x-x-
small tips for VM setup -- convert it into a blog post
- nat
- second adapter --> host only adapter --> sort of VPN mimic
- mostly enable dhcp server for vboxnet0
- login once
- run dhclient -r --> remove RTNETLINK answers: File exists
- run dhclient --> generate
- ifconfig to get ip, subnet, broadcast
- /etc/netplan/01-hostonly-init.yaml
network:
version: 2
renderer: networkd
ethernets:
enp0s3:
addresses: []
dhcp4: true
enp0s8:
addresses: [192.168.56.101/24]
dhcp4: no
- test with try --debug flag; generate with --debug flag (netplan --debug generate)
- once ready, apply
x-x-x-
shared folder with OS: (https://askubuntu.com/a/965197)
- prepare VM for guest additions (install dependancies)
- https://www.tecmint.com/install-virtualbox-guest-additions-in-ubuntu/
- install host additions
- insert host additions cd
- mount /dev/cdrom /media/
- run the installer: sudo sh
https://askubuntu.com/a/22745 (complete for guest additions)
when guest additions installed
- https://help.ubuntu.com/community/VirtualBox/SharedFolders
sudo mount -t vboxsf -o uid=$UID,gid=$(id -g) sharedFolderName ~/guest/local/folder
-x-x-x-
make shared folder persistent:
https://gist.github.com/estorgio/1d679f962e8209f8a9232f7593683265#make-the-mount-folder-persistent
- fstab entry
@SHARED_FOLDER_NAME /host/mount/folder/name vboxsf default 0 0
- module entry (load kernel module at boot)
vboxsf
-x-x-x-
making a celery worker
- make a working dir --> for env (guest cannot use hosts env)
- symlink shared folder into working dir
- create virtual env
- install from requirements.txt in the shared folder !!
-x-x-x-
sudo apt-get install build-essential autoconf libtool pkg-config python-opengl python-imaging python-pyrex python-pyside.qtopengl idle-python2.7 qt4-dev-tools qt4-designer libqtgui4 libqtcore4 libqt4-xml libqt4-test libqt4-script libqt4-network libqt4-dbus python-qt4 python-qt4-gl libgle3 python-dev
Page faults:
https://scoutapm.com/blog/understanding-page-faults-and-memory-swap-in-outs-when-should-you-worry
Dashboard example:
https://medium.com/meilleursagents-engineering/how-we-monitor-asynchronous-tasks-da25728173d6
Complete practical tutorial:
https://tests4geeks.com/python-celery-rabbitmq-tutorial/
https://fernandofreitasalves.com/using-celery-with-multiple-queues-retries-and-scheduled-tasks/
https://www.vinta.com.br/blog/2018/celery-wild-tips-and-tricks-run-async-tasks-real-world/
https://medium.com/@taylorhughes/three-quick-tips-from-two-years-with-celery-c05ff9d7f9ebf
-x-x-x-
http://docs.celeryproject.org/en/master/getting-started/brokers/rabbitmq.html#setting-up-rabbitmq
enable management plugin:
https://www.rabbitmq.com/management.html#getting-started
give rabbitmq user admin priviledges:
-x-x-x-
My slides:
https://docs.google.com/presentation/d/1ugCgfW7yxpSN25Nngl2tySV9mUGSLsrbYgv6NH-ck1A/edit?usp=sharing
https://docs.google.com/presentation/d/1K8sHTHPgttYW1qsuAE1CzfsTTHPoUztwnsaBNdtPdtY/edit?usp=sharing
Comments
Post a Comment