![]() Practices for monitoring to grab the right data.ĭecide which aspect of performance is most important for you (what you want to improve) This document does not go into details of particular metrics and tools that youĬan use, it just describes what kind of resources you should monitor, but you should follow your best It’s extremely important to monitor your system with the right set of tools that you usually use to Optimizations (we will not recommend any specific tools - just use the tools that you usually use Generally for fine-tuning, your approach should be the same as for any performance improvement and Most important for you and decide which knobs you want to turn in which direction. Some users are ok withģ0 seconds delays of new DAG parsing, at the expense of lower CPU usage, whereas some other usersĮxpect the DAGs to be parsed almost instantly when they appear in the DAGs folder at theĪirflow gives you the flexibility to decide, but you should find out what aspect of performance is Part of the job when managing theĭeployment is to decide what you are going to optimize for. To decide which knobs to turn to get best effect for you. How to approach Scheduler’s fine-tuning ¶Īirflow gives you a lot of “knobs” to turn to fine tune the performance but it’s a separate task,ĭepending on your particular deployment, your DAG structure, hardware availability and expectations, You can take a look at the Airflow Summit 2021 talkĭeep Dive into the Airflow Scheduler talk to perform the fine-tuning. In order to perform fine-tuning, it’s good to understand how Scheduler works under-the-hood. How often the scheduler should perform cleanup and check for orphaned tasks/adopting them How many new DAG runs should be created/scheduled per loop How many task instances scheduler processes in one loop How much time scheduler waits between re-parsing of the same DAG (it happens continuously) How many parsing processes you have in your scheduler Whether parsing your DAG file involves importing a lot of libraries or heavy processing at the top level how fast they can be parsed, how many tasks and dependencies they have) ![]() How large the DAG files are (remember DAG parser needs to read and parse the file every n seconds)
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |