Azure Data Lake Analytics: Greater flexibility with assigning Parallelism to U-SQL Jobs
The Data Lake team is always receiving very useful and thoughtful feedback from users on how we can improve the Data Lake service. For Data Lake Analytics, one very common request we have heard is that you want more control over allocating Parallelism for you U-SQL jobs. Specifically, we've been asked to allow a single job to use as much of the parallelism available in the account. The team is happy to announce today, that we've made implemented and released this improvement!
Description
Previously an Data Lake Analytics account was limited to a total of 60 units on parallelism available to 3 concurrent U-SQL jobs. And each U-SQL job was limited to only using a maximum of 20 units of parallelism. However, sometimes developers want to run a U-SQL job that uses more parallelism. For example: they might want two concurrently running jobs that each use 30 units. Today, we are removing the per-job limit. Now you can concurrently run jobs that use any amount of parallelism as long as the total for the running jobs doesn’t go beyond the maximum for your account (currently 60 units of parallelism).
If you are already running U-SQL jobs, there’s no need to worry. This change doesn’t require any action on your part. Your jobs run just as they did before. But now, if you want to you can take advantage of all the parallelism in your account.
Frequently Asked Questions
Q: How can I change the maximum number of concurrent U-SQL Jobs?
Right now you’ll have to contact us via a support ticket. Soon, we will give you direct control over this setting and you will not need to contact us.
Q: What happens if I use all my parallelism in one job and then submit another job?
The behavior is the same as today, if there is not enough parallelism available for a new job, then the new job will not start until there is enough parallelism available.
How can I change the total amount of parallelism for my account?
Right now you’ll have to contact us via a support ticket. In the future, we are looking at giving you more direct control over this setting.
Does this change affect the cost of using an Data Lake Analytics account?
No. There’s no change to billing.