Container Apps Job Being Preempted
I have a Container App Job using event scaling. The scaling is configured to listen to a Service Bus queue and run 1 job per message. Most of the time it is working fine. However, when multiple messages are added to the queue in quick succession, one or two of the jobs are getting preempted mid execution as their job execution stops and pulls a new message instead. This is the message I see in the System logs:
Preempted by pod 8c6a82b0-7d3d-46f3-be21-abd11fddcc8c on node aks-systempool-22769013-vmss000000
In my latest test, I had the max replicas set to 8 and published 14 messages to the queue. All except one were processed successfully. The one which failed logged the above message and the container shutdown, and then restarted.
There seems to be quite limited documentation around Container Apps Jobs and not a lot of tooling for debugging. Is this behavior expected? Is there anyway to prevent preemption from occurring?