How to fail a (cron) job after a certain number of retries? : <p>We have a Kubernetes cluster of web scraping cron jobs set up. All seems to go well until a cron job starts to fail (e.g., when a site structure changes and our scraper no longer works). It looks like every now and then a few failing cron jobs will continue to retry to the point it brings down our cluster. Running <code>kubectl get cronjobs</code> (prior to a cluster failure) will show too many jobs running for a failing job.</p>

<p>I've attempted following the note described <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#pod-backoff-failure-policy" rel="noreferrer">here</a> regarding a known issue with the pod backoff failure policy; however, that does not seem to work.</p>

<p>Here is our config for reference:</p>

<pre><code>apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: scrape-al
spec:
  schedule: '*/15 * * * *'
  concurrencyPolicy: Allow
  failedJobsHistoryLimit: 0
  successfulJobsHistoryLimit: 0
  jobTemplate:
    metadata:
      labels:
        app: scrape
        scrape: al
    spec:
      template:
        spec:
          containers:
            - name: scrape-al
              image: 'govhawk/openstates:1.3.1-beta'
              command:
                - /opt/openstates/openstates/pupa-scrape.sh
              args:
                - al bills --scrape
          restartPolicy: Never
      backoffLimit: 3
</code></pre>

<p>Ideally we would prefer that a cron job would be terminated after N retries (e.g., something like <code>kubectl delete cronjob my-cron-job</code> after <code>my-cron-job</code> has failed 5 times). Any ideas or suggestions would be much appreciated. Thanks!</p>
