K8 Pod not starting for larger load test script

[nitesh-mykaarma]

Brief summary

So I have large size load test script (size ~1MB), to run this I have used persistent volume to mount this file

Template YML file Code

- name: fetch-script
      inputs: {}
      outputs: {}
      metadata: {}
      container:
        name: ''
        image: alpine:latest
        command:
          - sh
          - '-c'
        args:
          - >
            apk add --no-cache curl && curl -X GET
            "{{workflow.parameters.gitlink}}" >
            /mnt/data/{{workflow.parameters.scriptFileName}}
        resources: {}
        volumeMounts:
          - name: stress-test-volume
            mountPath: /mnt/data
      volumes:
        - name: stress-test-volume
          persistentVolumeClaim:
            claimName: argo-pvc`

But when giving parallelism to 2 or 4 one or two of the pods failed to start (strange that other pods start and are able to fetch script correctly)

The error in load-test-initializer

error msg="The moduleSpecifier \"/test/authentication-utils.js\" couldn't be found on local disk. Make sure that you've specified the right path to the file. If you're running k6 using the Docker image make sure you have mounted the local directory (-v /local/path/:/inside/docker/path) containing your script and modules so that they're accessible by k6 from inside of the container, see https://grafana.com/docs/k6/latest/using-k6/modules/#using-local-modules-with-docker."

I am using k6 version - 0.55
K6 operator REVISION: 2 CHART: k6-operator VERSION: 3.8.0 APP_VERSION: 0.0.16

k6-operator version or image

3.8.0

Helm chart version (if applicable)

No response

TestRun / PrivateLoadZone YAML

manifest: |
          apiVersion: k6.io/v1alpha1
          kind: TestRun
          metadata:
            name: load-test
          spec:
            parallelism: {{workflow.parameters.parallelism}}
            script:
              volumeClaim:
                name: argo-pvc
                file: {{workflow.parameters.scriptFileName}}
            arguments: --tag testid={{workflow.parameters.testId}} --log-output=stdout --verbose
            runner:
              livenessProbe:
                httpGet:
                  path: /v1/status
                  port: 6565
                initialDelaySeconds: 60
                periodSeconds: 30
                timeoutSeconds: 25
                failureThreshold: 6
                successThreshold: 1
              readinessProbe:
                httpGet:
                  path: /v1/status
                  port: 6565
                initialDelaySeconds: 60
                periodSeconds: 30
                timeoutSeconds: 25
                failureThreshold: 6
                successThreshold: 1  
              securityContext:
                runAsNonRoot: false
                runAsUser: 0  # Run as root
                fsGroup: 0    # Set file system group to root
              resources:
                limits:
                  cpu: 2000m
                  memory: 4096Mi
                requests:
                  cpu: 1000m
                  memory: 2048Mi  
              volumes:
                - hostPath:
                    path: /var/log/kaarya
                  name: logfolder  
              volumeMounts:
                - mountPath: /tmp
                  name: logfolder 
              image: 578061096415.dkr.ecr.us-east-1.amazonaws.com/load-test:1.1.0
              imagePullSecrets:
                - name: dockercred
              imagePullPolicy: Always
              env: 
                - name: DB_USERNAME
                  value: {{workflow.parameters.db_username}}
                - name: DB_PASSWORD
                  value: {{workflow.parameters.db_password}}
                - name: DB_DATABASE
                  value: {{workflow.parameters.db_name}}
                - name: DB_HOST
                  value: {{workflow.parameters.db_host}}
                - name: API_BASE_URL
                  value: {{workflow.parameters.api_base_url}}
                - name: TEST_DEALER_UUID
                  value: {{workflow.parameters.test_dealer_uuid}}
                - name: TEST_DEPARTMENT_UUID
                  value: {{workflow.parameters.test_department_uuid}}
                - name: SERVICE_SUSBCRIBER_USERNAME
                  value: {{workflow.parameters.ss_username}}
                - name: SERVICE_SUSBCRIBER_PASSWORD
                  value: {{workflow.parameters.ss_password}}
                - name: TEST_DEALER_ID
                  value: "{{workflow.parameters.test_dealer_id}}"

Other environment details (if applicable)

No response

Steps to reproduce the problem

Mentioned the brief section

Expected behaviour

All k8 Pod should start and execute test.

Actual behaviour

One of the pods failed with the error exec /usr/bin/k6: exec format error

[yorugac]

Hi @nitesh-mykaarma,

But when giving parallelism to 2 or 4 one or two of the pods failed to start (strange that other pods start and are able to fetch script correctly)

So you're saying that parallelism: 1 works fine but failure happen only on 2+ parallelism? In that case, it might make sense to check specifics of your volume type. Perhaps, it needs something like a read-only to support mount by multiple pods.

[nitesh-mykaarma]

@yorugac no if I tried with parallelism: 1 that pod also failed with the same issue.

[yorugac]

Hi @nitesh-mykaarma, sorry for the delay; it's been a holiday season. IIUC, you mention 2 errors at once. Additionally, your script appears to be written to the volume with smth like ArgoCD Workflows. You didn't mention this explicitly, so feel free to correct me if I'm wrong.

Assuming the above, there are several possibilities according to your description:

• The error exec /usr/bin/k6: exec format error actually indicates a problem with architecture and / or Docker image. Regardless of the state of the volume, k6 binary should at least start. Given that your TestRun uses a custom image for runners, it makes sense to confirm if that image can actually run on your cluster.
• The error msg="The moduleSpecifier \"/test/authentication-utils.js\" couldn't be found on local disk. implies that either the file was not written successfully by ArgoCD Workflow or there is a misconfiguration with paths. So it makes sense to check the following:
  • create a separate debug Pod to confirm that Volume does contain the script. It should have the following configuration:

        volumes:
          - name: test-volume  
            volumeSource:
              persistentVolumeClaim:
                claimName: argo-pvc
                readOnly: false
        volumeMounts:
          - mountPath: /test
            name: test-volume

    and ls /test should contain authentication-utils.js.

Could you please check the above? Perhaps, it'd allow to see what the root issue is here.