Search
Ctrl+K
Usability
5
Automation
3
Security
4
Infrastructure
4
On this page
Documentation for troubleshooting and debugging known issues in Shuffle.
Please check the Debugging section in the Configuration documentation
If you're running Shuffle in the non-scale mode, you can get the worker logs like this:
Bash
Editable
Response
Similarly, for any of the app logs, you can use the same command, but replace "worker" with the app name (this gives the shuffle tools logs):
Bash
Editable
Response
When debugging or reporting an issue, the first thing to check is logs from both the server (frontend, backend, database) and the runtime (Orborus, workers, and apps).
Check what's actually running:
Bash
Editable
Response
Get the last 300 lines of logs from the main services:
Bash
Editable
Response
Follow logs live while you test or reproduce the problem:
Bash
Editable
Response
To see worker and app containers currently running:
Bash
Editable
Response
To get logs from a specific worker or app container:
Bash
Editable
Response
Check pods and events in your shuffle namespace:
Bash
Editable
Response
Get logs from the core deployments:
Bash
Editable
Response
Find and inspect worker/app pods:
Bash
Editable
Response
You can ping Shuffle's health endpoints directly to see if the API is responding:
API Call
Response
No response received.
To check OpenSearch health:
Bash
Editable
Response
Bash
Editable
Response
Here's a breakdown of the most common issues people run into with Shuffle, and how to fix them.
If you open Shuffle in your browser and get a blank page or connection refused:
shuffle-frontend is running (docker compose ps or kubectl get pods -n shuffle).3001 (HTTP) or 3443 (HTTPS) is open in your firewall (ufw, iptables, AWS security groups).shuffle-backend:5001.Bash
Editable
Response
Bash
Editable
Response
If you can't create your first admin user, or get a 401 Unauthorized / 500 Internal Error:
Bash
Editable
Response
SHUFFLE_OPENSEARCH_PASSWORD in your .env matches the password OpenSearch was initialized with (OPENSEARCH_INITIAL_ADMIN_PASSWORD).Bash
Editable
Response
Bash
Editable
Response
SHUFFLE_ENCRYPTION_MODIFIER, existing credentials won't be decryptable.If you go to /apps and the page is empty, or you get errors when clicking "Download from Github":
https://github.com/shuffle/python-apps to pull apps. Check if the backend has internet access:
Bash
Editable
Response
HTTP_PROXY and HTTPS_PROXY are set in .env../shuffle-apps directory is writable by Docker.If your workflows stay in WAITING status or execution never progresses:
Bash
Editable
Response
OUTER_HOSTNAME needs to be an IP or hostname that newly created containers can reach from their network (don't use localhost or 127.0.0.1!).shuffle_swarm_executions network exists and isn't corrupted. If in doubt: delete the network and restart Orborus (it will recreate it).shuffle namespace:
Bash
Editable
Response
SHUFFLE_ORBORUS_EXECUTION_CONCURRENCY).If a workflow starts, but an app node fails immediately with FAILURE or exit code 1:
Bash
Editable
Response
ghcr.io or Docker Hub).SHUFFLE_PASS_APP_PROXY=true is set.If OpenSearch restarts in a loop, exits with code 137 (OOM), or gives permission errors:
free -m). OpenSearch is memory-hungry; we recommend at least 8 GB RAM on the host.vm.max_map_count:
Bash
Editable
Response
262144, set it:
Bash
Editable
Response
Bash
Editable
Response
Bash
Editable
Response
If Shuffle is eating up too much RAM or CPU:
Bash
Editable
Response
.env:
Bash
Editable
Response
Bash
Editable
Response
.env:
Bash
Editable
Response
If you're stuck and need help from the Shuffle team or community, you can reach out on Discord, open an issue on GitHub, or email us at support@shuffler.io.
Before sending logs, please make sure to remove any passwords, API keys, or sensitive customer data!
Here's what helps us debug your issue fastest:
2.1.1 or nightly).env or values.yamlTo save time, here's a script that grabs all the relevant info into a single .tar.gz file you can share with us:
For Docker:
Bash
Editable
Response
For Kubernetes:
Bash
Editable
Response
MFA can be enabled for your account on the settings User page of a tenant, or on your settings page. If you have lost access to your account due to this however, follow these steps:
Cloud (shuffler.io): Send an email to support@shuffler.io using the email you want MFA removed for. Onprem: It's a bit more tricky onprem, as we'll need to modify the Opensearch database. Here is how:
550e8400-e29b-41d4-a716-446655440000.Bash
Editable
Response
Bash
Editable
Response
API Call
Response
No response received.
docker restart shuffle-backend. This is to fix potential caching problems.Due to the nature of Shuffle at scale, there are bound to be network issues. As Shuffle runs in Docker, and sometimes in swarm with k8s networking, it complicates the matter even further. Here's a list of things to help with debugging networking. If all else fails; reboot the machine & docker.
sysctl -p. This allows network cards to talk to each other on the same machine.JSON
Response
"Fixes" (in order):
In certain cases there may be DNS issues, leading to hanging executions. This is in most cases due to apps not being able to find the backend in some way. That's why the best solution if possible is to use the IP as hostname for Orborus -> Backend communication.
In certain cases, you may have an issue loading apps into Shuffle. If this is the case, it most likely means you have proxy issues, and can't reach github.com, where our apps are hosted.
Here's how to manually load them into Shuffle using git.
Code
Editable
Response
Alternatively: You can go download the latest Shuffle apps in your browser, and manually extract the .zip file into the ./shuffle/shuffle-apps folder.
Code
Editable
Response
Bash
Editable
Response
Code
Editable
Response
API Call
Response
No response received.
JSON
Response
Follow Python scripts allows to massively stop all running executions of a workflow
Python
Editable
Response
Copy the script into a file called abort_running_executions.py and run it with
Code
Editable
Response
In order to work requests Python library must be installed in your Python execution env.
Set the ownership of the shuffle-database folder that the shuffle-opensearch container expects.
Bash
Editable
Response
We recommend you to know what you're doing when you delete a user! It can have unpredictable consequences
Find the ID of the user (Replace the username of your user in the query with the placeholder!)
API Call
Response
No response received.
Find and take the "_id" value of your user from the returned!
Code
Editable
Response
Delete the user
API Call
Response
No response received.
If we run into "Failed to build: Error in Docker build: Error response from daemon: client version 1.51 is too new. Maximum supported APl version is 1.43" add the following environment variable into docker-compose under the backend section.
- DOCKER_API_VERSION=1.40
If you find yourself in a situation where you have forgotten your passowrd and need a reset for your user, you can reset your lost password in your local instance by doing the following:
docker exec to get bash session into OpenSearch container docker exec -it <container_id> bash
Dump the results of users index query into users.log file
API Call
Response
No response received.
open the users.log file with less and search for the admin user. Once found, scroll down to the apikey section. this value will be the api key of the admin user.
I jumped onto another server within the same vlan as my Shuffle server but these could be ran on local host too. We will create a new user and update the user's role to admin with the Shuffle API.
(Optional step): If you have multiple tenants, Change active tenant like this and repeat for each tenant:
API Call
Response
No response received.
Create a new user
API Call
Response
No response received.
Retrieve all the users and identify the user_id of the newly created user
API Call
Response
No response received.
Assign the new user to the admin role.
API Call
Response
No response received.
Log into webui with the new user, and you should now have admin rights.
API Call
Response
No response received.
API Call
Response
No response received.
Find all tenant IDs
API Call
Response
No response received.
If you prefer a visual interface over curl commands, you can add OpenSearch Dashboards to your on-prem Shuffle deployment. This gives you a UI to browse indexes, run queries, inspect documents, and monitor index health.
Add this service to your existing docker-compose.yml, inside the services: block:
YAML
Editable
Response
Then start it:
Bash
Editable
Response
The Dashboards version must match your OpenSearch major version. Check yours with:
API Call
Response
No response received.
Open http://<your-server-ip>:5601 in your browser. Log in with your OpenSearch credentials (default: admin / the value of SHUFFLE_OPENSEARCH_PASSWORD in your .env file).
Go to Dev Tools in the left sidebar to run queries directly against OpenSearch without needing curl. All the queries from the Useful OpenSearch Queries section work here.
List all Shuffle indexes:
Code
Editable
Response
Find a user:
Code
Editable
Response
OpenSearch Dashboards exposes read/write access to your database. Do not expose port 5601 to the public internet. Access it via:
ssh -L 5601:localhost:5601 your-serverThis procedure can help you extract workflows directly from OpenSearch even if the Backend and FrontEnd are in an awkward situation.
Extract the index info from OpenSearch. NOTE: You may need to create a bind mount for the location where the workflows will be extracted to.
API Call
Response
No response received.
Script to separate all workflows
Python
Editable
Response
This script need to be run on the folder with the file workflows.json, it will create a workflows_loaded directory with all the workflows in it.
This can also be very useful to either backup a copy your work or export it from a lab to a prod instance.
If you are running Shuffle in Docker inside a VM today, it is possible to move either to an on-prem Kubernetes deployment or to Shuffle SaaS. The important part is to migrate your data in a controlled way so that existing workflows, files and configuration are not lost.
docker-compose.yml, especially:
shuffle-databaseshuffle-filesshuffle-appsImportant: Do not run the old Docker-based OpenSearch instance and the new Kubernetes OpenSearch instance against the same raw data path at the same time.
Export workflows from OpenSearch as an application-level backup:
API Call
Response
No response received.
Back up the persistent storage used by Shuffle:
Keep a copy of your current docker-compose.yml and .env file.
This is the more direct migration path.
Storage note: shuffle-database in Docker is a local volume or bind mount for OpenSearch data. Kubernetes does not reuse the Docker volume name directly. Instead, it uses Persistent Volumes / Persistent Volume Claims.
This path is possible, but it is usually handled as an export/import or support-assisted migration instead of reusing raw database files directly.
Important: Raw OpenSearch storage from Docker should not be mounted directly into SaaS.
After moving to Kubernetes or SaaS, validate the following before decommissioning the old VM:
If you lost an index due to corruption or other causes, there is no easy way to handle it. Here's a workaround we have for certain scenarios. What you'll need: access to another Shuffle instance, OR someone willing to share. Lets do an example rebuilding the environments index. This assumes opensearch is on the same server.
API Call
Response
No response received.
API Call
Response
No response received.
API Call
Response
No response received.
After narrowing down your problem to opensearch is what is consuming your system resources you need to figure out why? Might be too many indices in OpenSearch or just a java heap size problem
Bash
Editable
Response
Bash
Editable
Response
Bash
Editable
Response
If you're on the default setup for shuffle and you start to notice that your workflows are getting stuck, It might be because you're running out of CPU needed to run the workflows.
To fix this, You will have to move towards setting up shuffle for production readiness in our configuration documentation.
If you are doing this in a production server you will have to comb through the indices and delete them manually with respect to you tenants priorities, old executions and such.
You should notice a reduction in memory consumption check this by running top. Do a docker-compose down then a docker-compose up -d for good measure and you are good to go.
If the above steps do not fix the issue then this might mean its a java heap size issue, go into your docker-compose.yml file, move down till you locate the opensearch configurations and navigate to the OPENSEARCH_JAVA_OPTS settings and change them if initially they were running at 4 gb half that to 2 gb and save the file.
Code
Editable
Response
In certain cases, you may experience OpenSearch continuously restarting. PS: All of these can be spotted in the logs. There are a few reasons for this which should be checked in the following order:
vm.max_map_count=262144 setting?1000:1000 by default)?1000:1000) working?.env file?For full setup instructions for SHUFFLE_OPENSEARCH_CERTIFICATE_FILE (quick start + production hardening), see:
In certain cases, especially when you're running in swarm mode (Make sure ports: 2377, 7946 and 4789 between your machines internally), you may experience timeouts, EOFs. Or maybe, in different cases a TLS timeout error, or a similar network request issue. This is most likely due to the network configuration of your Shuffle instances not matching the server it's running on.
The main configuration is "MTUs", AKA Maximum Transmission Unit. This has to match exactly - with the both the docker network driver bridge and shuffle_swarm_executions.
Find the MTU of your preferred network interface:
Code
Editable
Response
It is usually the network interface in the second line. Get it's MTU!
To set the MTU in Docker, do it in the docker-compose, in the networking section. Say the MTU you found was 1460, then use 1460, as can be seen below.
Code
Editable
Response
Next, if you're running on swarm mode, delete the existing shuffle_swarm_executions network if it already exists. You can do that by using:
Bash
Editable
Response
This might be an essential step to enforce what we did in the last step. Shuffle things a lot of things under the hood and syncing up the right interfaces is one of them so that you don't have to worry about it.
If it requires removing dependant services, proceed to do that.
When done, restart the docker-compose. Now the issue should be automatically taken care of. If not, and you're on swarm mode, Proceed to the next step of manually setting the network MTU:
We need to make a network named the same as the environment SHUFFLE_SWARM_NETWORK_NAME for Orborus (default: shuffle_swarm_executions):
Bash
Editable
Response
If the issue still persists, Please look into changing the environment variable SHUFFLE_SWARM_BRIDGE_DEFAULT_INTERFACE. Shuffle takes care of syncing the docker0 bridge interface to the preferred interface of the container. Changing this value might help docker sync up things better. We assume that the interface name is "eth0" by default, which is the default setting.
If none of this works, Often times it's simply because of the virtualisation used by your cloud provider. For example, We have found these issues to be persistent with providers using VMware underneath, Refer to this for a fix
ARM is supported on Shuffle since 1.3.0!
In certain scenarios, permissions inside and outside a container may be different. This has a lot of causes, and we'll try to help figure them out below. Thankfully most fixes are relatively simple. To test this try to go to /admin?tab=files in Shuffle, and upload a file. If the file is uploaded and it says status "active", all is good. If it's not being uploaded, then it's most likely a permission issue.

In the docker-compose.yml file, find the "shuffle-files" volume mounted for the backend service. Simply add a ":z" on the end of it like so:
Code
Editable
Response
Then restart the docker-compose (down & up -d), and try to upload a file again.
Disable Selinux to test. This should take immediate effect (run as root).
Code
Editable
Response
After, try to upload a file again
To find the folder permissions inside the container
Bash
Editable
Response
In certain scenarios or environments, you may find the docker socket to not have the right permissions. To work around this, we've built support for the docker socket proxy, which will give the containers the same permissions. Another good reason to use the docker socket proxy is to control the docker permissions required.
To use the docker socket proxy, add the following to your docker-compose.yml as a service:
Code
Editable
Response
When done, remove the "/var/run/docker.sock" volume from the backend and orborus services in the docker-compose. These containers should route their docker traffic through this proxy. To enable the docker rerouting, add this environment variable to both of them:
Code
Editable
Response
This will route all docker traffic through the docker-socket-proxy giving you granular access to each API.
PS: Adding :z to the end of the volume may fix this issue as well.
If the server Shuffle is running on is slow, it's likely due to the same constraints of any other server. One of these are typically the culprit:
The normal reason this happens is due to too many processes running concurrently in Docker (too many containers). To look at ideal configurations, look at production readiness in our configuration documentation.
First we check CPU. This is can be done using the "top" command.
Code
Editable
Response
The typical near the top is something like this. If the CPU usage is too high (see line three '%Cpu(s): 11.0 us'" - this means 11% is used total), you've most likely not configured Shuffle to run with the appropriate amount of containers as a maximum, with bad cleanup routines (CLEANUP=true).
Code
Editable
Response
Fix: Stop docker containers and reduce the amount that are allowed to run. If everything is TOO slow, reboot the server and stop all containers when it's started back up:
Bash
Editable
Response
Next up is RAM. This is can also be done using the "top" command.
Code
Editable
Response
As with CPU, the information is near the top of your screen and looks something like this. If the RAM usage is too high (see line four 'KiB Mem : 8008956 total, 142236 free'" - this means that almost no memory is left on the device). This is a typical problem if you've enabled app log forwarding into Shuffle. To disable log forwarding, add the environment "SHUFFLE_LOGS_DISABLED=true" to Orborus, then bring it down and back up again.
Code
Editable
Response
Fix: Stop docker containers and reduce the amount that are allowed to run. If everything is TOO slow, reboot the server and stop all containers when it's started back up:
Bash
Editable
Response
Next up is disk space - can Shuffle save anything? See whether there is space on the machine in the location Shuffle is running
Code
Editable
Response
To get more space, either delete some files, clean up the Opensearch instance or add more disk space.

Download the correct app version from shuffle cloud
Once downloaded upload it on your onprem shuffle instance by dragging and dropping it on the activated app list

Once done check the server for misp images present

You should see previous existing images and the newly added apps image
The last step is to refer the target image to the source image that you uploaded
You do this by using the docker tag command see more information here (https://docs.docker.com/engine/reference/commandline/tag/) docker tag frikky/shuffle:misp_1.0.0 davvyshuffle/shuffle:MISP-e72b9e9c5b0a40753e184c8ce0ba6c2b i.e docker tag source_image:{TAG} target_image:{TAG}.

Go back to your shuffle interface and your app should run success.
If you intend on uploading the app in a remote server you could push the app image onto docker hub using the docker push command more info here(https://docs.docker.com/engine/reference/commandline/image_push/)
Sign up on docker here (https://login.docker.com/u/login/) then push the intended image into your docker hub repository, from your server's cli. You might be prompted to enter your password, do so and your image will be uploaded successfully.


From your remote server cli pull the image from your docker hub repository. for more info about docker image pull see here (https://docs.docker.com/engine/reference/commandline/pull/)

Once this is done you have to tag the existing images of this app to the working app you just downloaded from your docker hub repo.

In certain cases, Docker may not be working due to too large an amount of containers running, and Docker not being able to keep up. The cause of this is typically Orborus starting too many workflows in unison. To fix this, either reduce the amount of containers able to run, or set up swarm mode (paid).
This can be controlled by the environment variables:
Code
Editable
Response
Then manually clean up the containers:
Code
Editable
Response
PS: You may need to use "systemctl stop docker" instead of using "service".
Now restart the Shuffle stack again, and all the containers should be gone
Bash
Editable
Response

Bash
Editable
Response

Once you've ensured that all services have replicas created and are running, move on to the next step.
Check logs for the worker. First, run the below command to get a list of tasks that are running
Bash
Editable
Response

Bash
Editable
Response
If you see Orborus logs like this:
Code
Editable
Response
it usually means Docker can not choose a unique host IP automatically.
Use a host NIC IP (not a container IP) and pin it:
10.0.10.25):Bash
Editable
Response
Bash
Editable
Response
Bash
Editable
Response
You may have had problems with an app and need some help getting it fixed. Apps created in the app creator of Shuffle also do generate underlying Python code utilizing the same capabilities as if you make a Python function from scratch. Here's how to find the code for a function.
Bash
Editable
Response
Bash
Editable
Response
Python
Editable
Response
Copy everything indented under this function and sent to support@shuffler.io for further help!
If you loose your tenants/suborgs for any reason at all and you need to reinstate them then you will need to do the following;
550e8400-e29b-41d4-a716-446655440000.Bash
Editable
Response
Bash
Editable
Response
API Call
Response
No response received.
Exit out of the container
Restart docker
Code
Editable
Response
Go back to your shuffle instance and you should see the tenant in question reinstated in the tenants tab.
As the Opensearch index may fill up over time, it is important to be able to debug the the available indexes. One particular issue we have had has been that it takes >60 seconds to load apps onprem at times. Here is how to resolve them.
Get into the Opensearch container of port 9200 is not exposed by default:
Bash
Editable
Response
Get the indexes and look at them:
API Call
Response
No response received.
Find the largest items in the workflowapp index (for apps in shuffle):
API Call
Response
No response received.
Delete an index if it's too large (normal ones to delete if problems: workflowexecution, workflowqueue-shuffle, environment_stats)
API Call
Response
No response received.
If you have lost access to Shuffle, it usually due to an unforeseen disconnect to the Database during startup, leading to more Tenants being added. To fix this, your user needs to be re-added to the original Tenant
Find the Tenant and User Id of your account. They are in the UUID format 550e8400-e29b-41d4-a716-446655440000
Bash
Editable
Response
Get into the Opensearch container of port 9200 is not exposed by default:
Bash
Editable
Response
Update the USERID and ORGID, ORGNAME fields, then run this command to re-add your account to the right tenant
API Call
Response
No response received.
If you want to install a custom module like pandas (although, 'execute python' isn't made for heavy processing. We recommend making a custom python app for it).
Important: Shuffle automatically pulls and updates the Shuffle Tools image on startup, which may overwrite your custom-built image. To prevent this, set the environment variable SHUFFLE_AUTO_IMAGE_DOWNLOAD=false in your Orborus/worker configuration before building a custom image.
There are two primary ways to it:
To build this app again, I would:
Bash
Editable
Response
This may be necessary if you want to do additional modifications or exploration of an existing Index. See this Issue on how it was fixed by a Shuffle user.