Search
Ctrl+K
Usability
5
Automation
3
Security
4
Infrastructure
4
On this page
Documentation for apps. If you'd like to make an app check out this guide
Apps are the heavy lifters of workflows within Shuffle. They give access to a library of functions, and are created using OpenAPI or pure Python. Shuffle gives you access to pre-defined integrations located here.
A subset of available apps can be found at https://shuffler.io/apps.
Apps are the primary building blocks in workflows. Apps can be auto-generated from OpenAPI specifications or using Shuffle's app sdk. To enforce stability and usability, we use a versioning system to prevent sudden updates to apps.
Apps can contain multiple actions, which can take multiple variables. They are made to be able to interact with each other by using each-others' data. Apps have the ability to be in multiple environments with different data (e.g. different credentials), before passing them on.
PS: In a future iteration, focus will move to an optional hybrid execution model (e.g. use cloud resources).
An app can perform more than one task based on predefined actions. These actions are defined by the developer, and are reusable and modifiable by the user of the app. An action should (for now) be a one-to-one representation of the function to run, and usually has arguments for authentication with the target application. Actions can contain multiple arguments.
You can view an app's actions by selecting the app and clicking the Edit button.

Arguments are the variables used to perform an action. Arguments with an orange dot next to them are required, with yellow ones being optional. Arguments should have example text to to indicate the expected value. The first arguments of an app are usually related to authentication or the target URL, where we suggest using variables
You can see what parameters and action has by going to /apps, selecting an app and then the action.
PS: This only applies to onprem Going to /apps, there exists a button called "Download from Github" which by default will download apps from the directory https://github.com/frikky/shuffle-apps. You can type in your own repository along with authentication options if applicable.

When the modal opens, there are two buttons:
When you set up Shuffle for the first time, it should provide you with >100 existing Apps. These are gathered from shuffle-apps, and will grow over time. Searching for apps is done by going to /apps and writing your search term. In the example below, we searched for "TheHive", which ends in TheHive being shown.

A goal for Shuffle is to make it possible to search outside the apps you currently have. This is an open issue as of 23.05.2020, but will be worked on.
PS: Extended search can be done using the shuffler search-engine
Apps may fail at times, usually due to bad coding on the creators' side. This means that to get more information, you may be required to troubleshoot and debug to get the logs.
More about this in the app debugging section
If the app you're looking for exists, it will be available on https://shuffler.io or Github. Apps available on the Shuffle website, can further be clicked then exported or tested directly.

After you've found a public or private app on https://shuffler.io, it's possible to test it directly. The view you get access to has the fully featured app included, meaning you won't need to build a workflow to test it.
Options:

Apps can be downloaded or exported from your local instance or https://shuffler.io as long as it's either your private app, or a public one AND is OpenAPI. If you find the Download button in any part of Shuffle, that means the item is exportable.

If you have an OpenAPI specification, either exported from Shuffle or otherwise, it can be imported in the /apps view. You need to be logged in.
Supported filetypes:
The options for importing are:
Any public app can be activated, giving you access to a copy of the original app. This app is editable, meaning you can change the configuration of the app in it's entirety in your own Tenant. Activation can be done by first Finding the app, then clicking the "Activate App" in the top right corner. If successful, you should se a notification that it's been activated.
Once an app is activated, you can use it within any Workflow, and find it under /apps. If you can already see the app under the /apps view, it means the app is already enabled. You need to be logged in.
If you want an app activated in your LOCAL environment, see importing apps
All apps can be published. Published apps are available to EVERYONE using Shuffle, as long as they activate it. This means if you publish an app, it is searchable AND shareable with others. The process for Python and the App Creator are different, as can be seen below.
Python:
App Creator:
PS: To remove a public app, contact us
Shuffle can load the apps by using the "Download OpenAPI" button in the /apps view.
PS: There is nothing stopping you from deleting an app that is used by a workflow. This is a destructive action, and will make some workflows using the app unusable.
Deleting an app is done by searching for it in /apps.
Required permissions (either or):

Apps are how work is done in Shuffle. They receive a piece of data, whether JSON or string, performs some action, then returns data back to the user. Apps are mostly community-made, and we aim to support that the best way we can. We allow apps to be made as Swagger/OpenAPI specifications using the app creator OR directly with Python. Each app contains multiple actions, which again can have multiple parameters. More about apps and how they work here.
The premise behind all apps that run in Shuffle, is that they each run in an isolated Docker container, for control, security and scalability. You provide arguments to an app in a Shuffle workflow, and when the workflow is run, your app is reached in the control flow, it will be launched as a new container. Shuffle then sends the apps argument data, and the container destroyed when the app's work is completed.
The underlying design of Apps in Shuffle are based on WALKOFF with minor differences. Most of the documentation below will therefore be close to their approach.
PS: There is no way of creating a Python app easily for the cloud yet. Apps released on Github will eventually be available on https://shuffler.io too.
This page is about how to develop a new Shuffle app. Want to know how to decide whether to make something new? Checkout the process document and then come back here.
There are many prebuilt apps in Shuffle, all performing unique actions. There may however be an integration you need that doesn't exist yet. You may either make this yourself or commission this to be made through Shuffle - we're always looking to expand our repository of apps!
A normal question we get asked all the time - should I use the app creator or Python directly? Find out by answering these questions (will be expanded):
Code
Editable
Response
When you build an app in Shuffle, it goes through a validation process before building the app. This process ends in a fully working Docker Image containing your apps, pointing to the original app specification to ensure users can use it in the front-end as well.
The app creator in Shuffle is built to handle any integration for HTTP apps you may think of. It's based on OpenAPI, and will generate the app using the Python SDK in the background.
If you have an OpenAPI config already
If you want to create an app
Creating or editing an app in Shuffle is made to be as simple and fast as possible. These are a few of the main main things in the UI.
Code
Editable
Response
There are many ways to authenticate Shuffle apps. The most important thing is to understand how authentication is reflected in a Workflow. Authentication fields become "required" fields, and can be stored in the App Authentication, utilized in a workflow. These are the authentication Options found in the App Creator:
Below is an explanation for each of these.
PS: Do not add your API key(s) to the App itself. This is done during authentication
Self-explanatory - no authentication is added, meaning no new required fields.
API-key allows you to add a Header or a Query (?field=value in URL) to the HTTP request. This will additionally add a URL field to the authentication, so that both the authentication and URL can be encrypted and re-used.
Normally used for keys like: "X-API-KEY" or "?auth="
Additionally, you can add a Prefix. This is NOT your API key itself. Example:
Bearer Auth is a special authentication that is very close to API key. It does what the previous authentication does, but in a very specific way. If Bearer Auth is enabled, the Header that is added will ALWAYS look like this:
Code
Editable
Response
This is used widely enough that we decided to add it as it's own authentication type.
Basic Auth is like a Login. It uses a Username and Password which has to be filled in. This will then get translated into base64 in the following format: base64(username:password), and further added as a Header to the request in the following format:
Code
Editable
Response
Very widely used in older systems.
Oauth2 is a special kind of multi-step authentication, meaning Shuffle will run a request to authorize your user's access before the actual HTTP request runs. There are two main forms of it that are widely used by larger companies: Delegated and Application.
Your Redirect URI: https://shuffler.io/set_authentication
As the creator of the app, you will need to fill in three (3) fields to help Shuffle and the user with Authorization:
All of these should be retrieved from the 3rd party platform itself. The Scopes are options the user can use to Authenticate the app, and we suggest adding as many relevant ones as possible to the list, with the most popular first.
For the most well-known apps, Shuffle may have even added a "One-step signin" for an app. If you want this for your app to make it easier for users, contact us. If this is not in place, the user needs to fill in a Client ID and Client Secret themselves.
Additionally, there is a field called "Extra Authentication". This is meant to add extra REQUIRED
Let's use "GreyNoise" as an example.
GreyNoise uses the URL "https://api.greynoise.io" and authentication type of "API key" with the header "key".


If you want extra variables added to the required authentication, you can add them with the "Extra configuration items". The defined header or query will then be added to every request from the user.

Oauth2 is a special authentication mechanism, most used by major providers like Google and Microsoft, but also others who want good authentication mechanisms. Oauth2 works by primarily defining two URL's (three with refresh token URL) and scopes that can be used. One good example is microsoft graph.
As per the provider's documentation, you need to find their authorization url, token URL and Scopes that are matching what you want to do. Here's what we found from Microsoft. Please keep in mind you do NOT have to add the queries, but just the main URL. For Microsoft the user will have to change the tenant.

After filling these in, you can now safely proceed to a Workflow to use the app. When authenticating the app, you will now have to fill in the following as a user:
Code
Editable
Response

The former two will have to be found by the USER, as the point of this authentication type is to run as the user itself. This has to be documented well, either in the description of the app itself, or in the OpenAPI documentation folder on Github to make it easily available to users. An example for Microsoft app registration can be found here, which includes how to get a client ID and secret.
Actions are the meat of how apps actually work. The action part of the app creator provides the ability to control each individual endpoint very specifically.
Action List information:
Here's what it entails:

You can add a file upload parameter to POST requests

Building the app is as straight forward as clicking the "Save" button. This builds the Docker image, and makes the app available in the App and Workflow UI for your tenant to use. We recommend building often to ensure you avoid losing any progress.
PS: The first time you build an app, it may take up to a few minutes. Do NOT update the app while it's building, as your progress will may be lost.
When building is done, you may want to test the app that you've built. This can currently be done by creating or editing a workflow, before finding the app in the left sidebar as you've always used apps.
PS: We are adding functionality for testing each app directly within the app creator
You may want to change an app later. This can be done by using the /apps UI to find the app, selecting the app, then clicking the "Edit" button.

Apps using Python can do pretty much anything you can do on a computer. As an example, most utility functions of Shuffle itself are written with as functions of Python in the app "Shuffle-Tools"
Uploading apps (cloud) or Local Hotloading (onprem) are the way to go. Look into this when you have a prototype app ready.
One of the first things you have to do is select an SDK. There are three images currently in Shuffle, Alpine, Kali, and Blackarch. Alpine is your standard slim docker image. Kali allows you access to Kali tooling, and Blackarch is arch, with a kitchen sink approach to tools.
In our example, we are going to develop an app that connects to an API for Office365, pulls some log data and returns it as a JSON data structure. We first think about the 3 primary elements:
With our 3 elements in mind, you can build the script on your own. This example is Python3, but some additional work would allow any app type/language. For now, the expectation is that you would build this in Python3.
Build your Python functions with the expected arguments, and ensure they return the expected output. Once the code works in your environment, you can then integrate it into Shuffle's apps directory to integrate as an app.
Don't forget to document your app in the docs.md file in each version's directory (see below)!
The Python SDK has a couple of utility functions and data you can utilize. These are as follows:
Python
Editable
Response
Example file API usage:
Python
Editable
Response
Example cache API usage:
Code
Editable
Response
You can learn more about the caching system in general in this Youtube video:
With your Python script built, we must build a directory structure, some metadata files and modify a base app.py file to run your code from Shuffle. In your shuffle server, there is a shuffle-apps directory, typically under the top level dir where you cloned the github repo.
/Shuffle/shuffle-apps/
This directory contains a list of folders, one per app in Shuffle. The minimal directory structure for a Shuffle app is as follows:
Code
Editable
Response
There is a template app called python-playground that we will clone and edit. First make your app folder. Note: This folder name should not contain spaces.
Code
Editable
Response
The above will make your app folder and copy the template files from python-playground to your app. Upload your python code files to the /src/ directory of your app's folder.
Now that your app's .py script is uploaded under /Shuffle/python-apps/office365mgmt/src, you'll tell Shuffle how to use it.
We will modify 4 files:
api.yaml is what provides Shuffle information about your app. Here we define your app's name, description, author info, actions (functions available) and the parameters that each function takes.
This example shows the office365mgmt app settings, but feel free to explore other app's api.yaml to see other obscure options.
Code
Editable
Response
Not all apps need this step, but if yours requires a key, secret, or password, it is recommended to use a specialized "authentication" declaration in api.yaml, so those credentials are stored safely and not displayed in cleartext in your Shuffle workflow.
Here I define that my app requires authentication and a list of parameters, which is a list of arguments specified by -name. The name value can be anything you want, but note this is the name of the argument when passed to app.py and ultimately to your app.
My app expects 4 arguments that should be stored as an authentication object. When this app is run, this credential object once configured in Shuffle will be sent to the app to use.
My first argument is a string called "planType". When the app is used in a workflow, this would be populated by workflow parameters and it would get sent to the app to use (e.g. "Enterprise").
Code
Editable
Response
Actions define the functions of your app, the args it accepts and the options available (static list of values for a given arg).
Note that one value, json_data, seems to be sent even if I don't need it, so I account for it, although it is unused at this time. You could send a JSON object here. This app has 1 function (action) called run_o365poller. This app only pulls logs, so it is limited.
The app's single function takes 2 args (besides the 4 from the authentication included above for every function), which are a string json_data arg and string PollInterval. The PollInterval string arg has a dropdown selection in GUI, listed by options, of poll_10min or poll_23hours. This is one method to give clients a limited set of options for arg values. The json_data structure, although unused, can insert any value from a prior app or input.
Code
Editable
Response
The app's logo can be encoded here using a base64 encoded representation of the image, listed after large_image, see example below, but omitted for brevity:
Code
Editable
Response
Use this file to document your app. This includes any prerequisites for using it as well as what the actions are for and how specific data objects are handled. Until we have created a nice documentation template, here's an example you can use.
This is your standard python module list that will get installed on Docker container launch
Now that your api.yaml file is modified, there is a base app.py script that is called from Shuffle when your app is run. This script will receive the configured arguments and options configured in your Shuffle workflow for the given app.
You must handle this appropriately and pass the arguments to the appropriate function. Note that we can handle this many ways. You can:
When our Office365 app is called, Shuffle assumes that app.py has an inherited class object containing a function matching the ones in our api.yaml file:
This is where many things are possible, I'll post the code below, but will describe it in detail.
Python
Editable
Response
Our app's api.yaml specifies we have a function called "run_o365poller" that accepts our 4 authentication args + 2 additional args (unused json_data and PollInterval), all string values.
In order for the app to run, app.py must have a function the same name as each function in api.yaml. We only have one above. Note that I intentionally left some commented json handling in place to show other ways to get args, you could send a json structure into the function for parsing, but should only be done for non-privileged info. The authentication data comes in as separate args.
Lastly, ensure your script returns the data you require, after mine is processed, I return a singular JSON array, with one JSON object per log entry. Another app will be written to parse this and take some action on this data, or it could simply be expanded on this app.
I have the function below, as specified in my api.yaml, that accepts the args I expect. From this script, I run my poll script, with different args based on the value of arg PollInterval.
Python
Editable
Response
Uploading and testing an app can be done in multiple ways. When the necessary files and directory structure is set up, you may continue to upload the app.
Cloud Only: To use a custom app in your instance, you can to use the Upload App API. When this has been ran, the app is available in your tenant in Shuffle. To test the app, run a workflow.
After it has been uploaded once, we suggest you run the app on a local environment, and run the docker build command instead of the full upload to make testing easier. To do this:
docker images | grep <appname>docker build . -t <imagename>Onprem Only: For self-hosted Shuffle instances, you can use the hot reloading feature to quickly add custom Python apps without restarting the server:
shuffle-apps directory in your Shuffle instance. For reference shuffle-apps.SHUFFLE_APP_HOTLOAD_FOLDER is set as ./shuffle-apps
Give the system about 20 seconds, and you will see a popup notification indicating success. This will load your custom Python app into your self-hosted Shuffle instance without requiring a restart.
Note: If you're updating an existing app, you'll need to delete the old version from any workflows that use it and re-add the new version.
When you have working apps, it is important to have them tested. Due to apps not working standalone (yet), you need to test them through Shuffle. To make a CI/CD pipeline, we suggest the following:
workflowexecution.workflow.validation.valid boolean.Keep in mind these are just suggestions, and that it may not align with your company policies for how CI/CD is done.
Just made an app and want others to get access to it? Here's how to get it in the hands of everyone:
OpenAPI: Before releasing an app, it needs to be uploaded and available in Shuffle. OpenAPI apps are by design easy to build and share - both inside and outside the Shuffle ecosystem.
Code
Editable
Response

Python: After making an app in Python, here's the steps to get it to everyone
Both of these methods makes your app highlighted on https://shuffler.io to be used by hundreds if not thousands of tenants.
As Shuffle has a lot of individual parts, debugging can be quite tricky. To get started, here's a list of the different parts, with the latter three being modular / location-independent.
| Type | Container name | Technology | Note |
|---|---|---|---|
| Frontend | shuffle-frontend | ReactJS | Cytoscape graphs & Material design |
| Backend | shuffle-backend | Golang | Rest API that connects all the different parts |
| Database | shuffle-database | Google Datastore | Has all non-volatile information. Will probably move to elastic or similar. |
| Orborus | shuffle-orborus | Golang | Runs workers in a specific environment to connect locations. Defaults to the environment "Shuffle" onprem. |
| Worker | worker-id | Golang |  Deploys Apps to run Actions defined in a workflow |
| app sdk | appname_appversion_id | Python | Used by Apps to talk to the backend |
| worker-8a666e4f-e544-440e-bf0f-4220e7cc9e25 |
Execution debugging might be the most notable issue you might explain. This is because there are a ton of reasons that it might crash. Before going into techniques to find what's going on, you'll need to understand what exactly happens when you click the big execution button.
Frontend click -> Backend verifies and deploys executions -> (based on environments) orborus deploys a new worker -> worker finds actions to execute -> your app is executed.
As previously stated, a lot can go wrong. Here's the most common issues:
Steps to follow:
This part is mean to describe how to go about finding the issue you're having with executions. In most cases, you should start from the top of the list previously described in the following way:
Find out what environment your action(s) are running under by clicking the App and seeing "Environment" dropdown. In this case (and default) is "Shuffle". Environments can be specified / changed under the path /admin

Check if the workflow executed at all by finding the execution line in the shuffle-backend container. Take note that it mentions environment "Shuffle", as found in the previous step.
Bash
Editable
Response

Check if shuffle-orborus is running
Bash
Editable
Response
Find whether it was deployed or not
Bash
Editable
Response

Check environment of running shuffle-orborus container.
Bash
Editable
Response
Expected env result where "Shuffle" corresponds to the environment

Find logs from a docker container
Bash
Editable
Response

As can be seen in the image above, is shows the exact execution order it takes. It starts by finding the parents, before executing the child process after it's finished. Take note of the specific apps being executed as well. It says "Time to execute <app_id> with app <app_name:app_version>. This indicates the app THAT WILL be executed. The following lines saying "Container <container_id> is the container created with this app.
Get the app logs
Bash
Editable
Response
As you will notice, app logs can be quite verbose (optional in a later build). In essence, if you see "RUNNING NORMAL EXECUTION" in the end, there's a 99.9% chance that it worked, otherwise some issue might have occurred.
Please notify me if you need help debugging app executions ASAP, as I've done a lot of it, but it's more tricky than the other steps.
Code
Editable
Response
Code
Editable
Response
Code
Editable
Response
Code
Editable
Response
Code
Editable
Response
Code
Editable
Response
Code
Editable
Response
Code
Editable
Response
Code
Editable
Response
Code
Editable
Response