Troubleshooting

This page contains tips on how to solve common issues with ShinyProxy.

App start failures

Whenever an app fails to start, ShinyProxy only shows the generic “Failed to start app” message to users. Therefore, the first step is to look at the ShinyProxy logs for more detailed messages. Look for one of these errors:

Kubernetes Pod did not start in time

If you see this error, ShinyProxy was able to create the Kubernetes pod, but the pod didn’t start within the configured timeout. Note that this error happened before your app (e.g. Shiny) was stared. This error is usually caused by either an incorrect configuration of the app (e.g. incorrect container-image) or an issue with the Kubernetes cluster itself. ShinyProxy tries to collect as much information as possible, therefore make sure to check the ShinyProxy logs for additional info. In addition, it can be useful to observer the Kubernetes pod using e.g. kubectl.

Docker container failed to start

If you see this error, ShinyProxy wasn’t able to create the Docker container. Note that this error happened before your app (e.g. Shiny) was stared. This error is usually caused by either an incorrect configuration of the app (e.g. incorrect container-image) or an issue with the Docker daemon. ShinyProxy tries to collect as much information as possible, therefore make sure to check the ShinyProxy logs for additional info. In addition, it can be useful to debug the container (while it’s being created) using the docker cli.

Swarm container did not start in time

If you see this error, ShinyProxy wasn’t able to create the Swarm service or it failed to start within the configured timeout. Note that this error happened before your app (e.g. Shiny) was stared. This error is usually caused by either an incorrect configuration of the app (e.g. incorrect container-image) or an issue with the Docker daemon. ShinyProxy tries to collect as much information as possible, therefore make sure to check the ShinyProxy logs for additional info. In addition, it can be useful to debug the service (while it’s being created) using the docker cli.

ECS container failed to start

If you see this error, ShinyProxy wasn’t able to create the ECS container or it failed to start within the configured timeout. Note that this error happened before your app (e.g. Shiny) was stared. This error is usually caused by either an incorrect configuration of the app (e.g. incorrect container-image). ShinyProxy tries to collect as much information as possible, therefore make sure to check the ShinyProxy logs for additional info. In addition, it can be useful to debug the service using the AWS console.

Failed: Checking application reachable at http://localhost:40000

This error indicates that ShinyProxy was unable to reach your app (step 5 of the app creation process). At this point you can be sure that the container has started. ShinyProxy tries to reach the app multiple times up to a configured time-out. When none of these attempts succeed within the time-out, this errors is thrown. Usually this means your app has failed to start. However, if your app takes a long time to start, you might have to increase the timeout.

As a first step to debug this problem, you must check the logs of the container. You can do this directly in the container orchestrator (e.g. docker) or using the monitoring stack if this is deployed. Another option is to have ShinyProxy write the logs of containers to the filesystem. This usually gives enough information to fix the problem.

If not, the next step is to launch the docker container ‘manually’ i.e. independently of ShinyProxy and see whether or not this succeeds. By means of example, let’s assume the hello app of the openanalytics/shinyproxy-demo image doesn’t work properly. In the default application.yml we can see that it’s being run using

  specs:
  - id: 01_hello
    container-cmd: ["R", "-e", "shinyproxy::run_01_hello()"]
    container-image: openanalytics/shinyproxy-demo
    access-groups: [scientists, mathematicians]

In order to run it ‘manually’ (using the docker command line interface) we can do the following:

docker run -p 3838:3838 openanalytics/shinyproxy-demo R -e 'shinyproxy::run_01_hello()'

and this generates the following output:

R version 3.2.4 Revised (2016-03-16 r70336) -- "Very Secure Dishes"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> shinyproxy::run_01_hello()
Loading required package: shiny

Listening on http://0.0.0.0:3838

There are no errors in this output, but we can see exactly what happens inside the R session. This trick can come handy if you have real errors that prevent your Shiny app to come online!

If you don’t see any errors in the logs and the app works locally, there is usually a network issue. Make sure the app uses a fixed (non-random) port and listens on 0.0.0.0 (not 127.0.0.1). Make sure to check the demos, since these have this already correctly configured. In addition, make sure the proxy.docker.internal-networking or proxy.kubernetes.internal-networking property is correctly configured. It should be true when running ShinyProxy in a container on the same backend (e.g. when using the operator) and false otherwise. When running ShinyProxy in Docker, you must create a Docker network, add the ShinyProxy container to this network and specify the network in the proxy.docker.default-container-network property. See the examples on GitHub for the various backends.

ShinyProxy fails to start

In certain situations, ShinyProxy might fail to start. Since ShinyProxy is a Java based application, this can sometimes generate long stack traces. However, ShinyProxy tries to log the most important error message as the last log line. For example:

ShinyProxy crashed! Exception: 'org.springframework.boot.web.server.PortInUseException', message: 'Port 8080 is already in use'

Port 8080 is already in use

ShinyProxy uses port 8080 (and 9090, see next section), if this port is already in use, you get an error message similar too:

ShinyProxy crashed! Exception: 'org.springframework.boot.web.server.PortInUseException', message: 'Port 8080 is already in use'

You can either stop the other application prior to launching ShinyProxy or use a custom port (e.g. 9999) for ShinyProxy by adding the proxy.port property in the application.yml file:

proxy:
  # ...
  port: 9999

Port 9090 is already in use

In addition to port 8080 for the standard web interface, ShinyProxy uses port 9090 for its management server. This server is accessed by the ShinyProxy Operator in order to check the health of the server. In addition the Prometheus metrics are served on this port (if enabled).

If port 9090 is already in use, you get an error message similar to:

ShinyProxy crashed! Exception: 'org.springframework.boot.web.server.PortInUseException', message: 'Port 9090 is already in use'

You can either stop the other application prior to launching ShinyProxy or use a custom port (e.g. 9091) for ShinyProxy by adding the management.server.port property in the application.yml file:

management:
  server:
      port: 9091

Note: the preceding code should be placed outside the proxy: block.

OpenID Connect (OIDC)

Invalid redirect_uri

When authenticating using OpenID, ShinyProxy first redirects the user to the IDP. In this redirection request, ShinyProxy specifies a redirect_uri. When the IDP successfully authenticates the user, the IDP redirects the user to this redirect_uri. Therefore, this redirect_uri should be the location at which ShinyProxy is hosted. For security reasons, the IDP contains a list of which redirect_uri are valid or allowed. Therefore, if ShinyProxy specifies a wrong redirect_uri to the IDP, the IDP stops the authentication request.

This error message is typically caused when ShinyProxy is accessed over HTTPS but the generated redirect_uri contains the HTTP scheme instead of HTTPS. By default, ShinyProxy generates a redirect_uri using the plain HTTP scheme, unless it detects that ShinyProxy is accessed over HTTPS. As discussed in Security the recommended way to setup HTTPS for ShinyProxy is by using a reverse proxy. The proxy server still accesses ShinyProxy using the plain HTTP protocol, therefore ShinyProxy can’t detect that it’s accessed (by the users) using HTTPS. To solve this issue, reverse proxies can add extra headers to the request, specifying the protocol used to access ShinyProxy. These headers are X-Forwarded-For and X-Forwarded-Proto.

When you are using a reverse proxy and ShinyProxy is generating a wrong redirect_uri,

• check that you correctly setup ShinyProxy to use the forward headers.
• check that the server part of this configuration is at the top level of the YAML file.
• check that you specified the correct valid redirect_uri in the IDP configuration.
• check whether your reverse proxy configuration contains the required options so that it adds the necessary headers.
• check that all proxies and/or loadbalancers in your setup support the forward-headers mechanism. For example, loadbalancers that operate at the network layer (rather than the application level) often don’t support this feature. This is, for instance, the case when using AWS NLB.
• if it still doesn’t work, enable request dumping and check whether the incoming requests to ShinyProxy contain the forward headers.

Instead of relying on the forward headers, starting with version 3.2.0 it’s possible to enforce that the generated URI uses HTTPS:

proxy:
  openid:
    enforce-https-redirect-uri: true

The browser reports a redirect loop when using OpenID Connect

As discussed in the previous entry, ShinyProxy and the IDP perform some redirects between each other. However, in the following scenario a redirect loop could happen:

1. users goes to ShinyProxy
2. ShinyProxy redirects the user to the IDP
3. users successfully logins into the IDP
4. IDP redirects you back to ShinyProxy
5. ShinyProxy needs to validate the token provided by the IDP. However, for some reason ShinyProxy is unable to do this, and therefore thinks that the user is unauthenticated.
6. therefore ShinyProxy redirects you back to the IDP
7. IDP thinks you are authenticated and redirects you back to ShinyProxy
8. see 5.

In ShinyProxy version 2.4.2 we prevented this from happening, instead the user is redirect to the /auth-error page, which explains what has gone wrong. Therefore, it’s advised to upgrade to the latest version of ShinyProxy. If you are still encountering a redirect loop when using the latest version, please open an issue at GitHub.

Authentication using OpenID doesn’t work because of Missing attribute 'email' in attributes exception

When using OpenID, ShinyProxy tries to use the email of the user to identify the user. However, when the email of a user isn’t specified in the IDP, ShinyProxy is unable to do so and therefore the authentication fails. To solve this change the username-attribute. See the next topic to get a list of the available claims.

Tip: when using OpenID, the sub attribute should always be available to use. You can use this property when no other property works.

Listing all claims sent by the OpenID Provider

The OpenID authentication backend supports reading a claim from the ID token and use this as the username or groups for the users. Try the following steps, when this doesn’t work:

1. make sure the claim is added to the ID token (ShinyProxy doesn’t read the groups from the access token). Since ShinyProxy 3.1.0 it’s also possible to add the claim to the userinfo endpoint (but only if you specify the userinfo-url option).

2. enable debug logging for the OpenID component by adding the following configuration:

   logging:
     level:
       eu:
         openanalytics:
           containerproxy:
             auth: DEBUG
   

   Note: the preceding code should be placed outside the proxy: block.

   ShinyProxy logs the contents of every claim it finds and also logs how it tries to parse the values of the roles claim.

Authentication using OpenID doesn’t work because of invalid_token_response

When using OpenID, ShinyProxy has to contact the IDP for validating the token. This exception indicates that this validation was unsuccessful, which can be caused by multiple things:

• ShinyProxy can’t reach the IDP because the access to the network access is blocked (e.g. because of some firewall), wrong port number, the connection timed out, etc. The error is similar to:

  org.springframework.security.oauth2.core.OAuth2AuthenticationException: [invalid_token_response] An error occurred while attempting to retrieve the OAuth 2.0 Access Token Response: I/O error on POST request for "https://idp.com/openid-connect/token": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
  

  Solution: fix the network issue.

• ShinyProxy can’t reach the IDP because it can’t resolve the hostname. The error is similar to:

  org.springframework.security.oauth2.core.OAuth2AuthenticationException: [invalid_token_response] An error occurred while attempting to retrieve the OAuth 2.0 Access Token Response: I/O error on POST request for "https://non-existingidp.com/protocol/openid-connect/token": non-existingidp.com; nested exception is java.net.UnknownHostException: non-existingidp.com
  

  Solution: ensure that the hostname is correct and that DNS is working properly.

• The firewall of the IDP is blocking the request. The error is similar to:

  org.springframework.security.oauth2.core.OAuth2AuthenticationException: [invalid_token_response] An error occurred while attempting to retrieve the OAuth 2.0 Access Token Response: 403 Forbidden: [403 Forbidden]
  

  Tip: when your IDP is protected by CloudFlare it could be that requests from ShinyProxy are blocked based on the user agent. Try changing the user-agent by starting ShinyProxy using the following command: java -Dhttp.agent="<user-agent>" shinyproxy.jar

• The client-secret is wrong and therefore the IDP is rejecting the request. The error is similar to:

  org.springframework.security.oauth2.core.OAuth2AuthenticationException: [invalid_token_response] An error occurred while attempting to retrieve the OAuth 2.0 Access Token Response: 401 Unauthorized: [no body]
  

  Solution: specify the correct client-secret.

• The jwks-url URL is wrong or giving bad responses. The error is similar to:

  org.springframework.security.oauth2.core.OAuth2AuthenticationException: [invalid_id_token] An error occurred while attempting to decode the Jwt: Couldn't retrieve remote JWK set: org.springframework.web.client.HttpClientErrorException$NotFound: 404 Not Found: [{"error":"RESTEASY003210: Could not find resource for full path: "}]
  

  Solution: fix the URL or find out why the IDP is giving bad responses.

SAML 2.0

The credentials of the user expire when using SAML

By default SAML credentials are valid for a certain amount of time. This also means that those credentials have an “age’. When a user tries to login into ShinyProxy it could be possible that the IDP still has some valid (i.e. not expired) credentials for this user. In that case the IDP may provide these existing credentials to ShinyProxy. When validating these credentials, ShinyProxy checks the age of the credentials to make sure these aren’t too old. By default ShinyProxy rejects credentials that are older than two hours. However, the IDP is unaware of this restriction and thus may provide ShinyProxy credentials that are older than these two hours.

If you are experiencing this problem, the user is faced with a page that contains a stacktrace containing the following error:

org.springframework.security.saml.SAMLStatusException: Response has invalid status code urn:oasis:names:tc:SAML:2.0:status:Responder, status message is null

In the ShinyProxy log the following errors are logged:

org.springframework.security.authentication.CredentialsExpiredException: Authentication statement is too old to be used with value ...

There are two ways to solve this:

• enable the proxy.saml.force-authn property. This option ensures that ShinyProxy asks the IDP to provide fresh credentials, even if the IDP believes the credentials are still valid. We’ve good experience with this option using Microsoft ADFS and Microsoft Azure AD. However, the downside of this option is that the user may have to refill their username and password, even if they’re still logged in.

  Note: it seems that some IDPs ignore this option, e.g. Google ignores it.

• set the proxy.saml.max-authentication-age option to a value greater or equal to the maximal age of the credentials provided by the IDP. This value is specified in seconds. This has the advantage that the user isn’t forced to refill their username and password.

How do I create a keystore for signing SAML messages?

By default SAML messages sent by ShinyProxy are not signed. However, the SAML standard makes it mandatory that an application signs the messages used to sign-out a user. Therefore, when you use the SAML Logout feature, you must provide ShinyProxy with a keystore. When your IDP encrypts the SAML messages, you already have a keystore and shouldn’t create a separate one. In the other case you do have to create a new keystore. This section describes the process of generating such a keypair, using the keytool utility included in Java.

The tool can be used as follows:

keytool -deststoretype pkcs12 \
  -genkeypair \
  -keyalg RSA \
  -keysize 4096 \
  -sigalg SHA256withRSA \
  -validity 1460 \
  -alias shinyproxy-saml \
  -keypass changeme \
  -keystore samlKeystore.jks

• validity: how long the certificate is valid, specified in days. In the example this is about 4 years.
• keyalg: use RSA instead of DSA (note: DSA doesn’t work for ShinyProxy, EC may not work with Active Directory products)
• keysize: use sensible keysize
• sigalg: use a secure signature algorithm
• alias: a name for your keypair
• keypass: the password used to encrypt the private key
• keystore: the filename of the keystore

The tool now asks to provide a password for the complete keystore after which it asks for some attribute to specify in the certificate. In theory, all these values are optional. You could specify your domain name as the value for the first attribute (common name) to easily identify the certificate.

If you wish you can now export the certificate and inspect it (e.g. to verify the remaining lifetime):

keytool -exportcert -alias shinyproxy-saml -keystore samlKeystore.jks -rfc -file cert.pem
openssl x509 -noout -text -in cert.pem

In order to configure ShinyProxy to use this keystore, use:

proxy:
  saml:
    keystore: /path/to/samlKeystore.jks
    keystore-password: changeme # the password you provided when the keytool asked for it
    encryption-cert-password: changeme # the password provided after the keypass parameter of the keytool
    encryption-cert-name: shinyproxy-saml # the alias of the certificate

The assets of my Shiny app sometimes fail to load (HTTP error 503)

This issue is most likely caused by bug in a library used by the HTTP server used by the Shiny package. This library validates that the header size of an incoming HTTP request isn’t too big. However, it seems that this validation is not 100% correct, especially when the TCP connection is re-used for multiple HTTP requests. Fortunately, it’s possible to disable this validation, by providing a build parameter when installing Shiny.

In order to disable the check, install Shiny using the following snippet in your Dockerfile:

RUN R -e "install.packages(c('withr'), repos='https://cloud.r-project.org/')"
RUN R -e "withr::with_makevars(c(PKG_CPPFLAGS='-DHTTP_MAX_HEADER_SIZE=0x7fffffff'), {install.packages(c('shiny'), repos='https://cloud.r-project.org/')}, assignment = '+=')"

Note: disabling this validation doesn’t introduce any security issues, when running the Shiny app using ShinyProxy, since ShinyProxy contains the same validation. (however, since the check is disabled there is no check when running the app standalone.)

References:

• https://github.com/nodejs/http-parser/issues/426#issuecomment-386215023
• https://github.com/nodejs/http-parser/issues/451
• https://github.com/rstudio/httpuv/blob/13bcd461e4228583b29e63d4eae8ad6bb00a4ab0/src/http-parser/http_parser.c#L144-L162

Unexpected 400 bad request

The framework used by ShinyProxy (Spring) returns a 400 bad request error when the total size of incoming headers is too big. Usually this error isn’t returned the whole time, but only for specific apps, specific users or during the OpenID login procedure. The error can be solved by increasing the limit as follows:

server:
  max-http-request-header-size: 128KB

Note: this should be placed outside the proxy: block.