Troubleshooting
This page contains tips on how to solve common issues with ShinyProxy.
App start failures
Note
Before debugging your app, it’s a good idea to understand the app lifecycle in ShinyProxy.Whenever an app fails to start, ShinyProxy only shows the generic “Failed to start app” message to users. Therefore, the first step is to look at the ShinyProxy logs for more detailed messages. Look for one of these errors:
Kubernetes Pod did not start in time
If you see this error, ShinyProxy was able to create the Kubernetes pod, but the
pod didn’t start within the configured timeout. Note that this error happened
before your app (e.g. Shiny) was stared. This error is usually caused by
either an incorrect configuration of the app (e.g. incorrect container-image
)
or an issue with the Kubernetes cluster itself. ShinyProxy tries to collect as
much information as possible, therefore make sure to check the ShinyProxy logs
for additional info. In addition, it can be useful to observer the Kubernetes
pod using e.g. kubectl
.
Docker container failed to start
If you see this error, ShinyProxy wasn’t able to create the Docker container.
Note that this error happened before your app (e.g. Shiny) was stared.
This error is usually caused by either an incorrect configuration of the app
(e.g. incorrect container-image
) or an issue with the Docker daemon.
ShinyProxy tries to collect as much information as possible, therefore make sure
to check the ShinyProxy logs for additional info. In addition, it can be useful
to debug the container (while it’s being created) using the docker cli.
Swarm container did not start in time
If you see this error, ShinyProxy wasn’t able to create the Swarm service or it
failed to start within the configured timeout. Note that this error happened
before your app (e.g. Shiny) was stared. This error is usually caused by
either an incorrect configuration of the app (e.g. incorrect container-image
)
or an issue with the Docker daemon. ShinyProxy tries to collect as much
information as possible, therefore make sure to check the ShinyProxy logs for
additional info. In addition, it can be useful to debug the service (while it’s
being created) using the docker cli.
ECS container failed to start
If you see this error, ShinyProxy wasn’t able to create the ECS container or it
failed to start within the configured timeout. Note that this error happened
before your app (e.g. Shiny) was stared. This error is usually caused by
either an incorrect configuration of the app (e.g. incorrect container-image
).
ShinyProxy tries to collect as much information as possible, therefore make sure
to check the ShinyProxy logs for additional info. In addition, it can be useful
to debug the service using the AWS console.
Failed: Checking application reachable at http://localhost:40000
This error indicates that ShinyProxy was unable to reach your app (step 5 of the app creation process). At this point you can be sure that the container has started. ShinyProxy tries to reach the app multiple times up to a configured time-out. When none of these attempts succeed within the time-out, this errors is thrown. Usually this means your app has failed to start. However, if your app takes a long time to start, you might have to increase the timeout.
As a first step to debug this problem, you must check the logs of the container. You can do this directly in the container orchestrator (e.g. docker) or using the monitoring stack if this is deployed. Another option is to have ShinyProxy write the logs of containers to the filesystem. This usually gives enough information to fix the problem.
If not, the next step is to launch the docker container ‘manually’ i.e.
independently of ShinyProxy and see whether or not this succeeds. By means of
example, let’s assume the hello
app of the
openanalytics/shinyproxy-demo
image doesn’t work properly. In the default
application.yml
we can see that it’s being run using
specs:
- id: 01_hello
container-cmd: ["R", "-e", "shinyproxy::run_01_hello()"]
container-image: openanalytics/shinyproxy-demo
access-groups: [scientists, mathematicians]
In order to run it ‘manually’ (using the docker command line interface) we can do the following:
docker run -p 3838:3838 openanalytics/shinyproxy-demo R -e 'shinyproxy::run_01_hello()'
and this generates the following output:
R version 3.2.4 Revised (2016-03-16 r70336) -- "Very Secure Dishes"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> shinyproxy::run_01_hello()
Loading required package: shiny
Listening on http://0.0.0.0:3838
There are no errors in this output, but we can see exactly what happens inside the R session. This trick can come handy if you have real errors that prevent your Shiny app to come online!
If you don’t see any errors in the logs and the app works locally, there is
usually a network issue. Make sure the app uses a fixed (non-random) port and
listens on 0.0.0.0
(not 127.0.0.1
). Make sure to check
the demos, since these have this already correctly configured. In
addition, make sure the proxy.docker.internal-networking
or proxy.kubernetes.internal-networking
property is correctly configured. It should be true
when running ShinyProxy in
a container on the same backend (e.g. when using the operator) and false
otherwise. When running ShinyProxy in Docker, you must create a Docker network,
add the ShinyProxy container to this network and specify the network in the
proxy.docker.default-container-network
property. See
the examples on GitHub
for the various backends.
ShinyProxy fails to start
In certain situations, ShinyProxy might fail to start. Since ShinyProxy is a Java based application, this can sometimes generate long stack traces. However, ShinyProxy tries to log the most important error message as the last log line. For example:
ShinyProxy crashed! Exception: 'org.springframework.boot.web.server.PortInUseException', message: 'Port 8080 is already in use'
Port 8080 is already in use
ShinyProxy uses port 8080 (and 9090, see next section), if this port is already in use, you get an error message similar too:
ShinyProxy crashed! Exception: 'org.springframework.boot.web.server.PortInUseException', message: 'Port 8080 is already in use'
You can either stop the other application prior to launching ShinyProxy or use a
custom port (e.g. 9999) for ShinyProxy by adding the proxy.port
property in the
application.yml
file:
proxy:
# ...
port: 9999
Port 9090 is already in use
In addition to port 8080 for the standard web interface, ShinyProxy uses port 9090 for its management server. This server is accessed by the ShinyProxy Operator in order to check the health of the server. In addition the Prometheus metrics are served on this port (if enabled).
If port 9090 is already in use, you get an error message similar to:
ShinyProxy crashed! Exception: 'org.springframework.boot.web.server.PortInUseException', message: 'Port 9090 is already in use'
You can either stop the other application prior to launching ShinyProxy or use a
custom port (e.g. 9091) for ShinyProxy by adding the management.server.port
property in the application.yml
file:
management:
server:
port: 9091
Note: the preceding code should be placed outside the proxy:
block.
OpenID Connect (OIDC)
Invalid redirect_uri
When authenticating using OpenID, ShinyProxy first redirects the user to the
IDP. In this redirection request, ShinyProxy specifies a redirect_uri
. When
the IDP successfully authenticates the user, the IDP redirects the user to this
redirect_uri
. Therefore, this redirect_uri
should be the location at which
ShinyProxy is hosted. For security reasons, the IDP contains a list of which
redirect_uri
are valid or allowed. Therefore, if ShinyProxy specifies a wrong
redirect_uri
to the IDP, the IDP stops the authentication request.
This error message is typically caused when ShinyProxy is accessed over HTTPS
but the generated redirect_uri
contains the HTTP scheme instead of HTTPS. By
default, ShinyProxy generates a redirect_uri
using the plain HTTP scheme,
unless it detects that ShinyProxy is accessed over HTTPS. As discussed
in Security the recommended way to setup HTTPS for
ShinyProxy is by using a reverse proxy. The proxy server still accesses
ShinyProxy using the plain HTTP protocol, therefore ShinyProxy can’t detect that
it’s accessed (by the users) using HTTPS. To solve this issue, reverse proxies
can add extra headers to the request, specifying the protocol used to access
ShinyProxy. These headers are X-Forwarded-For
and X-Forwarded-Proto
.
When you are using a reverse proxy and ShinyProxy is generating a wrong
redirect_uri
,
- check that you correctly setup ShinyProxy to use the forward headers.
- check that the
server
part of this configuration is at the top level of the YAML file. - check that you specified the correct valid
redirect_uri
in the IDP configuration. - check whether your reverse proxy configuration contains the required options so that it adds the necessary headers.
- check that all proxies and/or loadbalancers in your setup support the forward-headers mechanism. For example, loadbalancers that operate at the network layer (rather than the application level) often don’t support this feature. This is, for instance, the case when using AWS NLB.
- if it still doesn’t work, enable request dumping and check whether the incoming requests to ShinyProxy contain the forward headers.
Instead of relying on the forward headers, starting with version 3.2.0 it’s possible to enforce that the generated URI uses HTTPS:
proxy:
openid:
enforce-https-redirect-uri: true
The browser reports a redirect loop when using OpenID Connect
As discussed in the previous entry, ShinyProxy and the IDP perform some redirects between each other. However, in the following scenario a redirect loop could happen:
- users goes to ShinyProxy
- ShinyProxy redirects the user to the IDP
- users successfully logins into the IDP
- IDP redirects you back to ShinyProxy
- ShinyProxy needs to validate the token provided by the IDP. However, for some reason ShinyProxy is unable to do this, and therefore thinks that the user is unauthenticated.
- therefore ShinyProxy redirects you back to the IDP
- IDP thinks you are authenticated and redirects you back to ShinyProxy
- see 5.
In ShinyProxy version 2.4.2 we prevented this from happening, instead the user
is redirect to the /auth-error
page, which explains what has gone wrong.
Therefore, it’s advised to upgrade to the latest version of ShinyProxy. If you
are still encountering a redirect loop when using the latest version, please
open an issue
at GitHub.
Authentication using OpenID doesn’t work because of Missing attribute 'email' in attributes
exception
When using OpenID, ShinyProxy tries to use the email of the user to identify the user. However, when the email of a user isn’t specified in the IDP, ShinyProxy is unable to do so and therefore the authentication fails. To solve this change the username-attribute. See the next topic to get a list of the available claims.
Tip: when using OpenID, the sub
attribute should always be available to
use. You can use this property when no other property works.
Listing all claims sent by the OpenID Provider
The OpenID authentication backend supports reading a claim from the ID token and use this as the username or groups for the users. Try the following steps, when this doesn’t work:
-
make sure the claim is added to the ID token (ShinyProxy doesn’t read the groups from the access token). Since ShinyProxy 3.1.0 it’s also possible to add the claim to the
userinfo
endpoint (but only if you specify theuserinfo-url
option). -
enable debug logging for the OpenID component by adding the following configuration:
logging: level: eu: openanalytics: containerproxy: auth: DEBUG
Note: the preceding code should be placed outside the
proxy:
block.ShinyProxy logs the contents of every claim it finds and also logs how it tries to parse the values of the roles claim.
Authentication using OpenID doesn’t work because of invalid_token_response
When using OpenID, ShinyProxy has to contact the IDP for validating the token. This exception indicates that this validation was unsuccessful, which can be caused by multiple things:
-
ShinyProxy can’t reach the IDP because the access to the network access is blocked (e.g. because of some firewall), wrong port number, the connection timed out, etc. The error is similar to:
org.springframework.security.oauth2.core.OAuth2AuthenticationException: [invalid_token_response] An error occurred while attempting to retrieve the OAuth 2.0 Access Token Response: I/O error on POST request for "https://idp.com/openid-connect/token": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
Solution: fix the network issue.
-
ShinyProxy can’t reach the IDP because it can’t resolve the hostname. The error is similar to:
org.springframework.security.oauth2.core.OAuth2AuthenticationException: [invalid_token_response] An error occurred while attempting to retrieve the OAuth 2.0 Access Token Response: I/O error on POST request for "https://non-existingidp.com/protocol/openid-connect/token": non-existingidp.com; nested exception is java.net.UnknownHostException: non-existingidp.com
Solution: ensure that the hostname is correct and that DNS is working properly.
-
The firewall of the IDP is blocking the request. The error is similar to:
org.springframework.security.oauth2.core.OAuth2AuthenticationException: [invalid_token_response] An error occurred while attempting to retrieve the OAuth 2.0 Access Token Response: 403 Forbidden: [403 Forbidden]
Tip: when your IDP is protected by CloudFlare it could be that requests from ShinyProxy are blocked based on the user agent. Try changing the user-agent by starting ShinyProxy using the following command:
java -Dhttp.agent="<user-agent>" shinyproxy.jar
-
The
client-secret
is wrong and therefore the IDP is rejecting the request. The error is similar to:org.springframework.security.oauth2.core.OAuth2AuthenticationException: [invalid_token_response] An error occurred while attempting to retrieve the OAuth 2.0 Access Token Response: 401 Unauthorized: [no body]
Solution: specify the correct
client-secret
. -
The
jwks-url
URL is wrong or giving bad responses. The error is similar to:org.springframework.security.oauth2.core.OAuth2AuthenticationException: [invalid_id_token] An error occurred while attempting to decode the Jwt: Couldn't retrieve remote JWK set: org.springframework.web.client.HttpClientErrorException$NotFound: 404 Not Found: [{"error":"RESTEASY003210: Could not find resource for full path: "}]
Solution: fix the URL or find out why the IDP is giving bad responses.
SAML 2.0
The credentials of the user expire when using SAML
By default SAML credentials are valid for a certain amount of time. This also means that those credentials have an “age’. When a user tries to login into ShinyProxy it could be possible that the IDP still has some valid (i.e. not expired) credentials for this user. In that case the IDP may provide these existing credentials to ShinyProxy. When validating these credentials, ShinyProxy checks the age of the credentials to make sure these aren’t too old. By default ShinyProxy rejects credentials that are older than two hours. However, the IDP is unaware of this restriction and thus may provide ShinyProxy credentials that are older than these two hours.
If you are experiencing this problem, the user is faced with a page that contains a stacktrace containing the following error:
org.springframework.security.saml.SAMLStatusException: Response has invalid status code urn:oasis:names:tc:SAML:2.0:status:Responder, status message is null
In the ShinyProxy log the following errors are logged:
org.springframework.security.authentication.CredentialsExpiredException: Authentication statement is too old to be used with value ...
There are two ways to solve this:
-
enable the
proxy.saml.force-authn
property. This option ensures that ShinyProxy asks the IDP to provide fresh credentials, even if the IDP believes the credentials are still valid. We’ve good experience with this option using Microsoft ADFS and Microsoft Azure AD. However, the downside of this option is that the user may have to refill their username and password, even if they’re still logged in.Note: it seems that some IDPs ignore this option, e.g. Google ignores it.
-
set the
proxy.saml.max-authentication-age
option to a value greater or equal to the maximal age of the credentials provided by the IDP. This value is specified in seconds. This has the advantage that the user isn’t forced to refill their username and password.
How do I create a keystore for signing SAML messages?
By default SAML messages sent by ShinyProxy are not signed. However, the SAML
standard makes it mandatory that an application signs the messages used to
sign-out a user. Therefore, when you use the SAML Logout feature, you must
provide ShinyProxy with a keystore. When your IDP encrypts the SAML messages,
you already have a keystore and shouldn’t create a separate one. In the other
case you do have to create a new keystore. This section describes the process of
generating such a keypair, using the keytool
utility included in Java.
The tool can be used as follows:
keytool -deststoretype pkcs12 \
-genkeypair \
-keyalg RSA \
-keysize 4096 \
-sigalg SHA256withRSA \
-validity 1460 \
-alias shinyproxy-saml \
-keypass changeme \
-keystore samlKeystore.jks
validity
: how long the certificate is valid, specified in days. In the example this is about 4 years.keyalg
: use RSA instead of DSA (note: DSA doesn’t work for ShinyProxy, EC may not work with Active Directory products)keysize
: use sensible keysizesigalg
: use a secure signature algorithmalias
: a name for your keypairkeypass
: the password used to encrypt the private keykeystore
: the filename of the keystore
The tool now asks to provide a password for the complete keystore after which it
asks for some attribute to specify in the certificate. In theory, all these
values are optional. You could specify your domain name as the value for the
first attribute (common name
) to easily identify the certificate.
If you wish you can now export the certificate and inspect it (e.g. to verify the remaining lifetime):
keytool -exportcert -alias shinyproxy-saml -keystore samlKeystore.jks -rfc -file cert.pem
openssl x509 -noout -text -in cert.pem
In order to configure ShinyProxy to use this keystore, use:
proxy:
saml:
keystore: /path/to/samlKeystore.jks
keystore-password: changeme # the password you provided when the keytool asked for it
encryption-cert-password: changeme # the password provided after the keypass parameter of the keytool
encryption-cert-name: shinyproxy-saml # the alias of the certificate
The assets of my Shiny app sometimes fail to load (HTTP error 503)
This issue is most likely caused by bug in a library used by the HTTP server used by the Shiny package. This library validates that the header size of an incoming HTTP request isn’t too big. However, it seems that this validation is not 100% correct, especially when the TCP connection is re-used for multiple HTTP requests. Fortunately, it’s possible to disable this validation, by providing a build parameter when installing Shiny.
In order to disable the check, install Shiny using the following snippet in your
Dockerfile
:
RUN R -e "install.packages(c('withr'), repos='https://cloud.r-project.org/')"
RUN R -e "withr::with_makevars(c(PKG_CPPFLAGS='-DHTTP_MAX_HEADER_SIZE=0x7fffffff'), {install.packages(c('shiny'), repos='https://cloud.r-project.org/')}, assignment = '+=')"
Note: disabling this validation doesn’t introduce any security issues, when running the Shiny app using ShinyProxy, since ShinyProxy contains the same validation. (however, since the check is disabled there is no check when running the app standalone.)
References:
- https://github.com/nodejs/http-parser/issues/426#issuecomment-386215023
- https://github.com/nodejs/http-parser/issues/451
- https://github.com/rstudio/httpuv/blob/13bcd461e4228583b29e63d4eae8ad6bb00a4ab0/src/http-parser/http_parser.c#L144-L162
Unexpected 400 bad request
The framework used by ShinyProxy (Spring) returns a 400 bad request error when the total size of incoming headers is too big. Usually this error isn’t returned the whole time, but only for specific apps, specific users or during the OpenID login procedure. The error can be solved by increasing the limit as follows:
server:
max-http-request-header-size: 128KB
Note: this should be placed outside the proxy:
block.