IPMI Troubleshooting

Issue: IPMI is not reachable

If Tenantos can't establish a connection to IPMI, the power status is unavailable and an "error" label is added to the IPMI device assignment.

Move the mouse to the error label to receive the exact error message:

To solve the issue, please:

• Ensure that the correct remote agent is selected.
• Ensure that the IPMI connection details are correct.
• Ensure that the remote agent can establish a connection to IPMI.

Ensure that the remote agent can establish a connection to IPMI

Connect via SSH to your remote agent and execute following command:

ipmitool -I lanplus -H IPMI_IP -U ADMIN -P YOUR_PASSWORD chassis power status -v

If you see the same error message as in Tenantos, ensure via ping that you can reach the IPMI device:

ping IPMI_IP

If ping is not possible

If ping is not possible, it's likely a connectivity issue. Please ensure that your agent has access into your IPMI network. If IPMI is located within a private network, please read this page to find out how to include the agent into your private network.

If ping is possible

• Ensure that IPMI over LAN is enabled at the IPMI settings, like on this screenshot.
• Some IPMI versions do not accept connections via RMCP if the default password has not been changed. If you are forced to change the password after logging in into the IPMI web interface, please do so.

Issue: KVM console does not work / Error: "KVM type not supported"

Tenantos natively supports a variety of different IPMI models, but unfortunately, IPMI web interfaces and consoles are not standardized and each vendor does its own implementation. Often IPMI even differs from model to model.

If you have servers that are not supported and you would like to see them integrated, please either provide access to them (e.g. via public IP or OpenVPN) or provide access to your Tenantos installation.

• If you provide access to your installation, please include in your request the link to the server and whether it is currently in use.
• If you provide direct access to the IPMI device or a tunnel, please whitelist the IPs mentioned on the following page.

After you have prepared everything, please send us the data via support ticket.

Issue: IPMI status "Error in open session response message : insufficient resources for session"

If you encounter this issue on iDRACs, the issue can be solved by following these steps:

Menu Overview -> IDRAC SETTINGS -> User Authentication
-> Click on the user ID of your admin account -> Next
-> Check "change your password" checkbox and enter the same (or new) password
-> Click "Apply"

Credits and deeper explanation: https://stackoverflow.com/a/64297080

If the issue persists after following these steps, please verify that the IPMI encryption key is set to 0000000000000000000000000000000000000000. Additionally, ensure that the "Channel Privilege Level Limit" is configured to "Administrator". Click here to view a screenshot.

Issue: Set Session Privilege Level to ADMINISTRATOR failed: Unknown (0x80)

This error message usually indicates insufficient permissions of the IPMI account. An IPMI account with admin privileges needs to be stored in Tenantos.

Issue: HP iLO console not working due to "com.hp.ilo2.intgapp.intgapp"

If an outdated version of iLO is installed, an error message may appear when attempting to open the console, such as:

General Exception
Name: com.hp.ilo2.intgapp.intgapp
ExitException: ExitException[ 3]java.lang.StackoverflowError

To solve the issue, please update the iLO firmware. Once the firmware has been updated, the error message should no longer appear, and the console should function as expected.

Issue: Java console outputs "Connection refused" or a similar message

This message typically indicates a network issue, such as the inability to connect to the KVM port of the IPMI. Alternatively, the issue might arise if the SSL certificate has expired, causing the Java KVM viewer application to refuse the connection.

If the issue is due to an expired SSL certificate, the "Fake Date" option can be set at the IPMI device connection.

Explanation:

Some IPMIs use an expired SSL certificate for encrypted KVM communication, which can break the console connection depending on the Java viewer design of the vendor. For example, a "Connection Failed" error message on a Supermicro server might indicate an expired SSL certificate used for the viewer connection.

This option enables the simulation of a specific date in the Docker container. Please note that this functionality is experimental and its success depends on the Java viewer.

Format: yyyy-mm-dd

Issue: "cURL error 35: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure"

This error message typically indicates an SSL issue on the IPMI device's side. For HP iLO, the issue can be resolved by following these steps:

1. Go to Administration.
2. Navigate to Security.
3. Open the Encryption tab.
4. Set Enforce AES/3DES Encryption to "Enabled".

Issue: Console URL invalid or IP not permitted

NoVNC console sessions are limited to the IP address of the user who created the console for security reasons. If you open the NoVNC console and see the message "Console URL invalid or IP not permitted", it indicates that the IP address was not correctly recognized.

This error message typically appears when the console is opened via a billing integration and the end user IP isn't correctly identified by the billing integration. This might occur if a CDN is used and the original IP address isn't restored.

In general, it can be said that the cause is a server-side configuration problem. To verify this, please perform the following steps:

1. Open Tenantos and click on the "Logs" entry in the sidebar.
2. Look for the "Opened Server Console" entry and click on the info symbol on the right.
3. The "Post Data" will show the IP address that was passed to Tenantos.
4. Check the webserver access log at /var/log/nginx/access.log to find out the real IP, to cross-verify that the IP in the post data is indeed wrong: cat /var/log/nginx/access.log | grep /console/

Another common cause is a mismatch in protocol: The billing integration might be accessible via IPv6, while Tenantos isn't. When an IPv6 client opens the console, the IPv6 address is passed on, however, Tenantos is opened via IPv4. To avoid this, it is recommended to configure Tenantos to be accessible via the same protocols as the billing integration.

If the problem arises when the console is directly opened via Tenantos, it is likely that a CDN, such as Cloudflare, is causing the issue. The troubleshooting steps are the same, except that the authorized IP cannot be seen in the post data, but can be seen directly in the table, in the "User IP" column. If this is causing the issue, you may need to adjust the nginx configuration to restore the original IPs.