Troubleshooting an offline Spot AI Appliance

Table of Contents

 

If your Spot AI Appliance is offline, the dashboard will indicate that it is registered but not connected. When the appliance is offline, you will not be able to record live video, view historical footage, or add cameras to that appliance. We recommend the following troubleshooting steps if your appliance is offline:

1. Verify that the power cord is connected to the back of the appliance and is plugged into a working outlet
2. Verify that there is a network cable securely connected to the back of the appliance with a source of internet on the other end and that there are activity lights on both sides of the ethernet cable
3. Turn the Spot AI Appliance off with the power button, wait 30 seconds, and turn it back on again – wait 2-3 minutes to give the system time to boot back up
4. If a reboot doesn't bring the appliance back online, there's a hard drive activity light on the front of the appliance next to the power button. Check if this is lighting up or flickering. If not, contact Support.
5. Unplug the network cable from the back of the Spot AI appliance and plug it into the other port (there are 2 available network ports on the back of each Spot Business appliance)
6. Reboot the switch that the Spot AI appliance is plugged into
7. If you’re setting up this appliance for the first time, review our firewall configuration Knowledge Base article and confirm that the proper outbound ports and IPs are allowed through your firewall

You will know that the Spot Appliance has come online when you see two green check marks on the dashboard.

If you’ve tried all of the troubleshooting steps and the appliance still isn’t showing up as connected, contact Support.  Be sure to include the appliance serial number (it is on a white sticker and starts with 'sn'.

General test:

1. The appliance is unreachable through the dashboard.

• Are you able to access the remote troubleshooting page of the appliance
• If yes, verify the interface is configured correctly. If a customer uses both interfaces, make sure it's set with a different IP or non-overlapping subnet.
• Make sure the required upstream firewall ports are open KB:https://spotai.zendesk.com/hc/en-us/articles/360059872631-Upstream-Firewall-Configuration-Information
• Run ping test from the appliance to google.com are you getting a response? If yes, then the appliance is connected to the internet.
• Try to SSH the appliance; SSH can be done using SSHA (serial number of appliance ) or SSHA -o ( serial number of the appliance) if ssh works mean that the appliance is reachable remotely and there could be some back end issue which can be showing the appliance offline on the dashboard.
• If the appliance fails the above steps meaning the appliance is offline, might be physically disconnected, the location might have an internet issue, or there might be an outage at the site. At this point, I need to reach out to the customer to check what is going on at their location.

2.) Appliance is reachable but showing offline on the dashboard

• The most common issues can be containers such as Tunnel, MQTT, and Spotapp services can be restarted
• Verify if that is the case, In order to check the above container issue, SSH into the appliance as shown below

Example: ssha  (serial number )

ssha sn4e-8w33py ( now you are in the appliance )

Run command (docker ps ) it should show log as following

• Additionally, verify the Datadog logs on the appliance

If any of the services are restarting, open up a JIRA  escalation, give all information such as appliance serial number, what service is restarting, name of the organization, and location, and engineering should be able to resolve the issue or provide you with the next steps.

1. MQTT is not authorized and keeps restarting

This issue can happen because if MQTT’s authentication key expires in Google IoT core, it will enter the restart loop.

• SSH into appliance run docker ps, and you see MQTT restarting

• Check the appliance Datadog log as well

• Datadog output will look something like this:

  

Solution :

Initial things to check

While SSH'd into the appliance, you can check the MQTT log to see if it's talking or not by issuing the following command:

• docker logs -f mqtt 2>&1
• Here are some sample logs when there's an issue. In this example, the certificate was expired:

Check for network errors in the mqtt logs or DataDog:
network is unreachable

If you're able to access the appliance, ping 8.8.8.8 and check for any loss.

Checking expired certificate

To verify if the certificate is expired, you will need to identify the certificate. To do so, issue the following command.

• cat ~/vault_secrets/cert.pem

This will display the public key for the certificate. Next, we'll identify the key in the cloud:

1. Go to IoT Core
2. Click on 'my-registry'
3. Go to 'Devices'
4. Filter for the serial of the appliance
5. Go to Authentication

You can edit the certificate to see the public key and verify if it is the same as the device's. If the public keys match, you can check if the certificate is expired. If it is, there is a temporary fix that can be applied.

Temporary fix

Change the expiration to some time in the future. This is a temporary fix to bring the device back online. Once that's been changed, check MQTT logs again while SSH'd into the appliance. Once you see the following example logs, the device should come online within a couple of minutes:

After the fix

Tell Engineering of the issue in the 'support-eng' Slack chat.

If the issue is not fixed, create the JIRA ticket and add the organization, location, and serial number of the appliance facing the issue.

1. Tunnel restarting every few seconds

( Need to add a cause of tunnel restarting )

To check the tunnel status, SSH into the appliance and run docker ps as shown above.

Fix :

• To fix the above issue, open up JIRA escalation with engineering.
• In the JIRA ticket, add the organization, location, and the serial number of the appliance facing the issue.

Storage issue :

Solution: This can be because the video drive was previously not mounted.

If /dev/nvme is showing 100% storage issue, this can cause all cameras offline and appliances offline.

Solution: Please create the JIRA escalation ticket eng will be able to help with clean of the storage or may recommend replacing an appliance

How to access appliances showing offline on dashboard (SSH Hop)

Some things to consider

-Appliances might not actually be offline
-One appliance in a physical location being offline versus multiple appliances in a location being offline with one online

This scenario we'll go over with one appliance being offline in a location

• Try to see if you can connect remotely by ssh'ing into two available tunnels: ssha and ssha -o
• Check DataDog for that particular appliance to see when it stopped logging and to see what happened leading up to that time. See if data is still coming in or not
  • If you see data coming in sporadically could mean there's a duplicate IP address
  • ssha -r appliance number , if script is up to date every one second it will try to ssh into the appliance
• No data means usually means rebooting the appliance is the fix
• Is there network activity and flashing lights on the network port when the appliance is powered on?
  • Check on the appliance itself and on the switch where it is plugged in.
• If you turn off the appliance using the power button, does it turn off immediately or does it take a minute or so?
• After turning off the appliance, unplug the power cable and wait 30 seconds. When connecting it back up, does the appliance turn back on automatically?
• If you connect the network cable from the Spot appliance to a computer, does the computer get an IP address? Able to get to the internet? (Make sure that the computer has WiFi disabled for this experiment)

With multiple appliances and one being online:

• we will be basically going into an appliance that's currently online and try ssh'ing from there to the appliance showing offline
• if you have id.rsa.tt key (master key to all appliances), if you ever use this, you HAVE to remember to remove it from the appliance when finished 
  • use secure copy command in ssh excel sheet to move it to desired appliance (make sure that id.rsa.tt is in your home directory)
  • ssh into that appliance and run ls to make sure it copied over successfully
  • since it's a private key, have to change permissions to: chmod 400 id_rsa.tt
  • after try running the following ssh -i /home/pi/id_rsa.tt [e-mail]
  • If can't get in through that way try using this: Type in affected serial number -> sn4u-hdxkmn.l1.spotailan.co -> search this in dnschecker.org and see the first IP address that pops up and then insert that ip address into ssh -i /home/pi/id_rsa.tt pi@IPaddress
  • try running docker ps to see if docker is running at all
  • after you finish accessing go to your home directory and run "rm -rf id.rsa.tt" to remove the master key

If this is an enterprise appliance (e.g. snae), then check this article:

https://www.notion.so/spotai/IPMI-access-reboot-a-stubborn-Enterprise-appliance-d0271107b136429da5a4a2442eb84fd7

  

We hope this article was useful to you, please leave us a comment or feedback as it will help us improve our customer support center.