Troubleshooting Guide

When reporting problems, please provide a copy of all relevant configuration files and the complete log file. Select [File|Save Log...] from the fcmconsole main menu to save the fcmdriver log messages to a file. Make sure the [Verbose?] option is selected before starting the driver.

See the [Diagnostics] menu for other options.

Startup errors

(E) FCM: shmget(1464): File exists

Symptom

Error message in log window "shmget(1464): File exists"

Possible cause

Another FCMDRIVER process is already running

Solution

Make sure that only one instance of the FCMDRIVER is running on the current host.

Possible cause

A previous instance of the driver has crashed

Solution

One of the following

  • Select [Diagnostics|Remove Shared Memory] from the fcmconsole main menu
  • -or- run ipcrm -M 0x1464 at the shell prompt

More information

This is not a fatal error. FCMDRIVER only uses this shared memory segment to give external processes direct access to model data without going through FLCOMMS or the remote control protocol.

If this error occurs, the fcmconsole Monitor pane, the Record... facility, and a few other features will not work, but other than that the driver will still function properly.

(E) FCM: shmget(1464): Invalid argument

Symptom

Error message in log window "shmget(1464): Invalid argument"

Possible cause

The model data is larger than the operating system limit for the maximum size of a shared memory segment.

Solution

(On Linux systems): As root, run:

/sbin/sysctl -w kernel.shmmax=80000000

To make this setting persist across reboots, add the following to the file /etc/sysctl.conf:

kernel.shmmax = 80000000

Symptom

Error dialog when selecting monitor pane: "shmget(1464): Invalid argument", following earlier error in log window "(E) FCM: shmget(1464): file exists"

Possible cause

Another FCMDRIVER process is already running, or a previous instance of the driver has crashed. See earlier for details.

Solution

See description of earlier error.

More information

See earlier for discussion.

(E) FLCOMMS: Data block locked

Symptom

Error message in log window "Data block locked"

Possible cause

Another FCMDRIVER process is already running

Solution

Make sure that only one instance of the FCMDRIVER is running on the current host.

Possible cause

A previous instance of the driver has crashed

Solution

One of the following:

  • Select [Diagnostics|Clear FLCOMMS] from the fcmconsole main menu
  • -or-Run flcomms -c at the shell prompt.

(E) FCM: Method mismatch

Symptom

Error message in log window similar to:

Method CNAME::MNAME mismatch (xxxxxxxx vs. xxxxxxxx)

Probable cause

Components library mismatch. The .fcm file was generated against a different version of the components library than the one located at runtime

Solution

Use the version of the components library belonging to the same version of FLIGHTLAB with which the model was built.

More information

Select [Diagnostics|FCM Information] from the fcmconsole main menu to display the location and version of the components library in use.

This may be overridden by setting the $FL_FCM_COMPONENTS environment variable.

(E) model: cannnot find lookup_method_proc (undefined symbol):

Symptom

Error message

(E) model: (null): cannot find lookup_method_proc (/opt/heliflight/bin/fcmdriver: undefined symbol: lookup_method_proc)

Probable cause

Components library flcomp.so not specified or missing.

Solution

One of the following:

  • Set the environment variable FL_FCM_COMPONENTS before launching the driver
  • Specify components=/path/to/flcomp.so in the [MODEL] section of the fcmdriver configuration file
  • From fcmconsole, select the components library from the main options screen.

By default, the components library may be found in /opt/heliflight/lib/flcomp.so or in $FL_DIR/lib/$FL_MACHTYPE/flcomp.so. In some installations, there may be multiple versions of the components library installed in /opt/heliflight/lib

(E) Wrong ELF class: ELFCLASS64

Symptom

Error message "flcomp.so: wrong ELF class: ELFCLASS64" or "flcomp.so: wrong ELF class: ELFCLASS32"

Probable cause

ABI mismatch. Using a 32-bit version of the components library with a 64-bit executable or vice-versa.

Solution

Make sure that the selected FL_FCM_COMPONENTS is for the same architecture as the main executable.

More information

Note that linux i386 binaries will run on x86_64 computers (though not vice-versa). However, an x86_64 shared library cannot be used from an i386 binary, even on an x86_64 computer.

Missing libgfortran.so or libg2c.so

Symptom

Error message at startup:

flcomp.so: libgfortran.so.0: cannot open shared object file: No such file or directory
# or:
flcomp.so: libg2c.so.0: cannot open shared object file: No such file or directory

Probable cause

The FORTRAN system runtime package is not installed.

Solution

Install the FORTRAN runtime libraries. See http://www.flightlab.com/support/general/flmachtype.html for instructions.

(E) flcomp.so: cannot restore segment prot after reloc

Symptom

Error message flcomp.so: flcomp.so: cannot restore segment prot after reloc: Permission denied

Probable cause

This error condition is caused by SELinux.

Solution

Our usual recommendation is to disable SELinux.

The following shell command may also fix the problem:

sudo chcon -t texrel_shlib_t /path/to/flcomp.so

If neither of those are possible, ART can provide a version of the components library that does not trigger this condition. However, there is typically a 5-10% performance penalty with this version.

(E) FCM: Variable list @DYNAMICS not found

Symptom

(Older fcmconsole versions) driver complains about missing variable lists @DYNAMICS, @INSTRUMENT, or @ENGINE, and the [Enable PWS?] option is checked.

Probable cause

Model not configured to work with vis-opengl.

Solution

Deselect the [Enable PWS?] option.

More information

This is not a fatal error.

Network communications

If you are using netflc to synchronize data across multiple hosts, the following shell commands are useful:

  • netflc -v to see what netflc is doing.
  • flcomms -l to see if it's working.
  • netflc -k will instruct all remote netflc processes to shut down.
  • flcomms -c will clear locked data blocks on the local host

(E) netflc: No such device / Network is unreachable

Symptom

Error message in log window "netflc: No such device" and/or "Network is unreachable".

Probable cause

This can occur when there is no default network route.

Solution

See Multicast routing for details.

More information

To test, run netstat -r from the shell prompt. The output should contain something similar to:

224.0.0.0       *               240.0.0.0       U        40 0          0 eth0
default         gw              0.0.0.0         UG       40 0          0 eth0

If the 'default' line is missing, then broadcast UDP will not work. If both the 'default' and '224.0.0.0' lines are missing, then multicast UDP will not work either.

Multicast routing

You may need to run the following to configure multicast routing:

Under Linux distributions with iputils2, run:

ip route add 224.0.0.0/4 dev eth0

For older Linuxes and other Unix-like operating systems, use:

route add -net 224.0.0.0 netmask 240.0.0.0 gateway default dev eth0

Network communication with multiple LANs

By default, NETFLC traffic is sent over UDP multicast on an administratively scoped multicast group. If all hosts are on a single IP local area network, communication should work automatically.

However, if multiple LANs are involved, further configuration is required.

Option 1: Peer-to-peer communications

Use this option if the driver only needs to communicate with a single external host.

Suppose the driver is running on the host with IP addresss 192.168.0.1, and netflc is running on the host with IP address 192.168.0.2.

In the fcmdriver configuration file, use:

[NETWORK]
address = 192.168.0.2

On the remote host, run:

netflc -b -a 192.168.0.1

Option 2: Enable a default route for multicast traffic

Use this option if multiple hosts need to communicate using NETFLC, all of the hosts are on a common LAN, but some hosts have multiple ethernet cards.

On each host with multiple ethernet interfaces, use 'ifconfig' to discover the list of interfaces, then run:

/sbin/route add -net 224.0.0.0/8 ethN

where ethN is the interface of the ethernet card attached to the desired network. Repeat this process on each host with multiple ethernet cards.

You must have root permissions to change the routing table.

Option 3: Use UDP broadcast

Use this option if the driver host has multiple ethernet cards, and the driver needs to communicate with multiple hosts, all of which are on a common LAN.

In the fcmdriver configuration file, use:

[NETWORK]
address = 192.168.255.255

where 192.168.255.255 is the broadcast address of the desired interface. Use ifconfig to discover the list of network interfaces and the associated broadcast address.

Note: this option might not work.

Option 4: Set up multicast routing

If you know how to do this, please let me know. I haven't been able to figure it out ...

Selectively disabling input bocks

When the simulation is in 'Run' mode, the driver reads each input block from FLCOMMS and writes it to the math model at the beginning of each frame.

The default list of input blocks is stored in the FCM file, but may be overridden by specifying an 'inputs=' parameter in the [MODEL] section of the fcmdriver configuration file. This is a space-separated list of data block names, each prefixed with an @ sign. For example:

[MODEL]
inputs  = @FCSIN @ENVIRONMENT

To disable all FLCOMMS inputs, leave this setting empty:

[MODEL]
inputs  =

For troubleshooting, it may be useful to initially disable all input blocks then selectively reenable them one at a time. Stop and restart the driver after changing the configuration file.

You can also disable the pilot inputs by selecting 'Pilot inputs: None' in the fcmconsole options pane. You must stop and restart the driver for this setting to take effect as well.

To find out the default list of input blocks, select [Diagnostics|FCM Information]; the ICD-Inputs: line (near the end) shows the default list stored in the FCM file.

Joystick issues

(E) /dev/input/js0: Permission denied

Symptom

FCMDRIVER reports "Permission denied" when trying to open the joystick.

Probable cause

The device file has the wrong permissions.

Solution

Manually change permissions to allow all users to read and write the device.

sudo chmod 666 /dev/input/js0

More information

On modern Linux distributions the permissions should be automatically set to the correct value when the device is detected. This is controlled by the udev subsystem, details are beyond the scope of this manual.

(E) /dev/input/js0: No such file or directory

Symptom

FCMDRIVER reports "No such file or directory" when trying to open the joystick.

Possible cause

This will happen if a joystick is not plugged in.

This may also happen if the Linux kernel does not have joystick support; see next section for details.

Solution

If a joystick is not available, edit the configuration file and specify:

[MAIN]
pilotin = none

When running from fcmconsole, ensure that the 'Use joystick?' configuration option is deselected.

Joystick not recognized on RHEL 6 or CentOS 6

Description

Due to a regression in Red Hat Enterprise Linux, RHEL (and CentOS version 6) are missing kernel support for joystick devices.

Symptom

Running lsusb shows that a joystick is present, but file /dev/input/js0 is not present.

Probable cause

Joystick device support is not enabled in the Red Hat Enterprise Linux 6 kernel by default.

Solution

Kernel joystick module may be installed from ELRepo, as described here: https://access.redhat.com/knowledge/solutions/64114