Troubleshooting Guide
If you encountered any problems with the installation of BigBlueButton, this section covers how to resolve many of the common issues.
If you have not already done so, read through the getting help section.
Introduction
Start here: run sudo bbb-conf --check
We've built in a BigBlueButton configuration utility, called bbb-conf, to help you configure your BigBlueButton server and troubleshoot your setup if something doesn't work right.
If you think something isn't working correctly, the first step is enter the following command.
$ sudo bbb-conf --check
This will check your setup to ensure the correct processes are running, the BigBlueButton components have correctly started, and look for common configuration problems that might prevent BigBlueButton from working properly.
If you see text after the line ** Potential problems described below **, then it may be warnings (which you can ignore if you've change settings) or errors with the setup.
Recording
Recording not processing after upgrading
If after updating from BigBlueButton 2.0 to BigBlueButton 2.2 your recordings are not processing, and if you are seeing Permission denied errors in /var/log/bigbluebutton/bbb-rap-worker.log
I, [2019-06-07T14:26:09.034878 #14808] INFO -- : /usr/lib/ruby/2.5.0/logger.rb:754:in `initialize': Permission denied @ rb_sysopen - /var/log/bigbluebutton/presentation/process-02feca80700b3e95b877af85db972904397857a1-1559909318977.log (Errno::EACCES)
You can resolve the errors with the following command
$ sudo chown -hR bigbluebutton:bigbluebutton /var/log/bigbluebutton/presentation /var/log/bigbluebutton/screenshare
and then rebuild the recordings that had not yet processed. You can see the list of recordings with
$ bbb-record --list
and then to rebuild a recording, use sudo bbb-record --rebuild <internal_meeting_id>, as in
$ sudo bbb-record --rebuild 298b06603719217df51c5d030b6e9417cc036476-1559314745219
bbb-webrtc-sfu and mediasoup
Webcams/screen sharing aren't working
Certify that appropriate external addresses have been set for mediasoup. When installed via packages, mediasoup IPs are normally misconfigured. If installed via bbb-install, then IPv4 is generally correct, but IPv6 might be absent.
Nonetheless, we recommend double-checking the instructions in Updating mediasoup.
Configure mediasoup to use IPv6
mediasoup (bbb-webrtc-sfu) does not come with a IPv6 enabled by default when installed either via packages or bbb-install.
To configure IPv6, bbb-webrtc-sfu's override configuration file (located in /etc/bigbluebutton/bbb-webrtc-sfu/production.yml) should be used.
See Updating mediasoup for instructions and examples on how to do so.
I'm having troubles seeing webcams or screen sharing in Firefox
That's usually the symptom of a known Firefox issue where it doesn't comply with ICE-lite implementations (and mediasoup is one).
This issue can be worked around by forcing TURN usage in Firefox user agents. To achieve that, set the public.kurento.forceRelayOnFirefox configuration to true in /etc/bigbluebutton/bbb-html5.yml. For example:
public:
media:
forceRelayOnFirefox: true
How often does this Firefox issue happens?
Short (non) answer: that's difficult to measure.
Every Firefox installation is prone to the lack of ICE-lite spec compliance. However, the issue doesn't manifest itself on all Firefox installations as it is dependent on how the end user's network topology is organized. It's generally a small subset of Firefox users, but that can vary depending on the user base.
Where can I track progress on a definitive solution or better workaround?
This is a Firefox bug, so the best place to get an overview on progress and what the issue is about is Mozilla's issue.
You can also track BigBlueButton's issue for updates on additional workarounds.
Why isn't forceRelayOnFirefox enabled by default?
It's not on by default because bigbluebutton does not come with a TURN server by default, and that's what versioned-in-code setting presumes.
How do I know if mediasoup is being used?
The most direct and precise way to figure out whether mediasoup is being used is checking about:webrtc (Firefox) or chrome://webrtc-internals. For example: open one of those, share a camera. Look for the remote description (SDP); see if it contains mediasoup-client in the SDP header. If it does, you're using mediasoup.
Regardless of that: mediasoup is the default in 2.5 and should always be used unless default settings were explicitly changed.
mediasoup is the default in 2.5. Why is Kurento still around?
Because Kurento is still used for stream recording. It should be removed as a dependency as soon as this issue is addressed.
Is single-core performance still important with mediasoup?
Yes.
How can I control the number of mediasoup workers?
To control the number of mediasoup workers, bbb-webrtc-sfu's override configuration file (located in /etc/bigbluebutton/bbb-webrtc-sfu/production.yml) should be used.
There are a couple of configurations of interest here:
mediasoup.workers
This configuration controls the number of mediasoup workers intended for general use (media type agnostic, shared pool).
Accepted values are:
"auto"(default): createsceil((min(nproc,32) * 0.8) + (max(0, nproc - 32) / 2))workers;"cores": creates workers up to the host's core count (as provided by os.cpus().length);- <Number>: overrides the number of workers with a fixed value;
- The default and fallback values are
auto.
For example:
- To set the number of workers to
cores:yq w -i /etc/bigbluebutton/bbb-webrtc-sfu/production.yml mediasoup.workers "cores"
mediasoup.dedicatedMediaTypeWorkers
This configuration controls the number of mediasoup workers to be used by specific media types. If a dedicated pool is set, streams of its media type will always land on it. Otherwise, they will use the shared pool.
The configuration is an object of the following format:
mediasoup.dedicatedMediaTypeWorkers:
audio: "auto"|"cores"|<Number>
main: "auto"|"cores"|<Number>
content: "auto"|"cores"|<Number>
The semantics of auto, cores and Number are the same as in the mediasoup.workers configuration. Default values for all media types are 0 (no dedicated workers).
The media types semantics are:
audio: audio (listen only, microphone) streams;main: webcam video streams;content: screen sharing streams (audio and video).
For example:
- To set the number of dedicated audio workers to
auto:yq w -i /etc/bigbluebutton/bbb-webrtc-sfu/production.yml mediasoup.dedicatedMediaTypeWorkers.audio "auto"
Can I scale the number of streams up indefinitely with mediasoup?
No. Scalability improves a lot with mediasoup, but there are still a couple of bottlenecks that can be hit as far as far as the media stack is concerned. Namely:
- The signaling server (bbb-webrtc-sfu): it does not scale vertically indefinitely.
- The mediasoup worker balancing algorithm implemented by bbb-webrtc-sfu is still focused on multiparty meetings with a restrained number of users. If your goal is thousand-user 1-N (streaming-like) meetings, you may max out CPU usage on certain mediasoup workers even though there are other idle workers free.
bbb-webrtc-sfu fails to start with a SETSCHEDULER error
bbb-webrtc-sfu runs with CPUSchedulingPolicy=fifo. In systems without appropriate capabilities (SYS_NICE), the application will fail to start. The error can be verified in journalctl logs as 214/SETSCHEDULER.
Similar to bbb-html5, you can override this by running
mkdir /etc/systemd/system/bbb-webrtc-sfu.service.d
and creating /etc/systemd/system/bbb-webrtc-sfu.service.d/override.conf with the following contents
[Service]
CPUSchedulingPolicy=other
Nice=-10
Then do systemctl daemon-reload and restart BigBlueButton.
Kurento
WebRTC video not working with Kurento
Check the value for /proc/sys/net/ipv4/tcp_syncookies that it contains the value 1.
$ cat /proc/sys/net/ipv4/tcp_syncookies
1
If not, edit /etc/sysctl.conf and set the value for net.ipv4.tcp_syncookies to 1.
net.ipv4.tcp_syncookies = 1
Save the file and restart.
Unit kurento-media-server.service is masked
If sudo bbb-conf --check returns the warning
Restarting BigBlueButton 2.0.0-RC9 (and cleaning out all log files) ...
Stopping BigBlueButton
... cleaning log files
Starting BigBlueButton
Failed to start kurento-media-server.service: Unit kurento-media-server.service is masked.
You can unmask Kurento using the command
$ systemctl unmask kurento-media-server.service
Unable to share webcam
The default installation of BigBlueButton should work in most netowrk configurations; however, if your users ae behind a restrictive network that blocks outgoing UDP connections, they may encounter 1020 errors (media unable to reach server).
If you get reports of these errors, setup TURN server to help their browsers send WebRTC audio and video streams via TCP over port 443 to the TURN server. The TURN server will then relay the media to your BigBlueButton server.
See Configure TURN.
FreeSWITCH
Configure BigBluebutton/FreeSWITCH to support IPV6
The HTML5 client now enables users on mobile devices to connect to a BigBlueButton server. However, on some cellular networks iOS devices only receive an IPV6 address.
To enable BigBlueButton (FreeSWITCH) to accept incoming web socket connections on IPV6, the BigBlueButton server must have an IPV6 address. You also need to make the following changes to the server.
First, create the file /etc/nginx/conf.d/bigbluebutton_sip_addr_map.conf with this content:
map $remote_addr $freeswitch_addr {
"~:" [2001:db8::1];
default 192.0.2.1;
}
replacing the ip addresses 192.0.2.1 with the system's external IPV4 addresses, and replace 2001:db8::1 with the system's external IPV6 address. Next, edit the file /etc/bigbluebutton/nginx/sip.nginx to have the following:
proxy_pass https://$freeswitch_addr:7443;
Next, ensure all of the following params are present in freeswitch's sip_profiles/external-ipv6.xml:
- ws-binding
- wss-binding
- rtcp-audio-interval-msec
- rtcp-video-interval-msec
- dtmf-type
- liberal-dtmf
- enable-3pcc
If any are missing, copy them from sip_profiles/external.xml, then restart BigBlueButton (sudo bbb-conf --restart).
FreeSWITCH fails to bind to IPV4
In rare occasions after shutdown/restart, the FreeSWITCH database can get corrupted. This will cause FreeSWITCH to have problems binding to IPV4 address (you may see error 1006 when users try to connect).
To check, look in /opt/freeswitch/var/log/freeswitch/freeswitch.log for errors related to loading the database.
2018-10-25 11:05:11.444727 [ERR] switch_core_db.c:108 SQL ERR [unsupported file format]
2018-10-25 11:05:11.444737 [ERR] switch_core_db.c:223 SQL ERR [unsupported file format]
2018-10-25 11:05:11.444759 [NOTICE] sofia.c:5949 Started Profile internal-ipv6 [sofia_reg_internal-ipv6]
2018-10-25 11:05:11.444767 [CRIT] switch_core_sqldb.c:508 Failure to connect to CORE_DB sofia_reg_external!
2018-10-25 11:05:11.444772 [CRIT] sofia.c:3049 Cannot Open SQL Database [external]!
If you see these errors, clear the FreeSWITCH database (BigBlueButton doesn't use the database and FreeSWITCH will recreate it on startup).
$ sudo systemctl stop freeswitch
$ rm -rf /opt/freeswitch/var/lib/freeswitch/db/*
$ sudo systemctl start freeswitch