Jekyll2023-03-17T15:55:20+00:00https://www.steventwheeler.com/feed.xmlSteven WheelerA repository of things I occasionally need to look up. Hopefully, it helps others save time too.Steven WheelerSetting up MariaDB replication with mariabackup2021-08-06T14:54:00+00:002021-08-06T14:54:00+00:00https://www.steventwheeler.com/mariadb/2021/08/06/mariabackup-replication-setup<p>While working on migrating our databases to a new Galera cluster I ran into many
issues setting up replication with huge amounts of data. The databases in
question totaled roughly 900GB which made both the initial dump from the primary
database server and the import into the Galera cluster excruciatingly slow (9
days and change). Since the Galera cluster already had other databases I wasn’t
able to just copy the backup files created by <code class="language-plaintext highlighter-rouge">mariabackup</code> into
<code class="language-plaintext highlighter-rouge">/var/lib/mysql</code>, I had to come up with a method to import them without
disrupting the existing data. Luckily MariaDB provides the ability to
discard/import an InnoDB table’s tablespace.</p>
<p>First we need a backup of the primary server. Depending on how much data you
have this may take quite a while.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mariabackup <span class="nt">--backup</span> <span class="nt">--safe-slave-backup</span> <span class="nt">--innodb-file-per-table</span> <span class="nt">--target-dir</span> /var/lib/mariadb-backup/
</code></pre></div></div>
<blockquote>
<p>Note: I’ve included the <a href="https://mariadb.com/kb/en/mariabackup-options/#-safe-slave-backup"><code class="language-plaintext highlighter-rouge">--safe-slave-backup</code></a> argument here as I am actually
taking the backup from another replica and don’t want to break replication in
the process. Depending on your specific setup this option may not be required.</p>
</blockquote>
<p>Once we have data to work with it needs to be prepare for use.</p>
<p>Files generated by <code class="language-plaintext highlighter-rouge">mariabackup</code> during the <code class="language-plaintext highlighter-rouge">--backup</code> phase are not ready for
use on the server. Before they can be restored they need to be prepared. For
full backups, like the one we just took, they are not point in time consistent
since they were taken at different times. The <code class="language-plaintext highlighter-rouge">--prepare</code> phase will ready the
data to be restored to a MariaDB server.</p>
<p>The use of the <a href="https://mariadb.com/kb/en/mariabackup-options/#-export"><code class="language-plaintext highlighter-rouge">--export</code></a> parameter here is important. This argument will tell
<code class="language-plaintext highlighter-rouge">mariabackup</code> to generate the <code class="language-plaintext highlighter-rouge">.cfg</code> files for each table which are required to
import the table data using <code class="language-plaintext highlighter-rouge">ALTER TABLE ... IMPORT TABLESPACE</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mariabackup <span class="nt">--prepare</span> <span class="nt">--export</span> <span class="nt">--target-dir</span> /var/lib/mariadb-backup/
</code></pre></div></div>
<p>We will also require the <code class="language-plaintext highlighter-rouge">CREATE TABLE ...</code> statements for each table. This can
be generated with <code class="language-plaintext highlighter-rouge">mysqldump</code> like so, just make sure that the file is in the
backup directory and is named <code class="language-plaintext highlighter-rouge">schema.sql</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mysqldump <span class="nt">--no-data</span> <span class="nt">--databases</span> db1 db2 <span class="o">></span> /var/lib/mariadb-backup/schema.sql
</code></pre></div></div>
<p>Now we need to get the backed up files to the replica, I used rsync but feel
free to use whatever method is easiest for you.</p>
<p>On the replica start by importing the table schemas.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mysql < /var/lib/mariadb-backup/schema.sql
</code></pre></div></div>
<blockquote>
<p>🚨 If you are moving from a version lower than MySQL 5.6 and your tables have
temporal columns you will need to work around the
<a href="#column-precise-type-mismatch">Column precise time mismatch</a> error described
below. Check whether or not <code class="language-plaintext highlighter-rouge">mysql56_temporal_format</code> is enabled. If it is,
then disable it.</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">MariaDB [(none)]></span><span class="w"> </span>SHOW GLOBAL VARIABLES LIKE <span class="s1">'mysql56_temporal_format'</span><span class="p">;</span>
<span class="go">+-------------------------+-------+
| Variable_name | Value |
+-------------------------+-------+
| mysql56_temporal_format | ON |
+-------------------------+-------+
1 row in set (0.001 sec)
</span><span class="gp">MariaDB [(none)]></span><span class="w"> </span>SET GLOBAL <span class="nv">mysql56_temporal_format</span><span class="o">=</span>OFF<span class="p">;</span>
<span class="go">Query OK, 0 rows affected (0.000 sec)
</span></code></pre></div> </div>
<p>Make sure to re-enable it when you are done.</p>
</blockquote>
<p>To import the table data I wrote a quick bash script that will iterate over the
tables in a database and perform the import automatically. This also handles the
edge case of importing and swapping partitions which involves a few extra steps.</p>
<p><a href="/assets/article_files/2021-08-06-mariabackup-replication-setup/import_tables.sh"><code class="language-plaintext highlighter-rouge">import_tables.sh</code></a> takes the following options:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">-d|--database</code>: the database containing the tables to import.</li>
<li><code class="language-plaintext highlighter-rouge">-t|--target-dir</code>: the directory containing the exported table data.</li>
<li><code class="language-plaintext highlighter-rouge">-m|--mysql-dir</code>: the MySQL data directory to which the exported table data
will be copied. If not specified the default value is taken from <code class="language-plaintext highlighter-rouge">@@datadir</code>.</li>
<li><code class="language-plaintext highlighter-rouge">-r|--row-format</code>: The default row format of the old server. If unspecified,
then the files’ row format must match this server’s default.</li>
</ul>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>wget https://www.steventwheeler.com/assets/article_files/2021-08-06-mariabackup-replication-setup/import_tables.sh
<span class="gp">$</span><span class="w"> </span><span class="nb">chmod</span> +x import_tables.sh
<span class="gp">$</span><span class="w"> </span>./import_tables.sh <span class="nt">--target-dir</span> /var/lib/mariadb-backup <span class="nt">--database</span> db1
</code></pre></div></div>
<p>Once the tables have been imported the last step is to synchronize the Galera
nodes. This is necessary as importing the tablespace only changes the node on
which the command was run. All of the other nodes will still see the tables as
empty. There are a couple ways to do this. The first and likely safest route is
to just re-run the import script on each node while ensuring that no clients are
modifying the tables. The second is to stop MariaDB on all of the other nodes,
clear out <code class="language-plaintext highlighter-rouge">/var/lib/mysql</code>, and restart MariaDB to force them to perform SST
again. Once SST completes the node will contain all of the table data from your
original node. There are several potential pitfalls here. The biggest being that
you will have a single point of failure until the second node completes SST.
Also, if you are using rsync as your SST method the first and only node will be
inaccessible for all queries until SST completes.</p>
<h2 id="common-mysql-errors">Common MySQL Errors</h2>
<p>During the tablespace import you may run into a few different common errors due to the table formats changing between MySQL versions.</p>
<h3 id="table-flags-dont-match">Table flags don’t match</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ERROR 1808 (HY000) at line 1: Schema mismatch (Table flags don't match, server table has 0x21 and the meta-data file has 0x1; .cfg file uses ROW_FORMAT=COMPACT)
</code></pre></div></div>
<p>The default table format changed to DYNAMIC in MySQL 5.7.9, older versions used
COMPACT by default. We can work around this issue by altering our target table.
Unfortunately the failed tablespace import likely left our table in an
inconsistent state. To fix it we will make a copy, alter the copy, and then
replace the original.</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">MariaDB [db1]></span><span class="w"> </span>CREATE TABLE tbl_name_copy LIKE tbl_name<span class="p">;</span>
<span class="go">Query OK, 0 rows affected, 1 warning (0.042 sec)
</span><span class="gp">MariaDB [db1]></span><span class="w"> </span>ALTER TABLE tbl_name_copy <span class="nv">ROW_FORMAT</span><span class="o">=</span>COMPACT<span class="p">;</span>
<span class="go">Query OK, 0 rows affected (0.058 sec)
Records: 0 Duplicates: 0 Warnings: 0
</span><span class="gp">MariaDB [db1]></span><span class="w"> </span>DROP TABLE tbl_name<span class="p">;</span>
<span class="go">Query OK, 0 rows affected (0.193 sec)
</span><span class="gp">MariaDB [db1]></span><span class="w"> </span>RENAME TABLE tbl_name_copy TO tbl_name<span class="p">;</span>
<span class="go">Query OK, 0 rows affected (0.026 sec)
</span></code></pre></div></div>
<p>Alternatively, if all of your tables use a the same format which does not match
the new server’s default you can specify the <code class="language-plaintext highlighter-rouge">--row-format</code> parameter and the
script will automatically convert the tables to that format before importing the
tablespace.</p>
<h3 id="column-precise-type-mismatch">Column precise type mismatch</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ERROR 1808 (HY000) at line 1: Schema mismatch (Column created_at precise type mismatch, it's 0X8050C in the table and 0X50C in the tablespace meta file)
</code></pre></div></div>
<p>The default format for TIME, DATETIME, and TIMESTAMP changed in MySQL 5.6.4.
Importing tablesaces created with older versions will have an incompatible
format. As with the row format error we can work around this by altering our
target table. Again, the failed tablespace import likely left our table in an
inconsistent state which can be fixed by copying, altering, and replacing the
target table.</p>Steven WheelerWhile working on migrating our databases to a new Galera cluster I ran into many issues setting up replication with huge amounts of data. The databases in question totaled roughly 900GB which made both the initial dump from the primary database server and the import into the Galera cluster excruciatingly slow (9 days and change). Since the Galera cluster already had other databases I wasn’t able to just copy the backup files created by mariabackup into /var/lib/mysql, I had to come up with a method to import them without disrupting the existing data. Luckily MariaDB provides the ability to discard/import an InnoDB table’s tablespace.Debugging Asterisk Deadlocks2020-03-15T22:00:00+00:002020-03-15T22:00:00+00:00https://www.steventwheeler.com/linux,asterisk/2020/03/15/debugging-asterisk-deadlocks<p>Tracking down Asterisk deadlocks in production can prove very challenging.
Especially given the high overhead to enabling the compile time option
<a href="https://blogs.asterisk.org/2018/07/11/test_companyng-debug_threads-to-find-deadlocks/"><code class="language-plaintext highlighter-rouge">DEBUG_THREADS</code></a>.
This process has worked well for me in the past and may help save you some time
tracking down the GDB commands in the future.</p>
<h2 id="generate-a-core-dump">Generate a core dump</h2>
<p>If asterisk is still running you will want to generate a core dump. The easiest
way to do this is with the provided script.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ /var/lib/asterisk/scripts/ast_coredumper --running
</code></pre></div></div>
<p>This will generate a core file in <code class="language-plaintext highlighter-rouge">/tmp</code> by default.</p>
<h2 id="get-a-list-of-the-threads">Get a list of the threads</h2>
<p>The next step which will save us some hunting later is to get a list of all of
the threads with their thread number and LWP ID. The lock owner will be a LWP ID
but we will need the thread number in order to tell GDB to switch to the correct
thread.</p>
<p>The listing can be generated by running <code class="language-plaintext highlighter-rouge">info threads</code> on the core file.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ gdb asterisk /tmp/core-asterisk-running-2020-03-16T11-45-23-0500 --batch -ex 'info threads'
...
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
183 62: movl (%rsp), %edi
239 Thread 0x7f245f11a820 (LWP 9344) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
238 Thread 0x7f23a81ae700 (LWP 1432) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
237 Thread 0x7f23a8322700 (LWP 3273) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
236 Thread 0x7f23a8496700 (LWP 4377) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
235 Thread 0x7f23a82a6700 (LWP 4374) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
234 Thread 0x7f23a839e700 (LWP 4815) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
233 Thread 0x7f23a8512700 (LWP 4612) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
232 Thread 0x7f23a841a700 (LWP 4437) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
231 Thread 0x7f23a860a700 (LWP 4301) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
230 Thread 0x7f23a858e700 (LWP 4127) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
229 Thread 0x7f23a8702700 (LWP 2502) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
228 Thread 0x7f23a8686700 (LWP 2327) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
227 Thread 0x7f23a88f2700 (LWP 625) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
226 Thread 0x7f23a877e700 (LWP 460) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
225 Thread 0x7f23a87fa700 (LWP 323) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
224 Thread 0x7f23a8876700 (LWP 32613) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
223 Thread 0x7f23a89ea700 (LWP 30994) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
222 Thread 0x7f23a896e700 (LWP 30827) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
221 Thread 0x7f23a8ae2700 (LWP 30676) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
220 Thread 0x7f23a8a66700 (LWP 30511) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
219 Thread 0x7f23a8c56700 (LWP 28878) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
218 Thread 0x7f23a8b5e700 (LWP 28705) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
217 Thread 0x7f23a8d4e700 (LWP 26762) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
216 Thread 0x7f23a8cd2700 (LWP 26700) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
215 Thread 0x7f23a8bda700 (LWP 26574) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
214 Thread 0x7f23a8f3e700 (LWP 26445) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
213 Thread 0x7f23a8dca700 (LWP 24788) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
212 Thread 0x7f23a8e46700 (LWP 24665) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
211 Thread 0x7f23a8ec2700 (LWP 24477) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
210 Thread 0x7f23a912e700 (LWP 24348) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
209 Thread 0x7f23a8fba700 (LWP 22663) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
208 Thread 0x7f23a90b2700 (LWP 22535) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
207 Thread 0x7f23a9036700 (LWP 20880) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
206 Thread 0x7f23a91aa700 (LWP 20755) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
205 Thread 0x7f23a9226700 (LWP 20383) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
204 Thread 0x7f23a931e700 (LWP 20168) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
203 Thread 0x7f23a92a2700 (LWP 18149) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
202 Thread 0x7f23a950e700 (LWP 17941) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
201 Thread 0x7f23a939a700 (LWP 17386) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
200 Thread 0x7f23a9416700 (LWP 17263) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
199 Thread 0x7f23a9492700 (LWP 14913) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
198 Thread 0x7f23a9606700 (LWP 14698) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
197 Thread 0x7f23a958a700 (LWP 12606) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
196 Thread 0x7f23a98ee700 (LWP 12397) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
195 Thread 0x7f23a977a700 (LWP 11811) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
194 Thread 0x7f23a9ebe700 (LWP 11593) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
193 Thread 0x7f23a9ade700 (LWP 9534) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
192 Thread 0x7f23aa0ae700 (LWP 9301) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
191 Thread 0x7f23a996a700 (LWP 8731) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
190 Thread 0x7f23a9a62700 (LWP 8512) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
189 Thread 0x7f23a9c52700 (LWP 6453) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
188 Thread 0x7f23a9e42700 (LWP 6233) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
187 Thread 0x7f23a9d4a700 (LWP 4381) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
186 Thread 0x7f23e4a97700 (LWP 3806) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
185 Thread 0x7f23a9fb6700 (LWP 3351) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
184 Thread 0x7f23abfff700 (LWP 3133) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
183 Thread 0x7f245e8dd700 (LWP 874) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
182 Thread 0x7f23d9f7a700 (LWP 739) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
181 Thread 0x7f23abd92700 (LWP 550) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
180 Thread 0x7f23da072700 (LWP 32617) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
179 Thread 0x7f23abc1e700 (LWP 31208) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
178 Thread 0x7f23abba2700 (LWP 31075) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
177 Thread 0x7f23e59ed700 (LWP 30414) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
176 Thread 0x7f23d9e82700 (LWP 29263) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
175 Thread 0x7f23d93da700 (LWP 29089) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
174 Thread 0x7f23ab126700 (LWP 28962) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
173 Thread 0x7f23d9ff6700 (LWP 27295) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
172 Thread 0x7f23d9e06700 (LWP 27172) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
171 Thread 0x7f23d973e700 (LWP 26968) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
170 Thread 0x7f23d96c2700 (LWP 26091) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
169 Thread 0x7f23d9b1e700 (LWP 25162) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
168 Thread 0x7f23e41df700 (LWP 24755) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
167 Thread 0x7f23aa222700 (LWP 24206) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
166 Thread 0x7f23a99e6700 (LWP 22532) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
165 Thread 0x7f23aa12a700 (LWP 22355) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
164 Thread 0x7f23e5879700 (LWP 22227) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
163 Thread 0x7f23a9b5a700 (LWP 20567) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
162 Thread 0x7f245eacd700 (LWP 20336) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
161 Thread 0x7f23e40e7700 (LWP 20158) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
160 Thread 0x7f23d92e2700 (LWP 18396) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
159 Thread 0x7f23d91ea700 (LWP 18062) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
158 Thread 0x7f23a9bd6700 (LWP 17933) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
157 Thread 0x7f23e44c7700 (LWP 17752) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
156 Thread 0x7f23da16a700 (LWP 16152) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
155 Thread 0x7f23e42d7700 (LWP 15970) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
154 Thread 0x7f23aa032700 (LWP 15846) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
153 Thread 0x7f23d9d0e700 (LWP 14174) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
152 Thread 0x7f23e5a69700 (LWP 14038) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
151 Thread 0x7f23d9aa2700 (LWP 13828) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
150 Thread 0x7f23e4923700 (LWP 11856) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
149 Thread 0x7f23d916e700 (LWP 11678) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
148 Thread 0x7f23d935e700 (LWP 11548) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
147 Thread 0x7f23d98b2700 (LWP 11369) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
146 Thread 0x7f23e4543700 (LWP 9760) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
145 Thread 0x7f23d97ba700 (LWP 9587) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
144 Thread 0x7f23a9cce700 (LWP 9425) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
143 Thread 0x7f23abaaa700 (LWP 7735) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
142 Thread 0x7f23d9d8a700 (LWP 7595) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
141 Thread 0x7f23ab936700 (LWP 7406) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
140 Thread 0x7f23abf83700 (LWP 5800) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
139 Thread 0x7f23d9efe700 (LWP 5626) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
138 Thread 0x7f23e4353700 (LWP 5484) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
137 Thread 0x7f23e4163700 (LWP 5317) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
136 Thread 0x7f23abf07700 (LWP 3667) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
135 Thread 0x7f23d992e700 (LWP 3518) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
134 Thread 0x7f23d9456700 (LWP 3135) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
133 Thread 0x7f23aa396700 (LWP 32520) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
132 Thread 0x7f23ab8ba700 (LWP 31744) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
131 Thread 0x7f23abc9a700 (LWP 31639) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
130 Thread 0x7f245ea51700 (LWP 31633) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
129 Thread 0x7f23aa50a700 (LWP 31631) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
128 Thread 0x7f245e9d3700 (LWP 31628) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
127 Thread 0x7f23d9266700 (LWP 31627) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
126 Thread 0x7f23a9dc6700 (LWP 27413) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
125 Thread 0x7f23ab746700 (LWP 27107) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
124 Thread 0x7f23a97f6700 (LWP 27106) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
123 Thread 0x7f23d9a26700 (LWP 27100) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
122 Thread 0x7f23abe0e700 (LWP 27099) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
121 Thread 0x7f23e4733700 (LWP 25360) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
120 Thread 0x7f23abd16700 (LWP 25359) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
119 Thread 0x7f23d9836700 (LWP 25358) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
118 Thread 0x7f23e499f700 (LWP 25357) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
117 Thread 0x7f23e444b700 (LWP 25353) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
116 Thread 0x7f23d95ca700 (LWP 25350) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
115 Thread 0x7f23e43cf700 (LWP 25347) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
114 Thread 0x7f23e48a7700 (LWP 24868) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
113 Thread 0x7f23e46b7700 (LWP 23534) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
112 Thread 0x7f23d9c92700 (LWP 23141) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
111 Thread 0x7f23abe8b700 (LWP 23133) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
110 Thread 0x7f23d90f2700 (LWP 23132) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
109 Thread 0x7f23ab9b2700 (LWP 22833) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
108 Thread 0x7f23d9b9a700 (LWP 22831) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
107 Thread 0x7f23aba2e700 (LWP 22828) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
106 Thread 0x7f23a9f3a700 (LWP 22826) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
105 Thread 0x7f23ab0aa700 (LWP 22812) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
104 Thread 0x7f23aa31a700 (LWP 22811) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
103 Thread 0x7f23d9646700 (LWP 22810) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
102 Thread 0x7f23da0ee700 (LWP 22809) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
101 Thread 0x7f23d94d2700 (LWP 22808) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
100 Thread 0x7f23aa29e700 (LWP 22794) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
99 Thread 0x7f23e425b700 (LWP 22790) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
98 Thread 0x7f23ab83e700 (LWP 22757) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
97 Thread 0x7f23e4a1b700 (LWP 22756) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
96 Thread 0x7f23a96fe700 (LWP 20189) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
95 Thread 0x7f23e482b700 (LWP 20173) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
94 Thread 0x7f23d954e700 (LWP 20171) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
93 Thread 0x7f23ab6ca700 (LWP 20162) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
92 Thread 0x7f23aa1a6700 (LWP 16456) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
91 Thread 0x7f23d9c16700 (LWP 16455) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
90 Thread 0x7f23aa412700 (LWP 16454) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
89 Thread 0x7f23ab7c2700 (LWP 16046) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
88 Thread 0x7f23a9682700 (LWP 12183) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
87 Thread 0x7f23e58f5700 (LWP 812) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
86 Thread 0x7f23aa48e700 (LWP 8612) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
85 Thread 0x7f23abb26700 (LWP 8198) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
84 Thread 0x7f23e5971700 (LWP 7774) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
83 Thread 0x7f23a9872700 (LWP 7515) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
82 Thread 0x7f23d99aa700 (LWP 8673) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
81 Thread 0x7f23d9076700 (LWP 19256) 0x00000039c040e82d in read () at ../sysdeps/unix/syscall-template.S:82
80 Thread 0x7f23e57fd700 (LWP 18301) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
79 Thread 0x7f23ab1a2700 (LWP 14027) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
78 Thread 0x7f23ab21e700 (LWP 14026) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
77 Thread 0x7f23ab29a700 (LWP 14025) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
76 Thread 0x7f23ab316700 (LWP 13916) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
75 Thread 0x7f23ab5fe700 (LWP 13915) 0x00000039c040f00d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
74 Thread 0x7f23ab392700 (LWP 13914) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
73 Thread 0x7f23ab40e700 (LWP 13913) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
72 Thread 0x7f23ab48a700 (LWP 13912) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
71 Thread 0x7f23ab506700 (LWP 13910) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
70 Thread 0x7f23ab582700 (LWP 13909) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
69 Thread 0x7f23d80f6700 (LWP 13888) 0x00000039c00accbd in nanosleep () at ../sysdeps/unix/syscall-template.S:82
68 Thread 0x7f23d8172700 (LWP 13887) 0x00000039c00accbd in nanosleep () at ../sysdeps/unix/syscall-template.S:82
67 Thread 0x7f23d81ee700 (LWP 13886) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
66 Thread 0x7f23d826a700 (LWP 13883) 0x00000039c040f00d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
65 Thread 0x7f23d82e6700 (LWP 13882) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
64 Thread 0x7f23d8362700 (LWP 13881) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
63 Thread 0x7f23d83de700 (LWP 13880) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
62 Thread 0x7f23d845a700 (LWP 13879) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
61 Thread 0x7f23d84d6700 (LWP 13878) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
60 Thread 0x7f23d8552700 (LWP 13877) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
59 Thread 0x7f23d85ce700 (LWP 13876) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
58 Thread 0x7f23d864a700 (LWP 13875) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
57 Thread 0x7f23d86c6700 (LWP 13874) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
56 Thread 0x7f23d8742700 (LWP 13871) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
55 Thread 0x7f23d87be700 (LWP 13870) 0x00000039c040eb2d in accept () at ../sysdeps/unix/syscall-template.S:82
54 Thread 0x7f23d883a700 (LWP 13869) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
53 Thread 0x7f23d88b6700 (LWP 13868) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
52 Thread 0x7f23d8932700 (LWP 13867) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
51 Thread 0x7f23d89ae700 (LWP 13866) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
50 Thread 0x7f23d8a2a700 (LWP 13865) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
49 Thread 0x7f23d8aa6700 (LWP 13864) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
48 Thread 0x7f23d8b22700 (LWP 13863) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
47 Thread 0x7f23d8b9e700 (LWP 13862) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
46 Thread 0x7f23d8c1a700 (LWP 13861) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
45 Thread 0x7f23d8c96700 (LWP 13860) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
44 Thread 0x7f23d8d12700 (LWP 13859) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
43 Thread 0x7f23d8d8e700 (LWP 13858) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
42 Thread 0x7f23d8e0a700 (LWP 13857) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
41 Thread 0x7f23d8e86700 (LWP 13856) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
40 Thread 0x7f23d8f02700 (LWP 13855) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
39 Thread 0x7f23d8f7e700 (LWP 13854) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
38 Thread 0x7f23d8ffa700 (LWP 13853) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
37 Thread 0x7f23e45bf700 (LWP 13806) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
36 Thread 0x7f23e463b700 (LWP 13805) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
35 Thread 0x7f23e47af700 (LWP 13478) __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
34 Thread 0x7f23e4b8f700 (LWP 13112) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
33 Thread 0x7f23e4b13700 (LWP 11705) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
32 Thread 0x7f23e5591700 (LWP 9581) 0x00000039c00e9243 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
31 Thread 0x7f23e560d700 (LWP 9578) 0x00000039c00accbd in nanosleep () at ../sysdeps/unix/syscall-template.S:82
30 Thread 0x7f23e5689700 (LWP 9577) 0x00000039c00accbd in nanosleep () at ../sysdeps/unix/syscall-template.S:82
29 Thread 0x7f23e5705700 (LWP 9576) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
28 Thread 0x7f23e5781700 (LWP 9575) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
27 Thread 0x7f23e6dbe700 (LWP 9566) 0x00000039c00e9243 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
26 Thread 0x7f23e5be7700 (LWP 9565) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
25 Thread 0x7f23e5c63700 (LWP 9564) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
24 Thread 0x7f23e5cdf700 (LWP 9563) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
23 Thread 0x7f23e5d5b700 (LWP 9562) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
22 Thread 0x7f23e5dd7700 (LWP 9561) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
21 Thread 0x7f23e5e53700 (LWP 9560) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
20 Thread 0x7f23e5ecf700 (LWP 9559) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
19 Thread 0x7f23e5f4b700 (LWP 9558) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
18 Thread 0x7f23e5fc7700 (LWP 9557) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
17 Thread 0x7f23e6043700 (LWP 9556) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
16 Thread 0x7f23e60bf700 (LWP 9555) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
15 Thread 0x7f23e613b700 (LWP 9554) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
14 Thread 0x7f23e61b7700 (LWP 9553) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
13 Thread 0x7f245eb49700 (LWP 9467) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
12 Thread 0x7f245ebc5700 (LWP 9466) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
11 Thread 0x7f245ec41700 (LWP 9464) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
10 Thread 0x7f245ecbd700 (LWP 9463) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
9 Thread 0x7f245ed39700 (LWP 9462) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
8 Thread 0x7f245edb5700 (LWP 9461) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
7 Thread 0x7f245ee31700 (LWP 9459) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
6 Thread 0x7f245eead700 (LWP 9458) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
5 Thread 0x7f245ef29700 (LWP 9455) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
4 Thread 0x7f245efa5700 (LWP 9454) 0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
3 Thread 0x7f245f021700 (LWP 9452) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
2 Thread 0x7f245f09d700 (LWP 9451) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
* 1 Thread 0x7f245f119700 (LWP 9447) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
</code></pre></div></div>
<h2 id="find-a-thread-which-is-waiting-on-a-lock">Find a thread which is waiting on a lock</h2>
<p>This step is a bit of trial and error but in general you can look for threads in
the output above that are executing in <code class="language-plaintext highlighter-rouge">__lll_lock_wait</code>. Start by opening an
interactive GDB session.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ gdb asterisk /tmp/core-asterisk-running-2020-03-16T11-45-23-0500
</code></pre></div></div>
<p>We will choose <code class="language-plaintext highlighter-rouge">193</code> at random. Switch to the thread and print the mutex
information like so.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(gdb) thread 193
[Switching to thread 193 (Thread 0x7f23a9ade700 (LWP 9534))]#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
136 2: movl %edx, %eax
(gdb) bt
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1 0x00000039c04095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#2 0x00000039c04094d7 in __pthread_mutex_lock (mutex=0x7f24135ac7c0) at pthread_mutex_lock.c:82
#3 0x00007f241337dfc8 in handle_request_do (req=0x7f23a9adbe10, addr=0x7f24380e9328) at chan_sip.c:29373
#4 0x00007f241337eb99 in _sip_tcp_helper_thread (tcptls_session=0x7f24380e9320) at chan_sip.c:3137
#5 0x0000000000599b77 in dummy_start (data=<value optimized out>) at utils.c:1249
#6 0x00000039c0407aa1 in start_thread (arg=0x7f23a9ade700) at pthread_create.c:301
#7 0x00000039c00e8c4d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) frame 2
#2 0x00000039c04094d7 in __pthread_mutex_lock (mutex=0x7f24135ac7c0) at pthread_mutex_lock.c:82
82 LLL_MUTEX_LOCK (mutex);
(gdb) p *((pthread_mutex_t *) 0x7f24135ac7c0)
$2 = {__data = {__lock = 2, __count = 1, __owner = 3135, __nusers = 1, __kind = 1, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = "\002\000\000\000\001\000\000\000?\f\000\000\001\000\000\000\001", '\000' <repeats 22 times>, __align = 4294967298}
</code></pre></div></div>
<p>We switch to frame <code class="language-plaintext highlighter-rouge">2</code> since that’s where the mutex address is displayed. Then
print the mutex contents by dereferencing the pointer. We can see that the owner
of this mutex is LWP <code class="language-plaintext highlighter-rouge">3135</code>. The next step is to switch to that thread and
repeat the process until we find ourselves in a loop. From our previously saved
table we know that LWP ID <code class="language-plaintext highlighter-rouge">3135</code> maps to thread <code class="language-plaintext highlighter-rouge">134</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(gdb) thread 134
[Switching to thread 134 (Thread 0x7f23d9456700 (LWP 3135))]#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
136 2: movl %edx, %eax
(gdb) bt
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1 0x00000039c04095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#2 0x00000039c04094d7 in __pthread_mutex_lock (mutex=0x8d0ba0) at pthread_mutex_lock.c:82
#3 0x0000000000516c75 in ast_rdlock_contexts (c=0x0, con=0x0, context=0x7f23b022b560 "fail-call", exten=0x7f24133897cc "s", priority=1, label=0x0, callerid=0x7f23b0177c0b "monit", action=E_MATCH, found=0x0, combined_find_spawn=0)
at pbx.c:8487
#4 pbx_extension_helper (c=0x0, con=0x0, context=0x7f23b022b560 "fail-call", exten=0x7f24133897cc "s", priority=1, label=0x0, callerid=0x7f23b0177c0b "monit", action=E_MATCH, found=0x0, combined_find_spawn=0) at pbx.c:2886
#5 0x00000000005175c4 in ast_exists_extension (c=<value optimized out>, context=<value optimized out>, exten=<value optimized out>, priority=<value optimized out>, callerid=<value optimized out>) at pbx.c:4172
#6 0x00007f2413325370 in get_destination (p=<value optimized out>, oreq=0x7f23d9453e10, cc_recall_core_id=0x0) at chan_sip.c:18607
#7 0x00007f2413372099 in handle_request_options (p=0x7f23b0296fe0, req=0x7f23d9453e10, addr=0x7f243811f368, e=<value optimized out>) at chan_sip.c:25840
#8 0x00007f241337c2af in handle_incoming (p=0x7f23b0296fe0, req=0x7f23d9453e10, addr=0x7f243811f368, recount=0x7f23d94532cc, nounlock=0x7f23d94532c8) at chan_sip.c:29191
#9 0x00007f241337e0fd in handle_request_do (req=0x7f23d9453e10, addr=0x7f243811f368) at chan_sip.c:29403
#10 0x00007f241337eb99 in _sip_tcp_helper_thread (tcptls_session=0x7f243811f360) at chan_sip.c:3137
#11 0x0000000000599b77 in dummy_start (data=<value optimized out>) at utils.c:1249
#12 0x00000039c0407aa1 in start_thread (arg=0x7f23d9456700) at pthread_create.c:301
#13 0x00000039c00e8c4d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) frame 2
#2 0x00000039c04094d7 in __pthread_mutex_lock (mutex=0x8d0ba0) at pthread_mutex_lock.c:82
82 LLL_MUTEX_LOCK (mutex);
(gdb) p *((pthread_mutex_t *) 0x8d0ba0)
$2 = {__data = {__lock = 2, __count = 1, __owner = 31744, __nusers = 1, __kind = 1, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = "\002\000\000\000\001\000\000\000\000|\000\000\001\000\000\000\001", '\000' <repeats 22 times>, __align = 4294967298}
</code></pre></div></div>
<p>This output shows that this thread is blocked waiting for a mutex owned by LWP
<code class="language-plaintext highlighter-rouge">31744</code>. From our previously saved table we know that LWP ID <code class="language-plaintext highlighter-rouge">31744</code> maps to
thread <code class="language-plaintext highlighter-rouge">132</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(gdb) thread 132
[Switching to thread 132 (Thread 0x7f23ab8ba700 (LWP 31744))]#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
136 2: movl %edx, %eax
(gdb) bt
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1 0x00000039c04095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#2 0x00000039c04094d7 in __pthread_mutex_lock (mutex=0x25e6bf0) at pthread_mutex_lock.c:82
#3 0x000000000045b506 in __ao2_lock (user_data=<value optimized out>, lock_how=<value optimized out>, file=0x63d8a3 "pbx.c", func=0x641120 "ast_change_hint", line=4095, var=<value optimized out>) at astobj2.c:241
#4 0x000000000051d526 in ast_change_hint (con=0x7f24243da630, replace=1, extension=<value optimized out>, priority=608020264, label=<value optimized out>, callerid=<value optimized out>, application=<value optimized out>,
data=0x7f2454028580, datad=0x45a7c0 <ast_free_ptr>, registrar=0x7f245ddcf9b8 "pbx_config", registrar_file=0x0, registrar_line=0, lock_context=1) at pbx.c:4095
#5 add_priority (con=0x7f24243da630, replace=1, extension=<value optimized out>, priority=608020264, label=<value optimized out>, callerid=<value optimized out>, application=<value optimized out>, data=0x7f2454028580,
datad=0x45a7c0 <ast_free_ptr>, registrar=0x7f245ddcf9b8 "pbx_config", registrar_file=0x0, registrar_line=0, lock_context=1) at pbx.c:7223
#6 ast_add_extension2_lockopt (con=0x7f24243da630, replace=1, extension=<value optimized out>, priority=608020264, label=<value optimized out>, callerid=<value optimized out>, application=<value optimized out>, data=0x7f2454028580,
datad=0x45a7c0 <ast_free_ptr>, registrar=0x7f245ddcf9b8 "pbx_config", registrar_file=0x0, registrar_line=0, lock_context=1) at pbx.c:7480
#7 0x000000000051eae5 in ast_add_extension2 (context=<value optimized out>, replace=1, extension=0x7f23ab8b8b00 "test_user", priority=-1, label=0x0, callerid=0x0,
application=0x7f23ab8b8b0e "SIP/test_company-test_phone&SIP/test_company-test_user_softphone&SIP/test_company-test_phone&Custom:unavailable", data=0x7f2454028580, datad=0x45a7c0 <ast_free_ptr>, registrar=0x7f245ddcf9b8 "pbx_config") at pbx.c:7292
#8 ast_add_extension (context=<value optimized out>, replace=1, extension=0x7f23ab8b8b00 "test_user", priority=-1, label=0x0, callerid=0x0,
application=0x7f23ab8b8b0e "SIP/test_company-test_phone&SIP/test_company-test_user_softphone&SIP/test_company-test_phone&Custom:unavailable", data=0x7f2454028580, datad=0x45a7c0 <ast_free_ptr>, registrar=0x7f245ddcf9b8 "pbx_config") at pbx.c:6967
#9 0x00007f245ddcb204 in handle_cli_dialplan_add_extension (e=<value optimized out>, cmd=<value optimized out>, a=0x7f23ab8b8e90) at pbx_config.c:1156
#10 0x00000000004b8158 in ast_cli_command_full (uid=-1, gid=-1, fd=132, s=0x7f245433ee09 "dialplan add extension test_user,hint,SIP/test_company-test_phone&SIP/test_company-test_user_softphone&SIP/test_company-test_phone&Custom:unavailable into blf-test_company replace")
at cli.c:2833
#11 0x00000000005fcd9b in action_command (s=0x7f23ab8b9cd0, m=0x7f23ab8b93e0) at manager.c:5238
#12 0x00000000005fd5c0 in process_message (s=0x7f23ab8b9cd0, m=0x7f23ab8b93e0) at manager.c:6609
#13 0x0000000000600893 in do_message (data=0x7f23c7780dd0) at manager.c:6822
#14 session_do (data=0x7f23c7780dd0) at manager.c:6927
#15 0x0000000000599b77 in dummy_start (data=<value optimized out>) at utils.c:1249
#16 0x00000039c0407aa1 in start_thread (arg=0x7f23ab8ba700) at pthread_create.c:301
#17 0x00000039c00e8c4d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) frame 2
#2 0x00000039c04094d7 in __pthread_mutex_lock (mutex=0x25e6bf0) at pthread_mutex_lock.c:82
82 LLL_MUTEX_LOCK (mutex);
(gdb) p *((pthread_mutex_t *) 0x25e6bf0)
$3 = {__data = {__lock = 2, __count = 1, __owner = 32520, __nusers = 1, __kind = 1, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = "\002\000\000\000\001\000\000\000\b\177\000\000\001\000\000\000\001", '\000' <repeats 22 times>, __align = 4294967298}
</code></pre></div></div>
<p>This output shows that this thread is blocked waiting for a mutex owned by LWP
<code class="language-plaintext highlighter-rouge">32520</code>. From our previously saved table we know that LWP ID <code class="language-plaintext highlighter-rouge">32520</code> maps to
thread <code class="language-plaintext highlighter-rouge">133</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(gdb) thread 133
[Switching to thread 133 (Thread 0x7f23aa396700 (LWP 32520))]#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
136 2: movl %edx, %eax
(gdb) bt
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1 0x00000039c04095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#2 0x00000039c04094d7 in __pthread_mutex_lock (mutex=0x8d0ba0) at pthread_mutex_lock.c:82
#3 0x0000000000517e13 in ast_rdlock_contexts (hint=0x7f23aa390f10 "", hintsize=512, name=0x0, namesize=0, c=<value optimized out>, context=<value optimized out>, exten=0x7f2448446bb8 "813") at pbx.c:8487
#4 ast_hint_extension (hint=0x7f23aa390f10 "", hintsize=512, name=0x0, namesize=0, c=<value optimized out>, context=<value optimized out>, exten=0x7f2448446bb8 "813") at pbx.c:3000
#5 ast_get_hint (hint=0x7f23aa390f10 "", hintsize=512, name=0x0, namesize=0, c=<value optimized out>, context=<value optimized out>, exten=0x7f2448446bb8 "813") at pbx.c:4134
#6 0x00000000005f0a26 in manager_state_cb (context=0x7f24482f4078 "blf-test_company", exten=0x7f2448446bb8 "813", info=0x7f23aa391170, data=<value optimized out>) at manager.c:7219
#7 0x000000000050d870 in execute_state_callback (cb=0x5f09c0 <manager_state_cb>, context=0x7f24482f4078 "blf-test_company", exten=0x7f2448446bb8 "813", data=0x0, reason=<value optimized out>, hint=0x7f2448446e60, device_state_info=0x0)
at pbx.c:3277
#8 0x0000000000510a96 in ast_add_hint (e=0x7f2448446b20) at pbx.c:4048
#9 0x000000000051d1e7 in ast_add_extension2_lockopt (con=0x7f24482f3f80, replace=0, extension=<value optimized out>, priority=-1, label=<value optimized out>, callerid=<value optimized out>, application=<value optimized out>,
data=0x7f2448446b00, datad=0x45a7c0 <ast_free_ptr>, registrar=0x7f245ddcf9b8 "pbx_config", registrar_file=0x7f244801ea91 "exec.158437110324484.139791156471552", registrar_line=1002, lock_context=1) at pbx.c:7550
#10 0x000000000051da2a in ast_add_extension2 (con=<value optimized out>, replace=<value optimized out>, extension=<value optimized out>, priority=<value optimized out>, label=<value optimized out>, callerid=<value optimized out>,
application=0x7f2448446ae9 "park:813@test_company", data=0x7f2448446b00, datad=0x45a7c0 <ast_free_ptr>, registrar=0x7f245ddcf9b8 "pbx_config", registrar_file=0x7f244801ea91 "exec.158437110324484.139791156471552", registrar_line=1002)
at pbx.c:7292
#11 0x00007f245ddca41b in pbx_load_config (config_file=0x7f245ddcf9e0 "extensions.conf") at pbx_config.c:1890
#12 0x00007f245ddcac2e in pbx_load_module () at pbx_config.c:2101
#13 0x00007f245ddcae1e in handle_cli_dialplan_reload (e=<value optimized out>, cmd=<value optimized out>, a=<value optimized out>) at pbx_config.c:1597
#14 0x00000000004b8158 in ast_cli_command_full (uid=-1, gid=-1, fd=73, s=0x7f244872a649 "dialplan reload") at cli.c:2833
#15 0x00000000005fcd9b in action_command (s=0x7f23aa395cd0, m=0x7f23aa3953e0) at manager.c:5238
#16 0x00000000005fd5c0 in process_message (s=0x7f23aa395cd0, m=0x7f23aa3953e0) at manager.c:6609
#17 0x0000000000600893 in do_message (data=0x7f23c72d5750) at manager.c:6822
#18 session_do (data=0x7f23c72d5750) at manager.c:6927
#19 0x0000000000599b77 in dummy_start (data=<value optimized out>) at utils.c:1249
#20 0x00000039c0407aa1 in start_thread (arg=0x7f23aa396700) at pthread_create.c:301
#21 0x00000039c00e8c4d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) frame 2
#2 0x00000039c04094d7 in __pthread_mutex_lock (mutex=0x8d0ba0) at pthread_mutex_lock.c:82
82 LLL_MUTEX_LOCK (mutex);
(gdb) p *((pthread_mutex_t *) 0x8d0ba0)
$4 = {__data = {__lock = 2, __count = 1, __owner = 31744, __nusers = 1, __kind = 1, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = "\002\000\000\000\001\000\000\000\000|\000\000\001\000\000\000\001", '\000' <repeats 22 times>, __align = 4294967298}
</code></pre></div></div>
<p>This output shows that this thread is blocked waiting for a mutex owned by LWP
<code class="language-plaintext highlighter-rouge">31744</code>. From our previously saved table we know that LWP ID <code class="language-plaintext highlighter-rouge">31744</code> maps to
thread <code class="language-plaintext highlighter-rouge">132</code>.</p>
<p>And here we see that these two threads are the root cause of the deadlock. Both
threads are waiting on a mutex that the other already owns. Looking at their
respective backtraces we can see that this is likely an issue in how locking is
done around hint updates.</p>
<p>Unfortunately we can’t fix this ourselves. The next step is to report it to
Digium and hope we can get some help with a fix.</p>Steven WheelerTracking down Asterisk deadlocks in production can prove very challenging. Especially given the high overhead to enabling the compile time option DEBUG_THREADS. This process has worked well for me in the past and may help save you some time tracking down the GDB commands in the future.Migrate maven artifacts from Artifactory to Nexus 32018-10-30T15:33:00+00:002018-10-30T15:33:00+00:00https://www.steventwheeler.com/java/2018/10/30/migrate-artifactory-to-nexus<p>I recently needed to move build artifacts from <a href="https://jfrog.com/artifactory/">JFrog Artifactory</a> to
<a href="https://www.sonatype.com/nexus-repository-oss">Sonatype Nexus</a>. But I couldn’t migrate every project in one go and they were
all commingled in the same repository.</p>
<p>The first task was to export the existing artifacts using the
built-in utility. Start by navigating to the <em>Admin</em> menu and selecting
<em>Repositories</em> under the <strong>Import & Export</strong> section. I used the following
settings. Note that excluding the metadata is important here as they are
meaningless to Nexus and will just clutter things up.</p>
<p><img src="/assets/article_files/2018-10-30-migrate-artifactory-to-nexus/export.png" alt="Export configuration" /></p>
<p>The next step was to find a way to filter the uploaded artifacts. I started with
the <a href="https://github.com/DarthHater/nexus-repository-import-scripts/blob/master/mavenimport.sh"><code class="language-plaintext highlighter-rouge">mavenimport.sh</code></a> script by <a href="https://github.com/DarthHater/nexus-repository-import-scripts">DarthHater</a> but found that it didn’t meet my
needs. With that inspiration it was pretty easy to create my own <a href="/assets/article_files/2018-10-30-migrate-artifactory-to-nexus/import.sh"><code class="language-plaintext highlighter-rouge">import.sh</code></a>
using the <a href="https://linux.die.net/man/1/find"><code class="language-plaintext highlighter-rouge">find</code> command</a>.</p>
<p><a href="/assets/article_files/2018-10-30-migrate-artifactory-to-nexus/import.sh"><code class="language-plaintext highlighter-rouge">import.sh</code></a> takes the following options:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">-r|--repo-url</code>: the URL of the Nexus repository which the artifacts should be
uploaded to.</li>
<li><code class="language-plaintext highlighter-rouge">-u|--username</code>: the username of the account on the Nexus server to use when
authenticating the upload. If not specified the default value is <code class="language-plaintext highlighter-rouge">anonymous</code>.</li>
<li><code class="language-plaintext highlighter-rouge">-p|--password</code>: the password of the account on the Nexus server to use when
authenticating the upload. If not specified the default value is <code class="language-plaintext highlighter-rouge">anonymous</code>.</li>
<li><code class="language-plaintext highlighter-rouge">-g|--group</code>: the group whose artifacts will be uploaded. This must be
specified in dot separated notation. For example, <code class="language-plaintext highlighter-rouge">com.example.group.app</code>.</li>
<li><code class="language-plaintext highlighter-rouge">-b|--base-dir</code>: the directory to search in for artifacts. If not specified
the default value is <code class="language-plaintext highlighter-rouge">$(pwd)</code>.</li>
</ul>
<p>Using this I can now import just the artifacts I want.</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="go">cd /tmp/repositories/
wget https://www.steventwheeler.com/assets/article_files/2018-10-30-migrate-artifactory-to-nexus/import.sh
chmod +x import.sh
./import.sh --repo-url https://nexus.example.com/repository/maven-hosted-releases/ --group com.example.myapp --base-dir libs-release-local/
</span></code></pre></div></div>Steven WheelerI recently needed to move build artifacts from JFrog Artifactory to Sonatype Nexus. But I couldn’t migrate every project in one go and they were all commingled in the same repository.Debugging RPM Dependencies2018-10-18T18:04:08+00:002018-10-18T18:04:08+00:00https://www.steventwheeler.com/linux/2018/10/18/debugging-rpm-dependencies<p>While working on building Asterisk 16.0.0 RPMs I ran across a very unhelpful
error message. When attempting to install the generated RPM file via YUM it
finds a dependency on glibc 2.14 which is not available for CentOS 6. Since I
knew that my <code class="language-plaintext highlighter-rouge">asterisk.spec</code> used to generate the RPM didn’t declare the
dependency and that the Docker image I was using to compile the source was using
glibc 2.12 I was at a bit of a loss.</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">[root@4eece9df947a ~]#</span><span class="w"> </span>yum <span class="nb">install</span> ~/rpmbuild/RPMS/x86_64/asterisk-16.0.0-1.el6.x86_64.rpm
<span class="go">Loaded plugins: fastestmirror, ovl
Setting up Install Process
Examining ~/rpmbuild/RPMS/x86_64/asterisk-16.0.0-1.el6.x86_64.rpm: asterisk-16.0.0-1.el6.x86_64
Marking ~/rpmbuild/RPMS/x86_64/asterisk-16.0.0-1.el6.x86_64.rpm to be installed
Loading mirror speeds from cached hostfile
* base: mirrors.usinternet.com
* epel: mirror.steadfastnet.com
* extras: mirrors.usinternet.com
* updates: mirrors.usinternet.com
Resolving Dependencies
</span><span class="gp">--></span><span class="w"> </span>Running transaction check
<span class="gp">---></span><span class="w"> </span>Package asterisk.x86_64 0:16.0.0-1.el6 will be installed
<span class="gp">--></span><span class="w"> </span>Processing Dependency: libc.so.6<span class="o">(</span>GLIBC_2.14<span class="o">)(</span>64bit<span class="o">)</span> <span class="k">for </span>package: asterisk-16.0.0-1.el6.x86_64
<span class="gp">--></span><span class="w"> </span>Finished Dependency Resolution
<span class="go">Error: Package: asterisk-16.0.0-1.el6.x86_64 (/asterisk-16.0.0-1.el6.x86_64)
Requires: libc.so.6(GLIBC_2.14)(64bit)
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
</span></code></pre></div></div>
<p>Since my spec didn’t declare the dependency my next step was to try and narrow
down where the dependency was being introduced. In order to do that my first
step was to get the shared libraries out of the RPM. It is possible to extract
them from the RPM using
<a href="https://blog.packagecloud.io/eng/2015/10/13/inspect-extract-contents-rpm-packages/"><code class="language-plaintext highlighter-rouge">rpm2cpio</code> and <code class="language-plaintext highlighter-rouge">cpio</code></a>
but for me the easiest way was to re-run the <code class="language-plaintext highlighter-rouge">rpmbuild</code> command with the <code class="language-plaintext highlighter-rouge">-bi</code>
flag rather than the <code class="language-plaintext highlighter-rouge">-ba</code> flag I had been using.</p>
<p>According to the <a href="https://linux.die.net/man/8/rpmbuild">man page</a>:</p>
<blockquote>
<p><code class="language-plaintext highlighter-rouge">-ba</code> Build binary and source packages (after doing the %prep, %build, and %install stages).</p>
<p><code class="language-plaintext highlighter-rouge">-bi</code> Do the “%install” stage from the spec file (after doing the %prep and %build stages). This generally involves the equivalent of a “make install”.</p>
</blockquote>
<p>Once the shared libraries were compiled it is just a matter of iterating over
each of them and using <code class="language-plaintext highlighter-rouge">ldd</code> to print out their dependencies.</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">[root@4eece9df947a ~]#</span><span class="w"> </span>find ~/rpmbuild/BUILDROOT <span class="se">\</span>
<span class="nt">-type</span> f <span class="se">\</span>
<span class="nt">-regextype</span> posix-extended <span class="se">\</span>
<span class="nt">-regex</span> <span class="s1">'.*\.so(\.[0-9]+)?'</span> <span class="se">\</span>
<span class="nt">-exec</span> ldd <span class="o">{}</span> <span class="se">\;</span> | <span class="se">\</span>
<span class="nb">grep</span> <span class="s1">'GLIBC_2.14'</span>
<span class="go">~/rpmbuild/BUILDROOT/asterisk-16.0.0-1.el6.x86_64/usr/lib64/asterisk/modules/format_ogg_opus.so: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ~/rpmbuild/BUILDROOT/asterisk-16.0.0-1.el6.x86_64/usr/lib64/asterisk/modules/format_ogg_opus.so)
~/rpmbuild/BUILDROOT/asterisk-16.0.0-1.el6.x86_64/usr/lib64/asterisk/modules/codec_silk.so: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ~/rpmbuild/BUILDROOT/asterisk-16.0.0-1.el6.x86_64/usr/lib64/asterisk/modules/codec_silk.so)
~/rpmbuild/BUILDROOT/asterisk-16.0.0-1.el6.x86_64/usr/lib64/asterisk/modules/codec_g729a.so: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ~/rpmbuild/BUILDROOT/asterisk-16.0.0-1.el6.x86_64/usr/lib64/asterisk/modules/codec_g729a.so)
~/rpmbuild/BUILDROOT/asterisk-16.0.0-1.el6.x86_64/usr/lib64/asterisk/modules/codec_opus.so: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ~/rpmbuild/BUILDROOT/asterisk-16.0.0-1.el6.x86_64/usr/lib64/asterisk/modules/codec_opus.so)
</span></code></pre></div></div>
<p>It turns out that <code class="language-plaintext highlighter-rouge">make install</code> downloads these precompiled libraries.
Unfortunately they are not properly compiled to run on CentOS 6. Since I have no
need for these particular libraries I can just disable them using
<a href="https://wiki.asterisk.org/wiki/display/AST/Using+Menuselect+to+Select+Asterisk+Options#UsingMenuselecttoSelectAsteriskOptions-ControllingMenuselect"><code class="language-plaintext highlighter-rouge">menuselect</code></a>.</p>
<pre><code class="language-rpmspec">%if 0%{?el6}
menuselect/menuselect \
--disable codec_opus \
--disable codec_silk \
--disable codec_g729a \
menuselect.makeopts
%endif
</code></pre>
<p>And now I can rebuild the RPM and install it!</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">[root@4eece9df947a ~]#</span><span class="w"> </span>rpm <span class="nt">-ba</span> ~/rpmspec/SPECS/asterisk.spec
<span class="go">... lots of boring output ...
Checking for unpackaged file(s): /usr/lib/rpm/check-files ~/rpmbuild/BUILDROOT/asterisk-16.0.0-1.el6.x86_64
Wrote: ~/rpmbuild/SRPMS/asterisk-16.0.0-1.el6.src.rpm
Wrote: ~/rpmbuild/RPMS/x86_64/asterisk-16.0.0-1.el6.x86_64.rpm
Wrote: ~/rpmbuild/RPMS/x86_64/asterisk-devel-16.0.0-1.el6.x86_64.rpm
Wrote: ~/rpmbuild/RPMS/x86_64/asterisk-debuginfo-16.0.0-1.el6.x86_64.rpm
Executing(%clean): /bin/sh -e /var~/rpm-tmp.Nze4wO
+ umask 022
+ cd ~/rpmbuild/BUILD
+ cd asterisk-16.0.0
+ /bin/rm -rf ~/rpmbuild/BUILDROOT/asterisk-16.0.0-1.el6.x86_64
+ exit 0
</span><span class="gp">[root@4eece9df947a ~]#</span><span class="w"> </span>yum <span class="nb">install</span> <span class="nt">-y</span> ~/rpmbuild/RPMS/x86_64/asterisk-16.0.0-1.el6.x86_64.rpm
<span class="go">Loaded plugins: fastestmirror, ovl
Setting up Install Process
Examining ~/rpmbuild/RPMS/x86_64/asterisk-16.0.0-1.el6.x86_64.rpm: asterisk-16.0.0-1.el6.x86_64
Marking ~/rpmbuild/RPMS/x86_64/asterisk-16.0.0-1.el6.x86_64.rpm to be installed
Loading mirror speeds from cached hostfile
* base: mirrors.usinternet.com
* epel: d2lzkl7pfhq30w.cloudfront.net
* extras: mirrors.usinternet.com
* updates: mirrors.usinternet.com
Resolving Dependencies
</span><span class="gp">--></span><span class="w"> </span>Running transaction check
<span class="gp">---></span><span class="w"> </span>Package asterisk.x86_64 0:16.0.0-1.el6 will be installed
<span class="gp">--></span><span class="w"> </span>Finished Dependency Resolution
<span class="go">
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
asterisk x86_64 16.0.0-1.el6 /asterisk-16.0.0-1.el6.x86_64 196 M
Transaction Summary
================================================================================
Install 1 Package(s)
Total size: 196 M
Installed size: 196 M
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : asterisk-16.0.0-1.el6.x86_64 1/1
Verifying : asterisk-16.0.0-1.el6.x86_64 1/1
Installed:
asterisk.x86_64 0:16.0.0-1.el6
Complete!
</span></code></pre></div></div>
<p>Success!</p>Steven WheelerWhile working on building Asterisk 16.0.0 RPMs I ran across a very unhelpful error message. When attempting to install the generated RPM file via YUM it finds a dependency on glibc 2.14 which is not available for CentOS 6. Since I knew that my asterisk.spec used to generate the RPM didn’t declare the dependency and that the Docker image I was using to compile the source was using glibc 2.12 I was at a bit of a loss.