Debugging Asterisk Deadlocks

Blog Logo
on



read

Tracking down Asterisk deadlocks in production can prove very challenging. Especially given the high overhead to enabling the compile time option DEBUG_THREADS. This process has worked well for me in the past and may help save you some time tracking down the GDB commands in the future.

Generate a core dump

If asterisk is still running you will want to generate a core dump. The easiest way to do this is with the provided script.

$ /var/lib/asterisk/scripts/ast_coredumper --running

This will generate a core file in /tmp by default.

Get a list of the threads

The next step which will save us some hunting later is to get a list of all of the threads with their thread number and LWP ID. The lock owner will be a LWP ID but we will need the thread number in order to tell GDB to switch to the correct thread.

The listing can be generated by running info threads on the core file.

$ gdb asterisk /tmp/core-asterisk-running-2020-03-16T11-45-23-0500 --batch -ex 'info threads'
...
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
183	62:	movl	(%rsp), %edi
  239 Thread 0x7f245f11a820 (LWP 9344)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  238 Thread 0x7f23a81ae700 (LWP 1432)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  237 Thread 0x7f23a8322700 (LWP 3273)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  236 Thread 0x7f23a8496700 (LWP 4377)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  235 Thread 0x7f23a82a6700 (LWP 4374)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  234 Thread 0x7f23a839e700 (LWP 4815)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  233 Thread 0x7f23a8512700 (LWP 4612)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  232 Thread 0x7f23a841a700 (LWP 4437)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  231 Thread 0x7f23a860a700 (LWP 4301)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  230 Thread 0x7f23a858e700 (LWP 4127)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  229 Thread 0x7f23a8702700 (LWP 2502)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  228 Thread 0x7f23a8686700 (LWP 2327)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  227 Thread 0x7f23a88f2700 (LWP 625)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  226 Thread 0x7f23a877e700 (LWP 460)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  225 Thread 0x7f23a87fa700 (LWP 323)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  224 Thread 0x7f23a8876700 (LWP 32613)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  223 Thread 0x7f23a89ea700 (LWP 30994)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  222 Thread 0x7f23a896e700 (LWP 30827)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  221 Thread 0x7f23a8ae2700 (LWP 30676)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  220 Thread 0x7f23a8a66700 (LWP 30511)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  219 Thread 0x7f23a8c56700 (LWP 28878)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  218 Thread 0x7f23a8b5e700 (LWP 28705)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  217 Thread 0x7f23a8d4e700 (LWP 26762)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  216 Thread 0x7f23a8cd2700 (LWP 26700)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  215 Thread 0x7f23a8bda700 (LWP 26574)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  214 Thread 0x7f23a8f3e700 (LWP 26445)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  213 Thread 0x7f23a8dca700 (LWP 24788)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  212 Thread 0x7f23a8e46700 (LWP 24665)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  211 Thread 0x7f23a8ec2700 (LWP 24477)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  210 Thread 0x7f23a912e700 (LWP 24348)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  209 Thread 0x7f23a8fba700 (LWP 22663)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  208 Thread 0x7f23a90b2700 (LWP 22535)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  207 Thread 0x7f23a9036700 (LWP 20880)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  206 Thread 0x7f23a91aa700 (LWP 20755)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  205 Thread 0x7f23a9226700 (LWP 20383)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  204 Thread 0x7f23a931e700 (LWP 20168)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  203 Thread 0x7f23a92a2700 (LWP 18149)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  202 Thread 0x7f23a950e700 (LWP 17941)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  201 Thread 0x7f23a939a700 (LWP 17386)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  200 Thread 0x7f23a9416700 (LWP 17263)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  199 Thread 0x7f23a9492700 (LWP 14913)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  198 Thread 0x7f23a9606700 (LWP 14698)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  197 Thread 0x7f23a958a700 (LWP 12606)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  196 Thread 0x7f23a98ee700 (LWP 12397)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  195 Thread 0x7f23a977a700 (LWP 11811)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  194 Thread 0x7f23a9ebe700 (LWP 11593)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  193 Thread 0x7f23a9ade700 (LWP 9534)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  192 Thread 0x7f23aa0ae700 (LWP 9301)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  191 Thread 0x7f23a996a700 (LWP 8731)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  190 Thread 0x7f23a9a62700 (LWP 8512)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  189 Thread 0x7f23a9c52700 (LWP 6453)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  188 Thread 0x7f23a9e42700 (LWP 6233)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  187 Thread 0x7f23a9d4a700 (LWP 4381)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  186 Thread 0x7f23e4a97700 (LWP 3806)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  185 Thread 0x7f23a9fb6700 (LWP 3351)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  184 Thread 0x7f23abfff700 (LWP 3133)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  183 Thread 0x7f245e8dd700 (LWP 874)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  182 Thread 0x7f23d9f7a700 (LWP 739)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  181 Thread 0x7f23abd92700 (LWP 550)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  180 Thread 0x7f23da072700 (LWP 32617)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  179 Thread 0x7f23abc1e700 (LWP 31208)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  178 Thread 0x7f23abba2700 (LWP 31075)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  177 Thread 0x7f23e59ed700 (LWP 30414)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  176 Thread 0x7f23d9e82700 (LWP 29263)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  175 Thread 0x7f23d93da700 (LWP 29089)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  174 Thread 0x7f23ab126700 (LWP 28962)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  173 Thread 0x7f23d9ff6700 (LWP 27295)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  172 Thread 0x7f23d9e06700 (LWP 27172)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  171 Thread 0x7f23d973e700 (LWP 26968)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  170 Thread 0x7f23d96c2700 (LWP 26091)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  169 Thread 0x7f23d9b1e700 (LWP 25162)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  168 Thread 0x7f23e41df700 (LWP 24755)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  167 Thread 0x7f23aa222700 (LWP 24206)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  166 Thread 0x7f23a99e6700 (LWP 22532)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  165 Thread 0x7f23aa12a700 (LWP 22355)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  164 Thread 0x7f23e5879700 (LWP 22227)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  163 Thread 0x7f23a9b5a700 (LWP 20567)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  162 Thread 0x7f245eacd700 (LWP 20336)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  161 Thread 0x7f23e40e7700 (LWP 20158)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  160 Thread 0x7f23d92e2700 (LWP 18396)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  159 Thread 0x7f23d91ea700 (LWP 18062)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  158 Thread 0x7f23a9bd6700 (LWP 17933)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  157 Thread 0x7f23e44c7700 (LWP 17752)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  156 Thread 0x7f23da16a700 (LWP 16152)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  155 Thread 0x7f23e42d7700 (LWP 15970)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  154 Thread 0x7f23aa032700 (LWP 15846)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  153 Thread 0x7f23d9d0e700 (LWP 14174)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  152 Thread 0x7f23e5a69700 (LWP 14038)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  151 Thread 0x7f23d9aa2700 (LWP 13828)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  150 Thread 0x7f23e4923700 (LWP 11856)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  149 Thread 0x7f23d916e700 (LWP 11678)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  148 Thread 0x7f23d935e700 (LWP 11548)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  147 Thread 0x7f23d98b2700 (LWP 11369)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  146 Thread 0x7f23e4543700 (LWP 9760)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  145 Thread 0x7f23d97ba700 (LWP 9587)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  144 Thread 0x7f23a9cce700 (LWP 9425)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  143 Thread 0x7f23abaaa700 (LWP 7735)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  142 Thread 0x7f23d9d8a700 (LWP 7595)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  141 Thread 0x7f23ab936700 (LWP 7406)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  140 Thread 0x7f23abf83700 (LWP 5800)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  139 Thread 0x7f23d9efe700 (LWP 5626)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  138 Thread 0x7f23e4353700 (LWP 5484)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  137 Thread 0x7f23e4163700 (LWP 5317)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  136 Thread 0x7f23abf07700 (LWP 3667)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  135 Thread 0x7f23d992e700 (LWP 3518)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  134 Thread 0x7f23d9456700 (LWP 3135)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  133 Thread 0x7f23aa396700 (LWP 32520)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  132 Thread 0x7f23ab8ba700 (LWP 31744)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  131 Thread 0x7f23abc9a700 (LWP 31639)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  130 Thread 0x7f245ea51700 (LWP 31633)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  129 Thread 0x7f23aa50a700 (LWP 31631)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  128 Thread 0x7f245e9d3700 (LWP 31628)  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
  127 Thread 0x7f23d9266700 (LWP 31627)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  126 Thread 0x7f23a9dc6700 (LWP 27413)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  125 Thread 0x7f23ab746700 (LWP 27107)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  124 Thread 0x7f23a97f6700 (LWP 27106)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  123 Thread 0x7f23d9a26700 (LWP 27100)  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
  122 Thread 0x7f23abe0e700 (LWP 27099)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  121 Thread 0x7f23e4733700 (LWP 25360)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  120 Thread 0x7f23abd16700 (LWP 25359)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  119 Thread 0x7f23d9836700 (LWP 25358)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  118 Thread 0x7f23e499f700 (LWP 25357)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  117 Thread 0x7f23e444b700 (LWP 25353)  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
  116 Thread 0x7f23d95ca700 (LWP 25350)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  115 Thread 0x7f23e43cf700 (LWP 25347)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  114 Thread 0x7f23e48a7700 (LWP 24868)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  113 Thread 0x7f23e46b7700 (LWP 23534)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  112 Thread 0x7f23d9c92700 (LWP 23141)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  111 Thread 0x7f23abe8b700 (LWP 23133)  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
  110 Thread 0x7f23d90f2700 (LWP 23132)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  109 Thread 0x7f23ab9b2700 (LWP 22833)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  108 Thread 0x7f23d9b9a700 (LWP 22831)  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
  107 Thread 0x7f23aba2e700 (LWP 22828)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  106 Thread 0x7f23a9f3a700 (LWP 22826)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  105 Thread 0x7f23ab0aa700 (LWP 22812)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  104 Thread 0x7f23aa31a700 (LWP 22811)  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
  103 Thread 0x7f23d9646700 (LWP 22810)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  102 Thread 0x7f23da0ee700 (LWP 22809)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  101 Thread 0x7f23d94d2700 (LWP 22808)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  100 Thread 0x7f23aa29e700 (LWP 22794)  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
  99 Thread 0x7f23e425b700 (LWP 22790)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  98 Thread 0x7f23ab83e700 (LWP 22757)  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
  97 Thread 0x7f23e4a1b700 (LWP 22756)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  96 Thread 0x7f23a96fe700 (LWP 20189)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  95 Thread 0x7f23e482b700 (LWP 20173)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  94 Thread 0x7f23d954e700 (LWP 20171)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  93 Thread 0x7f23ab6ca700 (LWP 20162)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  92 Thread 0x7f23aa1a6700 (LWP 16456)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  91 Thread 0x7f23d9c16700 (LWP 16455)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  90 Thread 0x7f23aa412700 (LWP 16454)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  89 Thread 0x7f23ab7c2700 (LWP 16046)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  88 Thread 0x7f23a9682700 (LWP 12183)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  87 Thread 0x7f23e58f5700 (LWP 812)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  86 Thread 0x7f23aa48e700 (LWP 8612)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  85 Thread 0x7f23abb26700 (LWP 8198)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  84 Thread 0x7f23e5971700 (LWP 7774)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  83 Thread 0x7f23a9872700 (LWP 7515)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  82 Thread 0x7f23d99aa700 (LWP 8673)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  81 Thread 0x7f23d9076700 (LWP 19256)  0x00000039c040e82d in read () at ../sysdeps/unix/syscall-template.S:82
  80 Thread 0x7f23e57fd700 (LWP 18301)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  79 Thread 0x7f23ab1a2700 (LWP 14027)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  78 Thread 0x7f23ab21e700 (LWP 14026)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  77 Thread 0x7f23ab29a700 (LWP 14025)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  76 Thread 0x7f23ab316700 (LWP 13916)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  75 Thread 0x7f23ab5fe700 (LWP 13915)  0x00000039c040f00d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
  74 Thread 0x7f23ab392700 (LWP 13914)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  73 Thread 0x7f23ab40e700 (LWP 13913)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  72 Thread 0x7f23ab48a700 (LWP 13912)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  71 Thread 0x7f23ab506700 (LWP 13910)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  70 Thread 0x7f23ab582700 (LWP 13909)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  69 Thread 0x7f23d80f6700 (LWP 13888)  0x00000039c00accbd in nanosleep () at ../sysdeps/unix/syscall-template.S:82
  68 Thread 0x7f23d8172700 (LWP 13887)  0x00000039c00accbd in nanosleep () at ../sysdeps/unix/syscall-template.S:82
  67 Thread 0x7f23d81ee700 (LWP 13886)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  66 Thread 0x7f23d826a700 (LWP 13883)  0x00000039c040f00d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
  65 Thread 0x7f23d82e6700 (LWP 13882)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  64 Thread 0x7f23d8362700 (LWP 13881)  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
  63 Thread 0x7f23d83de700 (LWP 13880)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  62 Thread 0x7f23d845a700 (LWP 13879)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  61 Thread 0x7f23d84d6700 (LWP 13878)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  60 Thread 0x7f23d8552700 (LWP 13877)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  59 Thread 0x7f23d85ce700 (LWP 13876)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  58 Thread 0x7f23d864a700 (LWP 13875)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  57 Thread 0x7f23d86c6700 (LWP 13874)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  56 Thread 0x7f23d8742700 (LWP 13871)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  55 Thread 0x7f23d87be700 (LWP 13870)  0x00000039c040eb2d in accept () at ../sysdeps/unix/syscall-template.S:82
  54 Thread 0x7f23d883a700 (LWP 13869)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  53 Thread 0x7f23d88b6700 (LWP 13868)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  52 Thread 0x7f23d8932700 (LWP 13867)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  51 Thread 0x7f23d89ae700 (LWP 13866)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  50 Thread 0x7f23d8a2a700 (LWP 13865)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  49 Thread 0x7f23d8aa6700 (LWP 13864)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  48 Thread 0x7f23d8b22700 (LWP 13863)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  47 Thread 0x7f23d8b9e700 (LWP 13862)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  46 Thread 0x7f23d8c1a700 (LWP 13861)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  45 Thread 0x7f23d8c96700 (LWP 13860)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  44 Thread 0x7f23d8d12700 (LWP 13859)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  43 Thread 0x7f23d8d8e700 (LWP 13858)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  42 Thread 0x7f23d8e0a700 (LWP 13857)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  41 Thread 0x7f23d8e86700 (LWP 13856)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  40 Thread 0x7f23d8f02700 (LWP 13855)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  39 Thread 0x7f23d8f7e700 (LWP 13854)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  38 Thread 0x7f23d8ffa700 (LWP 13853)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  37 Thread 0x7f23e45bf700 (LWP 13806)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  36 Thread 0x7f23e463b700 (LWP 13805)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  35 Thread 0x7f23e47af700 (LWP 13478)  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  34 Thread 0x7f23e4b8f700 (LWP 13112)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  33 Thread 0x7f23e4b13700 (LWP 11705)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  32 Thread 0x7f23e5591700 (LWP 9581)  0x00000039c00e9243 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
  31 Thread 0x7f23e560d700 (LWP 9578)  0x00000039c00accbd in nanosleep () at ../sysdeps/unix/syscall-template.S:82
  30 Thread 0x7f23e5689700 (LWP 9577)  0x00000039c00accbd in nanosleep () at ../sysdeps/unix/syscall-template.S:82
  29 Thread 0x7f23e5705700 (LWP 9576)  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:239
  28 Thread 0x7f23e5781700 (LWP 9575)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  27 Thread 0x7f23e6dbe700 (LWP 9566)  0x00000039c00e9243 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
  26 Thread 0x7f23e5be7700 (LWP 9565)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  25 Thread 0x7f23e5c63700 (LWP 9564)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  24 Thread 0x7f23e5cdf700 (LWP 9563)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  23 Thread 0x7f23e5d5b700 (LWP 9562)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  22 Thread 0x7f23e5dd7700 (LWP 9561)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  21 Thread 0x7f23e5e53700 (LWP 9560)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  20 Thread 0x7f23e5ecf700 (LWP 9559)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  19 Thread 0x7f23e5f4b700 (LWP 9558)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  18 Thread 0x7f23e5fc7700 (LWP 9557)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  17 Thread 0x7f23e6043700 (LWP 9556)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  16 Thread 0x7f23e60bf700 (LWP 9555)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  15 Thread 0x7f23e613b700 (LWP 9554)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  14 Thread 0x7f23e61b7700 (LWP 9553)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  13 Thread 0x7f245eb49700 (LWP 9467)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  12 Thread 0x7f245ebc5700 (LWP 9466)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  11 Thread 0x7f245ec41700 (LWP 9464)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  10 Thread 0x7f245ecbd700 (LWP 9463)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  9 Thread 0x7f245ed39700 (LWP 9462)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  8 Thread 0x7f245edb5700 (LWP 9461)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  7 Thread 0x7f245ee31700 (LWP 9459)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  6 Thread 0x7f245eead700 (LWP 9458)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  5 Thread 0x7f245ef29700 (LWP 9455)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
  4 Thread 0x7f245efa5700 (LWP 9454)  0x00000039c00df403 in __poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
  3 Thread 0x7f245f021700 (LWP 9452)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  2 Thread 0x7f245f09d700 (LWP 9451)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
* 1 Thread 0x7f245f119700 (LWP 9447)  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183

Find a thread which is waiting on a lock

This step is a bit of trial and error but in general you can look for threads in the output above that are executing in __lll_lock_wait. Start by opening an interactive GDB session.

$ gdb asterisk /tmp/core-asterisk-running-2020-03-16T11-45-23-0500

We will choose 193 at random. Switch to the thread and print the mutex information like so.

(gdb) thread 193
[Switching to thread 193 (Thread 0x7f23a9ade700 (LWP 9534))]#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
136	2:	movl	%edx, %eax
(gdb) bt
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1  0x00000039c04095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#2  0x00000039c04094d7 in __pthread_mutex_lock (mutex=0x7f24135ac7c0) at pthread_mutex_lock.c:82
#3  0x00007f241337dfc8 in handle_request_do (req=0x7f23a9adbe10, addr=0x7f24380e9328) at chan_sip.c:29373
#4  0x00007f241337eb99 in _sip_tcp_helper_thread (tcptls_session=0x7f24380e9320) at chan_sip.c:3137
#5  0x0000000000599b77 in dummy_start (data=<value optimized out>) at utils.c:1249
#6  0x00000039c0407aa1 in start_thread (arg=0x7f23a9ade700) at pthread_create.c:301
#7  0x00000039c00e8c4d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) frame 2
#2  0x00000039c04094d7 in __pthread_mutex_lock (mutex=0x7f24135ac7c0) at pthread_mutex_lock.c:82
82	      LLL_MUTEX_LOCK (mutex);
(gdb) p *((pthread_mutex_t *) 0x7f24135ac7c0)
$2 = {__data = {__lock = 2, __count = 1, __owner = 3135, __nusers = 1, __kind = 1, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
  __size = "\002\000\000\000\001\000\000\000?\f\000\000\001\000\000\000\001", '\000' <repeats 22 times>, __align = 4294967298}

We switch to frame 2 since that’s where the mutex address is displayed. Then print the mutex contents by dereferencing the pointer. We can see that the owner of this mutex is LWP 3135. The next step is to switch to that thread and repeat the process until we find ourselves in a loop. From our previously saved table we know that LWP ID 3135 maps to thread 134.

(gdb) thread 134
[Switching to thread 134 (Thread 0x7f23d9456700 (LWP 3135))]#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
136	2:	movl	%edx, %eax
(gdb) bt
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1  0x00000039c04095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#2  0x00000039c04094d7 in __pthread_mutex_lock (mutex=0x8d0ba0) at pthread_mutex_lock.c:82
#3  0x0000000000516c75 in ast_rdlock_contexts (c=0x0, con=0x0, context=0x7f23b022b560 "fail-call", exten=0x7f24133897cc "s", priority=1, label=0x0, callerid=0x7f23b0177c0b "monit", action=E_MATCH, found=0x0, combined_find_spawn=0)
    at pbx.c:8487
#4  pbx_extension_helper (c=0x0, con=0x0, context=0x7f23b022b560 "fail-call", exten=0x7f24133897cc "s", priority=1, label=0x0, callerid=0x7f23b0177c0b "monit", action=E_MATCH, found=0x0, combined_find_spawn=0) at pbx.c:2886
#5  0x00000000005175c4 in ast_exists_extension (c=<value optimized out>, context=<value optimized out>, exten=<value optimized out>, priority=<value optimized out>, callerid=<value optimized out>) at pbx.c:4172
#6  0x00007f2413325370 in get_destination (p=<value optimized out>, oreq=0x7f23d9453e10, cc_recall_core_id=0x0) at chan_sip.c:18607
#7  0x00007f2413372099 in handle_request_options (p=0x7f23b0296fe0, req=0x7f23d9453e10, addr=0x7f243811f368, e=<value optimized out>) at chan_sip.c:25840
#8  0x00007f241337c2af in handle_incoming (p=0x7f23b0296fe0, req=0x7f23d9453e10, addr=0x7f243811f368, recount=0x7f23d94532cc, nounlock=0x7f23d94532c8) at chan_sip.c:29191
#9  0x00007f241337e0fd in handle_request_do (req=0x7f23d9453e10, addr=0x7f243811f368) at chan_sip.c:29403
#10 0x00007f241337eb99 in _sip_tcp_helper_thread (tcptls_session=0x7f243811f360) at chan_sip.c:3137
#11 0x0000000000599b77 in dummy_start (data=<value optimized out>) at utils.c:1249
#12 0x00000039c0407aa1 in start_thread (arg=0x7f23d9456700) at pthread_create.c:301
#13 0x00000039c00e8c4d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) frame 2
#2  0x00000039c04094d7 in __pthread_mutex_lock (mutex=0x8d0ba0) at pthread_mutex_lock.c:82
82	      LLL_MUTEX_LOCK (mutex);
(gdb) p *((pthread_mutex_t *) 0x8d0ba0)
$2 = {__data = {__lock = 2, __count = 1, __owner = 31744, __nusers = 1, __kind = 1, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
  __size = "\002\000\000\000\001\000\000\000\000|\000\000\001\000\000\000\001", '\000' <repeats 22 times>, __align = 4294967298}

This output shows that this thread is blocked waiting for a mutex owned by LWP 31744. From our previously saved table we know that LWP ID 31744 maps to thread 132.

(gdb) thread 132
[Switching to thread 132 (Thread 0x7f23ab8ba700 (LWP 31744))]#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
136	2:	movl	%edx, %eax
(gdb) bt
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1  0x00000039c04095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#2  0x00000039c04094d7 in __pthread_mutex_lock (mutex=0x25e6bf0) at pthread_mutex_lock.c:82
#3  0x000000000045b506 in __ao2_lock (user_data=<value optimized out>, lock_how=<value optimized out>, file=0x63d8a3 "pbx.c", func=0x641120 "ast_change_hint", line=4095, var=<value optimized out>) at astobj2.c:241
#4  0x000000000051d526 in ast_change_hint (con=0x7f24243da630, replace=1, extension=<value optimized out>, priority=608020264, label=<value optimized out>, callerid=<value optimized out>, application=<value optimized out>,
    data=0x7f2454028580, datad=0x45a7c0 <ast_free_ptr>, registrar=0x7f245ddcf9b8 "pbx_config", registrar_file=0x0, registrar_line=0, lock_context=1) at pbx.c:4095
#5  add_priority (con=0x7f24243da630, replace=1, extension=<value optimized out>, priority=608020264, label=<value optimized out>, callerid=<value optimized out>, application=<value optimized out>, data=0x7f2454028580,
    datad=0x45a7c0 <ast_free_ptr>, registrar=0x7f245ddcf9b8 "pbx_config", registrar_file=0x0, registrar_line=0, lock_context=1) at pbx.c:7223
#6  ast_add_extension2_lockopt (con=0x7f24243da630, replace=1, extension=<value optimized out>, priority=608020264, label=<value optimized out>, callerid=<value optimized out>, application=<value optimized out>, data=0x7f2454028580,
    datad=0x45a7c0 <ast_free_ptr>, registrar=0x7f245ddcf9b8 "pbx_config", registrar_file=0x0, registrar_line=0, lock_context=1) at pbx.c:7480
#7  0x000000000051eae5 in ast_add_extension2 (context=<value optimized out>, replace=1, extension=0x7f23ab8b8b00 "test_user", priority=-1, label=0x0, callerid=0x0,
    application=0x7f23ab8b8b0e "SIP/test_company-test_phone&SIP/test_company-test_user_softphone&SIP/test_company-test_phone&Custom:unavailable", data=0x7f2454028580, datad=0x45a7c0 <ast_free_ptr>, registrar=0x7f245ddcf9b8 "pbx_config") at pbx.c:7292
#8  ast_add_extension (context=<value optimized out>, replace=1, extension=0x7f23ab8b8b00 "test_user", priority=-1, label=0x0, callerid=0x0,
    application=0x7f23ab8b8b0e "SIP/test_company-test_phone&SIP/test_company-test_user_softphone&SIP/test_company-test_phone&Custom:unavailable", data=0x7f2454028580, datad=0x45a7c0 <ast_free_ptr>, registrar=0x7f245ddcf9b8 "pbx_config") at pbx.c:6967
#9  0x00007f245ddcb204 in handle_cli_dialplan_add_extension (e=<value optimized out>, cmd=<value optimized out>, a=0x7f23ab8b8e90) at pbx_config.c:1156
#10 0x00000000004b8158 in ast_cli_command_full (uid=-1, gid=-1, fd=132, s=0x7f245433ee09 "dialplan add extension test_user,hint,SIP/test_company-test_phone&SIP/test_company-test_user_softphone&SIP/test_company-test_phone&Custom:unavailable into blf-test_company replace")
    at cli.c:2833
#11 0x00000000005fcd9b in action_command (s=0x7f23ab8b9cd0, m=0x7f23ab8b93e0) at manager.c:5238
#12 0x00000000005fd5c0 in process_message (s=0x7f23ab8b9cd0, m=0x7f23ab8b93e0) at manager.c:6609
#13 0x0000000000600893 in do_message (data=0x7f23c7780dd0) at manager.c:6822
#14 session_do (data=0x7f23c7780dd0) at manager.c:6927
#15 0x0000000000599b77 in dummy_start (data=<value optimized out>) at utils.c:1249
#16 0x00000039c0407aa1 in start_thread (arg=0x7f23ab8ba700) at pthread_create.c:301
#17 0x00000039c00e8c4d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) frame 2
#2  0x00000039c04094d7 in __pthread_mutex_lock (mutex=0x25e6bf0) at pthread_mutex_lock.c:82
82	      LLL_MUTEX_LOCK (mutex);
(gdb) p *((pthread_mutex_t *) 0x25e6bf0)
$3 = {__data = {__lock = 2, __count = 1, __owner = 32520, __nusers = 1, __kind = 1, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
  __size = "\002\000\000\000\001\000\000\000\b\177\000\000\001\000\000\000\001", '\000' <repeats 22 times>, __align = 4294967298}

This output shows that this thread is blocked waiting for a mutex owned by LWP 32520. From our previously saved table we know that LWP ID 32520 maps to thread 133.

(gdb) thread 133
[Switching to thread 133 (Thread 0x7f23aa396700 (LWP 32520))]#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
136	2:	movl	%edx, %eax
(gdb) bt
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1  0x00000039c04095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#2  0x00000039c04094d7 in __pthread_mutex_lock (mutex=0x8d0ba0) at pthread_mutex_lock.c:82
#3  0x0000000000517e13 in ast_rdlock_contexts (hint=0x7f23aa390f10 "", hintsize=512, name=0x0, namesize=0, c=<value optimized out>, context=<value optimized out>, exten=0x7f2448446bb8 "813") at pbx.c:8487
#4  ast_hint_extension (hint=0x7f23aa390f10 "", hintsize=512, name=0x0, namesize=0, c=<value optimized out>, context=<value optimized out>, exten=0x7f2448446bb8 "813") at pbx.c:3000
#5  ast_get_hint (hint=0x7f23aa390f10 "", hintsize=512, name=0x0, namesize=0, c=<value optimized out>, context=<value optimized out>, exten=0x7f2448446bb8 "813") at pbx.c:4134
#6  0x00000000005f0a26 in manager_state_cb (context=0x7f24482f4078 "blf-test_company", exten=0x7f2448446bb8 "813", info=0x7f23aa391170, data=<value optimized out>) at manager.c:7219
#7  0x000000000050d870 in execute_state_callback (cb=0x5f09c0 <manager_state_cb>, context=0x7f24482f4078 "blf-test_company", exten=0x7f2448446bb8 "813", data=0x0, reason=<value optimized out>, hint=0x7f2448446e60, device_state_info=0x0)
    at pbx.c:3277
#8  0x0000000000510a96 in ast_add_hint (e=0x7f2448446b20) at pbx.c:4048
#9  0x000000000051d1e7 in ast_add_extension2_lockopt (con=0x7f24482f3f80, replace=0, extension=<value optimized out>, priority=-1, label=<value optimized out>, callerid=<value optimized out>, application=<value optimized out>,
    data=0x7f2448446b00, datad=0x45a7c0 <ast_free_ptr>, registrar=0x7f245ddcf9b8 "pbx_config", registrar_file=0x7f244801ea91 "exec.158437110324484.139791156471552", registrar_line=1002, lock_context=1) at pbx.c:7550
#10 0x000000000051da2a in ast_add_extension2 (con=<value optimized out>, replace=<value optimized out>, extension=<value optimized out>, priority=<value optimized out>, label=<value optimized out>, callerid=<value optimized out>,
    application=0x7f2448446ae9 "park:813@test_company", data=0x7f2448446b00, datad=0x45a7c0 <ast_free_ptr>, registrar=0x7f245ddcf9b8 "pbx_config", registrar_file=0x7f244801ea91 "exec.158437110324484.139791156471552", registrar_line=1002)
    at pbx.c:7292
#11 0x00007f245ddca41b in pbx_load_config (config_file=0x7f245ddcf9e0 "extensions.conf") at pbx_config.c:1890
#12 0x00007f245ddcac2e in pbx_load_module () at pbx_config.c:2101
#13 0x00007f245ddcae1e in handle_cli_dialplan_reload (e=<value optimized out>, cmd=<value optimized out>, a=<value optimized out>) at pbx_config.c:1597
#14 0x00000000004b8158 in ast_cli_command_full (uid=-1, gid=-1, fd=73, s=0x7f244872a649 "dialplan reload") at cli.c:2833
#15 0x00000000005fcd9b in action_command (s=0x7f23aa395cd0, m=0x7f23aa3953e0) at manager.c:5238
#16 0x00000000005fd5c0 in process_message (s=0x7f23aa395cd0, m=0x7f23aa3953e0) at manager.c:6609
#17 0x0000000000600893 in do_message (data=0x7f23c72d5750) at manager.c:6822
#18 session_do (data=0x7f23c72d5750) at manager.c:6927
#19 0x0000000000599b77 in dummy_start (data=<value optimized out>) at utils.c:1249
#20 0x00000039c0407aa1 in start_thread (arg=0x7f23aa396700) at pthread_create.c:301
#21 0x00000039c00e8c4d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) frame 2
#2  0x00000039c04094d7 in __pthread_mutex_lock (mutex=0x8d0ba0) at pthread_mutex_lock.c:82
82	      LLL_MUTEX_LOCK (mutex);
(gdb) p *((pthread_mutex_t *) 0x8d0ba0)
$4 = {__data = {__lock = 2, __count = 1, __owner = 31744, __nusers = 1, __kind = 1, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
  __size = "\002\000\000\000\001\000\000\000\000|\000\000\001\000\000\000\001", '\000' <repeats 22 times>, __align = 4294967298}

This output shows that this thread is blocked waiting for a mutex owned by LWP 31744. From our previously saved table we know that LWP ID 31744 maps to thread 132.

And here we see that these two threads are the root cause of the deadlock. Both threads are waiting on a mutex that the other already owns. Looking at their respective backtraces we can see that this is likely an issue in how locking is done around hint updates.

Unfortunately we can’t fix this ourselves. The next step is to report it to Digium and hope we can get some help with a fix.

Written by
Blog Logo

Steven Wheeler

Published

Supported by
Image

Steven Wheeler

A repository of things I occasionally need to look up. Hopefully, it helps others save time too.

Back to Overview