Solving MySQL connection problems caused by a dead name server

A client site went down today. This is what happened and how it was fixed.


The immediate symptom is that your application servers can’t connect to your database servers. Attempted connections get an error message:

Can’t connect to MySQL server on ‘’ (111)

The relevant MySQL process list reveals a long list of attempted connections in state login:

root@server-db1:~ $ mysqladmin processlist
| Id  | User                 | Host               | db | Command | Time | State |
| 261 | unauthenticated user | |    | Connect |      | login |
| 262 | unauthenticated user | |    | Connect |      | login |
| 263 | unauthenticated user | |    | Connect |      | login |
| 264 | unauthenticated user | |    | Connect |      | login |
| 265 | unauthenticated user |  |    | Connect |      | login |

Your site is probably down as no request can connect to the database - people are starting to get upset.


When a client connects to MySQL, the newly spawned thread attempts to resolve the host name (see MySQL’s documentation on DNS). This problem is caused by the first name server in /etc/resolv.conf being down, causing the DNS request to time-out. Hence, every connection to MySQL sits for a minute waiting for the timeout to occur. Within a few seconds, no client will be able to connect.

You can verify this using the dig command to exercise the name servers in /etc/resolv.conf. Here’s the broken one (you’ll have to wait for the time-out):

dig @

; <<>> DiG 9.7.0-P1 <<>> @
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

A working name server would return something like:

dig @

; <<>> DiG 9.7.0-P1 <<>> @
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14538
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 4, ADDITIONAL: 4


;; ANSWER SECTION:      427763  IN  CNAME    97      IN  A    97      IN  A    97      IN  A    97      IN  A    97      IN  A

;; AUTHORITY SECTION:          250791  IN  NS          250791  IN  NS          250791  IN  NS          250791  IN  NS

;; ADDITIONAL SECTION:      76667   IN  A      76667   IN  A      76667   IN  A      76667   IN  A

;; Query time: 13 msec
;; WHEN: Thu Feb  2 16:18:38 2012
;; MSG SIZE  rcvd: 268


The root solution is to fix the name server, but sometimes that isn’t in your control.

You can work around the borked name server by restarting MySQL with the --skip-name-resolve option. This prevents MySQL trying to resolve the host name for each thread, bypassing the name server problem.

Alternatively, you can remove the broken DNS server from your /etc/resolv.conf.


Note that running MySQL with --skip-name-resolve means you can’t use hostnames in your privileges table. Thus, you may have to reconfigure your client users to get your site back up. You can verify this by using the following SQL to inspect your configured users and hosts:

mysql> SELECT user, host FROM mysql.user;

Check to see if the host column uses domain names.


Thanks to Tangent’s operations team @timbobsteve and @kuramanga for their help in debugging and fixing this.


Something wrong? Suggest an improvement or add a comment (see article history)
Tagged with: mysql
Filed in: tips

Previous: Auto-setting terminal titles for python virtual environments
Next: A Fabric function for git tagging

Copyright © 2005-2017 David Winterbottom
Content licensed under CC BY-NC-SA 4.0.