MySQL is the second famous database in the world according to the DB Engine website behind Oracle. What makes MySQL famous is probably because it is a very fast, reliable and flexible Database Management System. MySQL is also one of the supported databases in ClusterControl. You could easily deploy, scale, monitor and do a lot of things with ClusterControl.
Today we are not going to talk about any of those, but we will discuss one of the common errors for MySQL and possible troubleshooting tips. When working with tickets, a lot of time when we check the error reports or logs, we saw this line “Got an error reading communication packet” quite frequently. We think it would be beneficial if we write a blog related to this error not only for our customers but also for other readers. Let’s wait no further, it’s time to dive more!
MySQL Client/Server Protocol
First of all, we need to understand the way MySQL communicate between client and server. Both client and server are using MySQL protocol which is implemented by Connectors, MySQL Proxy and also the communication between master and slave replication servers. MySQL protocol supports the features like transparent encryption via SSL, transparent compression, a connection phase and as well as a command phase.
Both integers and strings are the basic data types that are used throughout the MySQL protocol. Whenever MySQL client and server wants to communicate to each other or sending the data, it will divide the data into packets with a maximum size of 16MB and also will prepend a packet header to every chunk. Inside each packet, there will be a payload which is where the data types (integers/strings) play their parts.
Considering the CLIENT_PROTOCOL_41 is enabled, for almost every command that the client sends to the server, the server will reply any of the following packets as a response:
OK_Packet | This is the signal for every successful command. |
ERR_Packet | The signal indicates an error for the packet. |
EOF_Packet | This packet contains a warning or status flag. |
How To Diagnose The Problems
Typically, there are two types of connection problems which is communication errors or aborted connections. Whenever any of these connection problems happen, the following sources of information are the good starting point for the troubleshooting and analysis:
Connection Errors And Possible Reasons
In the event of any connection errors happen and depending on the errors, it will increment the status counter for either Aborted_clients or Aborted_connects in the status variables. As taken from the MySQL documentation, Aborted_clients means the number of connections that were aborted because the client died without closing the connection properly. As for Aborted_connects, it means the number of failed attempts to connect to the MySQL server.
If you start the MySQL server with the --log-warnings option, chances are you probably would see the example of the following message in your error log. As you noticed, the message clearly said it relates to the abort connection, hence Aborted_connects status counter will be incremented in the status variable:
[Warning] Aborted connection 154669 to db: 'wordpress' user: 'wpuser' host: 'hostname' (Got an error reading communication packets)
Normally, unsuccessful connection attempts could happen due to the following reasons. When you noticed this, it possibly indicates that an unauthorized person is about to breach the database and you might want to look at it the soonest possible:
A client has no privileges to access the database.
A wrong credential has been used.
A connection packet that has incorrect information.
Due to the limit reached for connect_timeout to connect.
The status variable for Aborted_clients will be incremented by the server should a client manage to connect but get disconnected or terminated in an improper way. In addition to that, the server also will log an Aborted connection message to the error log. For this type of error, commonly it could be due to the following reason:
The client does not close the connection properly before exiting (does not call mysql_close ()).
The client has exceeded wait_timeout or interactive_timeout seconds.
The client program or application suddenly ended in the middle of data transfer.
Besides the reasons earlier, other likely reasons for both aborted connections and aborted clients issues could be related to any of the following:
TCP/IP configuration messed up.
The variable value is too small for max_allowed_packet.
Insufficient memory allocation for queries.
Faulty hardware like ethernets, switches, cables, etc.
Issues with the thread library.
Duplex syndrome issue whereby the transfer goes in burst-pause-burst-pause mode (if you use ethernet protocol with Linux, both half and full duplex).
How To Fix MySQL Communication Errors
Now that we learned a lot of potentialities that caused MySQL connection errors. Based on our experience, most of the time this issue is related to the firewall or network issues. It is also fair to say that it is not easy to diagnose this kind of issue. Nonetheless, the following solution might be helpful for you in solving this error:
If your application is relying on the wait_timeout to close the connection, it’s worth changing the application logic so that it’s properly closed at the end of any operation.
Making sure the value for max_allowed_packet is within the acceptable range so that the client does not receive any error related to the “packet too large”.
For connection delay issues that could be due to the DNS, it’s worth checking if you have skip-name-resolve enabled.
If you are using a PHP application or any other programming, the best is to make sure it does not abort the connections which are typically set at max_execution_time.
If you noticed a lot of TIME_WAIT notifications from netstat, it’s worth confirming that the connections are well managed on the application end.
If you are using Linux and suspect the issue is due to the networking, it’s best to check the networking interface by using ifconfig-a command and examine the output on the MySQL server for any error.
For ClusterControl users, you could enable Audit Log from the Cluster -> Security -> Audit Log. By enabling this feature, it could assist you to narrow down finding which query is the culprit.
Networking tools like tcpdump and Wireshark could be useful in identifying potential network issues, timeouts and resources issues for MySQL.
Regularly check on the hardware by making sure there are no faulty devices especially for ethernets, hubs, switches, cables etc. It’s worth replacing the faulty appliance to make sure the connection is good all the time.
Conclusion
There are a lot of reasons that could perhaps lead to the MySQL connection packet issues. Whenever this issue occurs, it will definitely affect the business and day to day operations. Even though this type of issue is not easy to diagnose and most of the time it is due to the network or firewall, it’s worth taking into consideration all the steps that have been suggested previously in order to fix the issue. We really hope this blog post could help you in some way especially when you face this problem.