Konu ile ilgili güzel bir yazi : https://maximelagresle.fr/posts/a-reason-for-unexplained-connection-timeouts-on-kubernetes-docker
Understanding the conntrack Command in Linux
The conntrack command-line tool is used to interact with the Netfilter connection tracking system in the Linux kernel. It allows system administrators to view, modify, and manage the stateful connection tracking table, which keeps track of all network sessions passing through the system (e.g., TCP sessions, UDP streams, ICMP requests).
This tool is essential for debugging firewalls (iptables or nftables), analyzing Network Address Translation (NAT) behavior, and troubleshooting network performance issues like connection table exhaustion.
Core Functions of conntrack
The command generally operates in one of several major modes, defined by its primary options:
- Viewing (
-Lor-G): Lists or gets specific tracked connections. - Event Monitoring (
-E): Displays real-time connection state changes (e.g., when a connection opens or closes). - Management (
-D,-I,-U): Deletes, inserts, or updates connection tracking entries. - Statistics (
-S): Displays performance and health metrics of the connection tracking system.
Common Options and Syntax
Here are the most frequently used command options and what they do:
1. Listing Connections (-L, --dump)
Lists all currently tracked connections in the kernel table.
Bash
sudo conntrack -L
- Filter by Proto:
sudo conntrack -L -p tcp(Lists only TCP connections). - Filter by IP:
sudo conntrack -L --src 192.168.1.50(Lists connections originating from this IP).
2. Monitoring Events (-E, --event)
Streams real-time updates as the kernel creates, updates, or destroys connections. This is excellent for live troubleshooting.
Bash
sudo conntrack -E
3. Deleting Connections (-D, --delete)
Forces the kernel to forget a connection. This is useful if a connection is hung or if you want to force a client to re-authenticate through a firewall.
Bash
sudo conntrack -D -p tcp --reply-dst 10.0.0.5
(Deletes TCP connections where the reply destination IP is 10.0.0.5)
4. Summary Statistics (-S, --stats)
Shows a breakdown of the connection tracking table’s health, including the number of searched, inserted, or ignored entries, and if there have been any packet drops due to a full table.
Bash
sudo conntrack -S
Understanding the Output Format
When you run sudo conntrack -L, you will see lines of output that look like this:
Plaintext
tcp 6 431999 ESTABLISHED src=192.168.1.100 dst=8.8.8.8 sport=54321 dport=53 src=8.8.8.8 dst=192.168.1.100 sport=53 dport=54321 [ASSURED] mark=0 use=1
Let’s break down exactly what each part of this output means:
| Output Token | Meaning |
tcp | The network layer protocol being used. |
6 | The decimal protocol number for TCP (UDP is 17, ICMP is 1). |
431999 | The Time-to-Live (TTL) in seconds. If no more packets are seen for this connection, the entry expires after this countdown. |
ESTABLISHED | The state of the connection (specific to the protocol, e.g., SYN_SENT, ESTABLISHED, TIME_WAIT). |
First Set:src=192.168.1.100 dst=8.8.8.8sport=54321 dport=53 | Original Direction: The source/destination IPs and ports expected for packets traveling from the initiator to the recipient. |
Second Set:src=8.8.8.8 dst=192.168.1.100sport=53 dport=54321 | Reply Direction: The source/destination IPs and ports expected for the return traffic. If NAT is occurring, these values will reflect the translated IPs/ports. |
[ASSURED] | A flag indicating that traffic has been seen in both directions, and this connection is unlikely to be dropped early if the connection table fills up. |
mark=0 | The Netfilter firewall mark (fwmark) associated with this connection, often used for advanced routing or traffic shaping. |
use=1 | The internal kernel reference count for this connection structure. |
Note on UDP Output
Because UDP is a stateless protocol, the output won’t have states like ESTABLISHED. Instead, you will see it transition from an unreplied state to marked as [ASSURED] once a two-way exchange is detected.
Important Diagnostic Files
While conntrack is the tool to interact with the system, the Linux kernel also exposes this data via the /proc filesystem:
/proc/net/nf_conntrack: Contains the raw list of currently tracked connections (similar toconntrack -L)./proc/sys/net/netfilter/nf_conntrack_max: The maximum number of connections the system can track simultaneously. If your server runs out of space here, it will start dropping new connections.
💡 Troubleshooting Tip: If your logs show
nf_conntrack: table full, dropping packet, you can check your limit usingsysctl net.netfilter.nf_conntrack_maxand temporarily increase it viasudo sysctl -w net.netfilter.nf_conntrack_max=131072.
Bir kac örnek
tcp 6 40 TIME_WAIT src=192.168.69.56 dst=192.168.69.56 sport=40350 dport=7473 packets=5 bytes=378 src=192.168.69.56 dst=192.168.69.56 sport=7473 dport=40350 packets=5 bytes=362 [ASSURED] mark=0 use=1
tcp 6 27 TIME_WAIT src=192.168.69.56 dst=192.168.69.56 sport=44284 dport=7472 packets=7 bytes=484 src=192.168.69.56 dst=192.168.69.56 sport=7472 dport=44284 packets=7 bytes=10141 [ASSURED] mark=0 use=1
tcp 6 101 TIME_WAIT src=172.31.134.173 dst=10.97.42.120 sport=51104 dport=3306 packets=20 bytes=1567 src=172.19.106.120 dst=172.31.134.173 sport=3306 dport=51104 packets=12 bytes=966 [ASSURED] mark=0 use=1
tcp 6 65 ESTABLISHED src=192.168.69.56 dst=10.96.0.1 sport=36940 dport=443 packets=17 bytes=13970 src=192.168.69.54 dst=192.168.69.56 sport=6443 dport=13694 packets=1 bytes=60 [ASSURED] mark=0 use=1
tcp 6 86372 ESTABLISHED src=192.168.69.56 dst=192.168.69.55 sport=47511 dport=179 packets=2370 bytes=145885 src=192.168.69.55 dst=192.168.69.56 sport=179 dport=47511 packets=2367 bytes=145786 [ASSURED] mark=0 use=1
tcp 6 91 ESTABLISHED src=192.168.69.56 dst=10.96.0.1 sport=43840 dport=443 packets=17 bytes=13970 src=192.168.69.54 dst=192.168.69.56 sport=6443 dport=9897 packets=1 bytes=60 [ASSURED] mark=0 use=1
ve S
conntrack -S
cpu=0 found=1 invalid=87 insert=0 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0 clash_resolve=1 chaintoolong=0
cpu=1 found=0 invalid=2 insert=0 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0 clash_resolve=0 chaintoolong=0
cpu=2 found=1 invalid=1 insert=0 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0 clash_resolve=0 chaintoolong=0
cpu=3 found=2 invalid=0 insert=0 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0 clash_resolve=0 chaintoolong=0