t-shoot SSL connection: problem F5

1, Get ssl debug log by:
On F5 cli:
#modify /sys db log.ssl.level value Warning
#tail -f /var/log/ltm
On client side:
#openssl s_client -connect -key client1.key -cert client1.crt
GET / HTTP/1.0

Check log and find error number

2, Add irule in the virtual server to get more information about client cert verification:
log LOCAL0.debug “nbr certs: [SSL::cert count] verifyResult: [SSL::verify_result] // [X509::verify_cert_error_string [SSL::verify_result]]”
set i 0
while {$i < [SSL::cert count]} {
log LOCAL0.debug "[X509::subject [SSL::cert $i]]"
incr i

3, ssldump for trouble ssl session
a) use find to find the path of keyfile
[user@xxx:/S1-green-P:Active:In Sync] / # find -iname *.key*
for example
b) ssldump -A -d -k -n -i
-A Print all fields
-d Show application data when private key is provided via -k
-k Private key file, found in /config/ssl/ssl.key/; the key file can be located under client SSL profile
-n Do not try to resolve PTR records for IP addresses
-i The capture VLAN name is the ingres VLAN for the TLS traffic
For example:
[user@xxx:/S1-green-P:Active:Changes Pending] / # ssldump -A -k ./config/filestore/files_d/partition_d/certificate_key_d/:partition:1410ws.verifiering.hsa.sjunet.org.key_98094_1 -i 0.0 host and port 443

key file is not needed to be specified when we only want to check ssl handshake information, not application data:
[user@xxx:/S1-green-P:Active:Changes Pending] / # ssldump -ni 0.0 host and port 443


Increase virtual disk for vCMP guest

Sometime we need increase virtual disk for vCMP guest in order to have space to install new module (for example, ASM). This the the steps to follow:

0, backup and save UCS for vCMP guest
1, set vCMP guest to “configured” statue
2, detach current virtual disk to the vCMP guest
3, delete virtual disk
4, modify the size of the virtual disk:

#tmsh modify sys db vcmp.vdisk.new_image_size value (default size is 100GB)
#tmsh show vamp disk (check disk status of current vcmp guest )
Be careful for this step, it may need to change another variable also to avoid to create the new disk with the same size as previously created disk:
#tmsh modify sys db vcmp.installer.use_vdisk_templates value disabled

5, set vCMP guest to “provisioned” state, this will create a new virtual-disk for the image

Currently there is a bug for F5 that this step will trapped in “stopping” pending statues for vCMP guest, there is no fix right now. the workaround is to either delete the vCMP guest totally and create a new vCMP guest, or restart vcmpd process, which will affect all vCMP guests running in this vCMP host.

6. set vCMP guest to “deployed” state to start up the guest

F5 vCMP upgrade summary

Process of software upgrading:
1, Sync of HA, then reactive license on each node(license should be reactivated in vCMP host when upgrading either vCMP host of vCMP guest)
2, backup and save UCS for each node
3, upload and install new image (new image can be uploaded in vCMP host only,vCMP guest has access to the image in vCMP host, but installation need to be done on each node)
4, reboot
5, check HA status, it may shows “disconnected” because software version do not match on the peer nodes.But failover should work.
6, failover
7, upgrade the previously active node and reload
8, Check HA status, it should be connected and requires sync
9, Sync

1, in vCMP solution, you may upgrade vCMP host first or vCMP guest first. Which one goes first does not really matter. Check software support matrix for the compatible vCMP host version and guest version

F5 APM study notes

1, port number used for LDAP protocol
A client starts an LDAP session by connecting to an LDAP server, called a Directory System Agent (DSA), by default on TCP and UDP port 389, or on port 636 for LDAPS.

2, LDAP vs LDAPs
LDAP (Lightweight Directory Application Protocol) and Secure LDAP (LDAPS) is the connection protocol used between application and the Network Directory or Domain Controller within the infrastructure.
Note, LDAP transmits communications in Clear Text, and LDAPS communication is encrypted and secure.

Kerberos uses UDP port 88 by default.  tacacs+ uses UDP port 49

4,Why use form-based client-initiated SSO authentication?

OCSP is a mechanism used to retrieve the revocation status of an X.509 certificate by sending the certificate information to a remote OCSP responder. This responder maintains up-to-date information about the certificate’s revocation status. OCSP ensures that Access Policy Manager always obtains real-time revocation status during the certificate verification process.

Access Policy Manager supports retrieving Certificate Revocation Lists (CRLs) from network locations (distribution points). A Certificate Revocation List Distribution Point (CRLDP) AAA server defines how to access a CRL file from a distribution point. A distribution point is either an LDAP Uniform Resource Identifier (URI), a directory path that identifies the location where the CRLs are published, or a fully qualified HTTP URL.

The LDAP and HTTP CRLDP mechanisms both work by sharing CRL information. There are two key problems with this:

CRLs can get large. This is not an issue for a server that is checking lots of certificates. However, where the RP is an end user checking the occasional certificate, retrieving the CRL can cause performance problems, particularly if the client is connected over a constrained link.
CRLs contain “next update” information, which enables an RP to detect if a CRL is out of date (and so a new CRL needs to be retrieved). The difficulty with this is that most RPs will use the cached CRL until “next update” time. Certificates are often revoked for reasons where quick response to the revocation is highly desirable.
OCSP (Online Certificate Status Protocol) was designed to address this, by providing a client/server protocol that enables an RP to check the status of a certificate. An OCSP server is indicated by a special certificate extension.

From the Authentication tab in VPE, select either Client Cert Inspection or On-Demand Cert Auth, and click Add item. Client Cert Inspection checks the result of an SSL handshake request that occurs at the start of an SSL session. On Demand Cert Auth performs an SSL re-handshake and checks the result. The CRLDP and OCSP Auth actions require certificate information made available by one of these access policy items.

6, client cert as SSO authentication
We can create “form based HTTP -client initiated” SSO for this purpose, so that we can customize a http header to insert client ssl certification into the http request.

7, The ssldump utility cannot decrypt traffic for which the handshake including the key exchange was not seen.

8, log setup:
tmsh modify sys dbkey reset-to-default

9, tools of t-shoot
tcpdump -vvv -s 0 -nni internal -w /var/tmp/www-ssl-server.cap host and net and port 8080

ssldump -r /var/tmp/www-ssl-client1.cap -k /config/filestore/files_d/Common_d/certificate_key_d/\:Common\:test.org.key_1 -M /var/tmp/client1.pms
Important: Not all ciphers provide the ability to decrypt SSL traffic using a utility such as ssldump. Depending on the cipher negotiated, the ssldump utility may not be able to derive enough information from the SSL handshake and the server’s private key to decrypt the application data. Examples of such SSL ciphers would be the Diffie-Hellman Ephemeral (DHE) cipher suites and export-grade RSA cipher suites. If it is necessary to decrypt application data for a virtual server, you can change the cipher suite used in your SSL profile so that traffic can be decrypted with ssldump. To do so, make a note of the cipher string currently configured in the SSL profile, then temporarily modify the SSL profile to specify a custom cipher string such as NONE:AES128-SHA. For specific configuration steps, refer to the examples appropriate for your version of BIG-IP, in the following articles:

1,To locate an active session ID from the BIG-IP command line, type the following command where  is the user’s user name:
sessiondump -allkeys | grep -i
2,To locate a session ID that is no longer active, search for the user name in the /var/log/apm file. The session ID is listed in the column to the left of the user name.

To display the session ID during the logon sequence, configure a message box action in the access policy with the session variable %{session.user.sessionid} in the Message field.
 1. Enable a web applications trace for the session ID you recorded in the previous step, by using the following command syntax, where  is the session ID recorded in step 3.
 tmsh modify /sys db "log.webapplications.trace.sessionid" { value "" }
 For example:
 tmsh modify /sys db "log.webapplications.trace.sessionid" { value "65a6b075" }
 2. Instruct the user to access the affected web application.
 3. After the user accesses the application, change directories to the directory of the webtrace by using the following command syntax:
 cd /var/tmp/WebAppTrace/
 In this command, note the following:
 •  is the user's session ID
 For example:
 cd /var/tmp/WebAppTrace/65a6b075/
 4. To create the webtrace.tgz file and add the webtrace data, type the following command:
 5. Copy the webtrace.tgz file your local workstation.
 6. On the local workstation, extract the webtrace.tgz file, and open the index.html file to view the trace file.
 7. To disable the web applications trace, type the following command:
 tmsh modify /sys db "log.webapplications.trace.sessionid" { value " " }
 The BIG-IP APM adtest tool can be used to test query and authentication to an Active Directory server. The basic syntax of the command can be viewed by typing adtest -h from the command line, and output will appear similar to the following example:
 [root@apm_01:Active] config # adtest -h
 The auth test type will test authentication. Active Directory authentication does not require administrative credentials. For example:
 adtest -t auth -h "adserver.example.com" -r "exampledomain.local" -A Administrator -W password123 -u jones -w letmein
 Test done: total tests: 1, success=1, failure=0
 If a query attempt yields the following error message, either the Active Directory domain controller is not resolving through DNS, or is not properly configured:
 ERROR: query with '(sAMAccountName=student1)' failed in ldap_sasl_interactive_bind_s(): Local error, SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Cannot find KDC for requested realm) (-2)

Useful F5 commands

1, When copy configuration from one unit to the other unit, or creating a lot of vips at the same time, it would be easier to do it via CLI:
a) Edit the configuration on editor
b) Copy and paste the configuration throught F5 cli terminal
user@(xxx)(cfg-sync In Sync)(/S1-green-P:Active)(/partition)(tmos)# load sys config from-terminal merge
Enter configuration. Press CTRL-D to submit or CTRL-C to cancel.

2, ssldump for trouble ssl session
a) use find to find the path of keyfile
[user@xxx:/S1-green-P:Active:In Sync] / # find -iname *.key*
for example
b) ssldump -A -d -k <key file> -n -i <capture VLAN> <traffic expression>
-A Print all fields
-d Show application data when private key is provided via -k
-k Private key file, found in /config/ssl/ssl.key/; the key file can be located under client SSL profile
-n Do not try to resolve PTR records for IP addresses
-i The capture VLAN name is the ingres VLAN for the TLS traffic
For example:
[user@xxx:/S1-green-P:Active:Changes Pending] / # ssldump -A -k ./config/filestore/files_d/partition_d/certificate_key_d/:partition:1410ws.verifiering.hsa.sjunet.org.key_98094_1 -i 0.0 host and port 443

3, Use command biptop on cli to check all currenct connections
For example:
[user@xxx:/S1-green-P:Active:In Sync] ~ # bigtop
QUERYING… | bits since | bits in prior | current
| Mar 4 00:16:03 | 0 seconds | time
BIG-IP ACTIVE |—In—-Out—Conn-|—In—-Out—Conn-| 14:28:57
xxx.sth.basefarm.net 691.0T 1.418P 8.543G 0 0 0

VIRTUAL ip:port |—In—-Out—Conn-|—In—-Out—Conn-|-Nodes Up–
/partition/ 668.2T 213.5T 704.4M 0 0 0 0

4, You can use the openssl command to verify the client certificate against the Trusted Certificate Authority bundle prior to importing it onto the BIG-IP system. For example, the following openssl command verifies the client certificate, client.crt, against the Trusted Certificate Authority bundle:

openssl verify -purpose sslclient -CAfile /path/to/trusted-ca-bundle.crt /path/to/client.crt

If the chain of trust can be established for the server certificate using the specified chain, the command returns output similar to the following example:

client.crt: OK
5, Use tcpdump to show more tmm information,
for example, to check routed vip:
tcpdump -vs0 -i 0.0:nnn host
it shows below, the red part showed routed vip when received request from client.
16:28:44.474154 IP (tos 0x0, ttl 255, id 37043, offset 0, flags [DF], proto: TCP (6), length: 52) vip.http > host.38934: ., cksum 0x4282 (incorrect (-> 0x8e1f), ack 159 win 4297 <nop,nop,timestamp 913578425 3245706819> out slot1/tmm2 lis=/partition/xxxhttp-vip flowtype=64 flowid=5701C0361D00 peerid=5

failover of F5 LTM


1, Normally we use HA group (fast failover) because failover when using VLAN fail-safe or Gateway fail-safe will take about 10 secs. HA group failover happens almost immediately.

2, We are using version 11.6 and I have found that we need change failover method (in traffic group) to HA group in order to make HA group failover works.
You may check HA score with command show /sys ha-group

When you have failover method as HA order configured, it shows like this:
LB(Active)(/Common)(tmos)# show /sys ha-group detail

Sys::HA Group: lb01-ha
State enabled
Active Bonus 10
Score 0

| Sys::HA Group Trunk: nko-lb01-ha:lb-trunk
| Threshold 1
| Percent Up 100
| Weight 20

HA group score is always 0, no failover will happen even if you shutdown the trunk. When you change failover method to HA group, then it shows as below:
LB(Active)(/Common)(tmos)# show /sys ha-group

Sys::HA Group: lb01-ha
State enabled
Active Bonus 10
Score 20

| Sys::HA Group Trunk: nko-lb01-ha:lb-trunk
| Threshold 1
| Percent Up 100
| Weight 20
| Score Contribution 20

3, HA failover unicast configuration
Always you need configure 2 ips in order to make failover works: MGMT IP and failover IP. Especially failover IP is in a dedicated failover link among LTM nodes.
Removing mgmt IP will cause both LTM nodes switch to active statue even failover ip is configured and reachable. Removing failover IP will cause the same issue even if the mgmt ip is configured and reachable.

Sync and mirror ip can be configured as failover IP only, mgmt ip is not necessary here.

4, What will triger failover?

refer to above link:

The BIG-IP system initiates failover according to any of several events that you define. These events fall into these categories:

System fail-safe
With system fail-safe, the BIG-IP system monitors various hardware components, as well as the heartbeat of various system services. You can configure the system to initiate failover whenever it detects a heartbeat failure.
Gateway fail-safe
With gateway fail-safe, the BIG-IP system monitors traffic between an active BIG-IP system in a device group and a pool containing a gateway router. You can configure the system to initiate failover whenever some number of gateway routers in a pool of routers becomes unreachable.
VLAN fail-safe
With VLAN fail-safe, the BIG-IP system monitors network traffic going through a specified VLAN. You can configure the system to initiate failover whenever the system detects a loss of traffic on the VLAN and the fail-safe timeout period has elapsed.
HA groups
With an HA group, the BIG-IP system monitors trunk, pool, or cluster health to create an HA health score for a device. You can configure the system to initiate failover whenever the health score falls below configurable levels.
When you enable auto-failback, a traffic group that has failed over to another device fails back to a preferred device when that device is available. If you do not enable auto-failback for a traffic group, and the traffic group fails over to another device, the traffic group remains active on that device until that device becomes unavailable.

5, failover methods:

refer to link https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/bigip-device-service-clustering-admin-11-5-0/8.html

  • Select Load Aware when the device group contains heterogeneous platforms and you want to ensure that a traffic group fails over to the device with the most capacity at the moment that failover occurs.
  • Select HA Order to cause the traffic group to fail over to the first available device in the Failover Order list.
  • Select HA Group to cause the BIG-IP system to trigger failover based on an HA health score for the device.

302 study

Types of Resource Record

1, SOA start of authority: created one SOA when created one master zone file

2, A record: for each ip address of the machine (AAAA record for ipv6 address)

3, C name: alias name of a host name registered in A record

4, D name: for ipv6

5,HINFO: host information (OS, hardware) for DNS servers

6, MX(mail exchanger), mail system for a given domain

7,PTR (pointer), associate hostname with a give ip address, used for reverse name lookups.

8,NS (name server ) for a given zone