Increase virtual disk for vCMP guest

Sometime we need increase virtual disk for vCMP guest in order to have space to install new module (for example, ASM). This the the steps to follow:

0, backup and save UCS for vCMP guest
1, set vCMP guest to “configured” statue
2, detach current virtual disk to the vCMP guest
3, delete virtual disk
4, modify the size of the virtual disk:

#tmsh modify sys db vcmp.vdisk.new_image_size value (default size is 100GB)
#tmsh show vamp disk (check disk status of current vcmp guest )
Be careful for this step, it may need to change another variable also to avoid to create the new disk with the same size as previously created disk:
#tmsh modify sys db vcmp.installer.use_vdisk_templates value disabled

5, set vCMP guest to “provisioned” state, this will create a new virtual-disk for the image

Currently there is a bug for F5 that this step will trapped in “stopping” pending statues for vCMP guest, there is no fix right now. the workaround is to either delete the vCMP guest totally and create a new vCMP guest, or restart vcmpd process, which will affect all vCMP guests running in this vCMP host.

6. set vCMP guest to “deployed” state to start up the guest

Advertisements

F5 vCMP upgrade summary

Process of software upgrading:
1, Sync of HA, then reactive license on each node(license should be reactivated in vCMP host when upgrading either vCMP host of vCMP guest)
2, backup and save UCS for each node
3, upload and install new image (new image can be uploaded in vCMP host only,vCMP guest has access to the image in vCMP host, but installation need to be done on each node)
4, reboot
5, check HA status, it may shows “disconnected” because software version do not match on the peer nodes.But failover should work.
6, failover
7, upgrade the previously active node and reload
8, Check HA status, it should be connected and requires sync
9, Sync

Notes:
1, in vCMP solution, you may upgrade vCMP host first or vCMP guest first. Which one goes first does not really matter. Check software support matrix for the compatible vCMP host version and guest version

F5 APM study notes

1, port number used for LDAP protocol
A client starts an LDAP session by connecting to an LDAP server, called a Directory System Agent (DSA), by default on TCP and UDP port 389, or on port 636 for LDAPS.

2, LDAP vs LDAPs
LDAP (Lightweight Directory Application Protocol) and Secure LDAP (LDAPS) is the connection protocol used between application and the Network Directory or Domain Controller within the infrastructure.
Note, LDAP transmits communications in Clear Text, and LDAPS communication is encrypted and secure.

2,
Kerberos uses UDP port 88 by default.  tacacs+ uses UDP port 49

4,Why use form-based client-initiated SSO authentication?

5, OCSP vs CLRDP
OCSP is a mechanism used to retrieve the revocation status of an X.509 certificate by sending the certificate information to a remote OCSP responder. This responder maintains up-to-date information about the certificate’s revocation status. OCSP ensures that Access Policy Manager always obtains real-time revocation status during the certificate verification process.

Access Policy Manager supports retrieving Certificate Revocation Lists (CRLs) from network locations (distribution points). A Certificate Revocation List Distribution Point (CRLDP) AAA server defines how to access a CRL file from a distribution point. A distribution point is either an LDAP Uniform Resource Identifier (URI), a directory path that identifies the location where the CRLs are published, or a fully qualified HTTP URL.

The LDAP and HTTP CRLDP mechanisms both work by sharing CRL information. There are two key problems with this:

CRLs can get large. This is not an issue for a server that is checking lots of certificates. However, where the RP is an end user checking the occasional certificate, retrieving the CRL can cause performance problems, particularly if the client is connected over a constrained link.
CRLs contain “next update” information, which enables an RP to detect if a CRL is out of date (and so a new CRL needs to be retrieved). The difficulty with this is that most RPs will use the cached CRL until “next update” time. Certificates are often revoked for reasons where quick response to the revocation is highly desirable.
OCSP (Online Certificate Status Protocol) was designed to address this, by providing a client/server protocol that enables an RP to check the status of a certificate. An OCSP server is indicated by a special certificate extension.

From the Authentication tab in VPE, select either Client Cert Inspection or On-Demand Cert Auth, and click Add item. Client Cert Inspection checks the result of an SSL handshake request that occurs at the start of an SSL session. On Demand Cert Auth performs an SSL re-handshake and checks the result. The CRLDP and OCSP Auth actions require certificate information made available by one of these access policy items.

6, client cert as SSO authentication
We can create “form based HTTP -client initiated” SSO for this purpose, so that we can customize a http header to insert client ssl certification into the http request.

7, The ssldump utility cannot decrypt traffic for which the handshake including the key exchange was not seen.

8, log setup:
tmsh modify sys dbkey reset-to-default
dbkey:
log.access.db
log.access.syslog

9, tools of t-shoot
tcpdump -vvv -s 0 -nni internal -w /var/tmp/www-ssl-server.cap host 192.168.22.33 and net 10.1.1.0/24 and port 8080

ssldump -r /var/tmp/www-ssl-client1.cap -k /config/filestore/files_d/Common_d/certificate_key_d/\:Common\:test.org.key_1 -M /var/tmp/client1.pms
Important: Not all ciphers provide the ability to decrypt SSL traffic using a utility such as ssldump. Depending on the cipher negotiated, the ssldump utility may not be able to derive enough information from the SSL handshake and the server’s private key to decrypt the application data. Examples of such SSL ciphers would be the Diffie-Hellman Ephemeral (DHE) cipher suites and export-grade RSA cipher suites. If it is necessary to decrypt application data for a virtual server, you can change the cipher suite used in your SSL profile so that traffic can be decrypted with ssldump. To do so, make a note of the cipher string currently configured in the SSL profile, then temporarily modify the SSL profile to specify a custom cipher string such as NONE:AES128-SHA. For specific configuration steps, refer to the examples appropriate for your version of BIG-IP, in the following articles:

1,To locate an active session ID from the BIG-IP command line, type the following command where  is the user’s user name:
sessiondump -allkeys | grep -i
2,To locate a session ID that is no longer active, search for the user name in the /var/log/apm file. The session ID is listed in the column to the left of the user name.

To display the session ID during the logon sequence, configure a message box action in the access policy with the session variable %{session.user.sessionid} in the Message field.
 1. Enable a web applications trace for the session ID you recorded in the previous step, by using the following command syntax, where  is the session ID recorded in step 3.
 tmsh modify /sys db "log.webapplications.trace.sessionid" { value "" }
 For example:
 tmsh modify /sys db "log.webapplications.trace.sessionid" { value "65a6b075" }
 2. Instruct the user to access the affected web application.
 3. After the user accesses the application, change directories to the directory of the webtrace by using the following command syntax:
 cd /var/tmp/WebAppTrace/
 In this command, note the following:
 •  is the user's session ID
 For example:
 cd /var/tmp/WebAppTrace/65a6b075/
 4. To create the webtrace.tgz file and add the webtrace data, type the following command:
 /usr/bin/webtrace.finish
 5. Copy the webtrace.tgz file your local workstation.
 6. On the local workstation, extract the webtrace.tgz file, and open the index.html file to view the trace file.
 7. To disable the web applications trace, type the following command:
 tmsh modify /sys db "log.webapplications.trace.sessionid" { value " " }
 The BIG-IP APM adtest tool can be used to test query and authentication to an Active Directory server. The basic syntax of the command can be viewed by typing adtest -h from the command line, and output will appear similar to the following example:
 [root@apm_01:Active] config # adtest -h
 The auth test type will test authentication. Active Directory authentication does not require administrative credentials. For example:
 adtest -t auth -h "adserver.example.com" -r "exampledomain.local" -A Administrator -W password123 -u jones -w letmein
 Test done: total tests: 1, success=1, failure=0
 If a query attempt yields the following error message, either the Active Directory domain controller is not resolving through DNS, or is not properly configured:
 ERROR: query with '(sAMAccountName=student1)' failed in ldap_sasl_interactive_bind_s(): Local error, SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Cannot find KDC for requested realm) (-2)

Windows client can not ping itself after any connect vpn is up

Windows client can not ping itself with IP address assigned by address pool that is configured in Cisco ASA.  On the same time, Macbook client can ping its IP address or hostname without any problem.

This happens when using split tunnel. In order to make windows client able to ping itself we need include IP address pool into the split access list.

Use ansible to remotely restart mac machine

1, allow ssh connection on Mac machine:

systempreference ->sharing->remote inlogging-> allow users that can be remotely access to this machine

2, create yaml task file:

  - name: restart the mac host
    become: yes
    shell: sleep 2 && /sbin/shutdown -r now "Ansible restart the host"
    ansible_become_pass: sudopassword
    async: 1
    poll: 0
    ignore_errors: true

  - name: "Wait for reboot"
    local_action: wait_for host={{ ansible_default_ipv4.address }} port=22 delay=10 state=started
    become: false

“shutdown” command requires super user privilege, that can be achieved by become command, use ansible_become_pass to pass sudo password to the command. There is option to avoid writing sudo password in the yaml file, referring to ansible-vault.

Add “sleep 2” in front of the shutdown command to avoid “ssh connection lost” error when implement sensible-playbook, which will cause the interruption of the playbook.

After this task, set wait_for task applied on the local controller, to periodically check ssh connectivity of the remote Mac host.

 

 

Connect to Windows host using domain account from ansible controller not joined into AD domain

The discussion we had previously is only useful to manage a windows PC with local username/password. In order to manage a domain windows PC we have to install kerberos module for Ansible. I will give the guide regarding the setup of ansible controller to manage a domain windows PC while ansible controller itself is not within the domain.

1, Even through ansible controller is not necessary to join the domain, it is necessary that  machine of ansible controller can reach domain DNS and NTP services.  We can configure NTP server as the same domain NTP server if possible.

To get NTP used with windows host, enter “w32tm /query /peers” in cmd of window.

2, Join domain network, acquired IP address, WINS, DNS etc från domain network, this can happens automatically via DHCP when connected ansible controller to the domained network.

3, Install kerberos for ansible (example for Mac OS X)

pip install request kerberos
pip install pywinrm[kerberos]

4, Configure kerberos

edit etc/krb5.conf file, if krb5.conf is not existed, create one:

below is an example of krb5.conf file:

mac-c02t6npagtfj:etc grayin$ vi krb5.conf

        krb5_run_aklog = 1
        krb5_aklog_path = /usr/local/krb5/bin/aklog
        default_lifetime = 25h
        telnet = {
                autologin = 1
        }
        xdm = {
               retain_ccache = 1
                afs_retain_token = 1
        }
       pam = {
                ticket_lifetime = 90000
                renew_lifetime = 90000
                forwardable = true
        }
[libdefaults]
        forwardable = true
        default_realm = LFAD.LFNET.SE
[realms]
        DOMAIN.COM = {
                kdc = E00DC0008.DOMAIN.COM
                admin_server = E00DC0008.DOMAIN.COM
                default_domain = domain.com
        }
[domain_realm]
         .domain.com = DOMAIN.COM
         domain.com = DOMAIN.COM
[kdc]
        profile = /etc/kdc.conf
[logging]
        default = SYSLOG:INFO:LOCAL6

 

[realms] and [domain_realm] are the area where new domain information need to be added. Using echo %username% will allow you to identify the authenticating domain controller.If you just desire to identify which domain controller the user retrieved group policies from you can type gpresult /r.

5, Test kerberos connection

#kinit user@MY.DOMAIN.COM

#To see what tickets if any you have acquired, use the command klist

#klist

mac-c02t6npagtfj:ansible_test grayin$ kinit username@DOMAIN.COM
username@DOMAIN.COM's password: 

'mac-c02t6npagtfj:ansible_test grayin$ klist
Credentials cache: API:FA05485A-8E6B-45F8-9526-D4B4EEDA5D1D
        Principal: username@DOMAIN.COM
  Issued                Expires               Principal
Mar  8 14:49:48 2018  Mar  9 00:49:44 2018  krbtgt/DOMAIN.COM@DOMAIN.COM

Once you have a valid ticket, you can check to ensure that everything is working as expected from command line. To test this, make sure that your inventory looks like the following:

[windows]
win01.DOMAIN.COM

[windows:vars]
ansible_user = username@DOMAIN.COM
ansible_winrm_transport=kerberos
ansible_connection = winrm
ansible_port = 5986
# The following is necessary for Python 2.7.9+ when using default WinRM self-signed certificates:
ansible_winrm_server_cert_validation = ignore

Especially “ansible_winrm_transport” can be changed to ssl if you want to authenticate with a local account of windows PC.  When connecting to windows host there are several authentication options that can be used, refer to here

Option Local Accounts Active Directory Accounts Credential Delegation
Basic, ssl Yes No No
Certificate Yes No No
Kerberos No Yes Yes
NTLM Yes Yes No
CredSSP Yes Yes

6, Make sure that managed windows pc is listening on 5986 and the firewall on PC is turned off or traffic to/from port 5986 is allowed. To check the state of configuration settings, type winrm get winrm/config. To check if tcp 5986 is listening and no firewall blocking, type telnet win01 5986 from ansible controller.

It can be a very headache to turn off windows defenders when “off” button is grey i no matter windows security center or windows firewall&network, or windows defender. the most efficient way for me is to go directly to regedit and change value there.

HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows Defender find “start” value from domain/private/public separately and set the value from 1 to 0.

Of course you can allow winrm https 5986 application from firewall allow application setup. But this option is grey too in my test environment. Then we can try to change configuration from pgedit.msc, to use group policy editor to create incoming traffic rules for winrm:

Go to gpedit.msc, computer configuration ->windows settings ->security settings -> windows firewall with advanced security -> inbound rules add rules

7, T-SHOOT

In my test I got stuck in “password incorrect” message when set correct password in the inventory sensible_password=password, even though I am pretty sure that I have typed the correct password.

mac-c02t6npagtfj:ansible_test grayin$ ansible-playbook -i host main.yaml
PLAY [Hit a Specific Host on the Server] *************************************************************************************************************************************************************************************************
TASK [Gathering Facts] *******************************************************************************************************************************************************************************************************************
fatal: [D75C0004.LFAD.LFNET.SE]: UNREACHABLE! => {"changed": false, "msg": "Kerberos auth failure: a940bf@LFAD.LFNET.SE's password: \nkinit: Password incorrect", "unreachable": true}
to retry, use: --limit @/Users/grayin/ansible_test/main.retry

Here is the discussion regarding this problem. It seems that  winrm module work if you get a kerberos token via kinit before executing ansible, even if the host isn’t joined to the domain. It fails though if you try to rely on the ansible_user/ansible_password combination.

I also encountered another problem that I must have account which owns administrative right for window computer in order to get connection successfully, Otherwise ansible kerberos connection will get error for either  time_out or credential refused or need paaeven though I have a valid kerberos toke listed by klist.

 

Install multiple versions of python i OS X

If you need install multi python versions in the host, and use different version of python for different application, it is very useful to use pyenv to manage different version installed in the host. For example, I have python 3.6 and 2.7 installed in my Mac, where python –version shows default version is 3.6

mac-c02t6npagtfj:ansible_test grayin$ python –version

Python 3.6.4

But I want to change my default python to version 2.7 in my Mac, and use pip in package 2,7 to install new packages in python2.7.

What I have done is to install pyenv first, then use pyenv to reinstall version 2,7 and 3.6, after that, change version of python used in the host:

$ brew update
$ brew install pyenv

To enable pyenv in your Bash shell, you need to run:

$ eval "$(pyenv init -)"
After this, we can use pyenv to reinstall python
$ pyenv install 2.7.14
$ pyenv install 3.6.4

 

And you can switch between python versions with the command:

$ pyenv global 2.7.14

Also you can set a python version for the current directory with:

$ pyenv local 3.6.4
mac-c02t6npagtfj:ansible_test grayin$ pyenv versions
  system
  2.7
* 2.7.14 (set by /Users/grayin/.pyenv/version)
  3.6.4
After this:
mac-c02t6npagtfj:ansible_test grayin$ python --version
Python 2.7.14
mac-c02t6npagtfj:ansible_test grayin$ pip --version
pip 9.0.1 from /Users/grayin/.pyenv/versions/2.7.14/lib/python2.7/site-packages (python 2.7)

Even pip as default is attached to python2.7 now.

Pyenv can be helpful to avoid home-brew lost track of python

Sometime people may feel very confused that he/she has installed a package already but when importing it in the script, it would show that package is not installed or can not find the module.  This is because that when pip is attached to version 2.7, all package installed using pip will be installed for version 2.7, when pip is attached to version 3.6, all package installed using pip will be for version 3.6.  If version 2.7 is used but actually pip is attached to version 3.6, package will be installed in version 3.6.

Another alternative to address this issue is to use different /path/pip to install the package for the different version of python. Anyhow I have not tested this option.

Some also recommend the easier way for using pip in different version installation:

$ pip2.6 install otherpackage
$ pip2.7 install mybarpackage

However, this does not work for me:

mac-c02t6npagtfj:ansible_test grayin$ pip2.7 --version
pip 9.0.1 from /Users/grayin/.pyenv/versions/2.7.14/lib/python2.7/site-packages (python 2.7)
mac-c02t6npagtfj:ansible_test grayin$ pip3.6 --version
pyenv: pip3.6: command not found
The `pip3.6' command exists in these Python versions:
  3.6.4
mac-c02t6npagtfj:ansible_test grayin$ pip3.6.4 --version
-bash: pip3.6.4: command not found