Veeam Replication Job Trouble Ticket # 1

There was an email notification received from the Veeam B&R server stating that one of the VM replication job had failed with the following error:

Error: Failed to open VDDK disk [[Datastore_Name] VM_replica/VM2-000001.vmdk] ( is read-only mode – [false] ) Logon attempt with parameters [VC/ESX: [vCSA_Name];Port: 443;Login: [domain\account_name];VMX Spec: [moref=vm-40195];Snapshot mor: [snapshot-112166];Transports: [nbd];Read Only: [false]] failed because of the following errors: Failed to open disk for write. Failed to download disk. Reconnectable protocol device was closed. Failed to upload disk. Age

The first thing done was to log into the Veeam backup server and open the Veeam Backup & Replication console.   veeam_b_r_console

Under the Home, navigate to the Replicas, search for the VM. Then highlight it and right click and open the properties. It will show all the replica jobs which are snapshots on the replica VM in vSphere.

 

The two last replica job shows that they were incomplete. This sometimes indicate that there was a problem with the snapshot that was created. Logon to the vSphere web client and check the snapshot manager for the replica VM (the default naming convention is “VMName_replica“). Open the option Snapshots–>Manage Snapshots and you will expand the snapshots until you reach the last two set before “You are here”.

vsphere_snapshot

In the image, you will notice that the snapshot names “Veeam Replica Working Snapshot” which indicate that the snapshot was not completed. To eliminate the job continue failing is to delete all the snapshots with the title “Veeam Replica Working snapshot” and rerun the job from veeam so it can recreate the snapshot. If these snapshot are allowed to remain, the Veeam replica job will continue to fail.

I hope this help to resolve some of the veeam replication job failures.

Computer blocked by Port Security

My colleague was working on an issue where a computer was communicating on the network through a Cisco switch that had port security enabled. As usually, he disables port security and re-enables it in an attempt to get the new mac address registered on the port. The computer (A) was now able to get an IP address from the DHCP server but unfortunately, the computer (A) still was not able to communicate with the server. In an effort to identify whether the problem was with computer (A) or the network port, another computer (B) was connected to the port which was successful. But when computer (A) was reconnected to the same port, there was still no communication. All network settings on Computer (A) were verified to be correct but still no break through. The same computer (A) was taken to another switch in another location and surprisingly it worked. So hhhmmm….what could be the problem? It was escalated to me to investigate further as he could not identify what was the root of the problem.

I started to ask the question “What is working?”

The responses were:

  1. Other devices were communicating when connected to the port
  2. The computer (A) is getting an IP address from the DHCP server
  3. The computer (A) is able to communicate on other switches

Next question, did they try the computer (A) on another port on the same switch? Response was No.

Anyways, everything was connected back to how it was before and then I started the troubleshooting.

The following observations were made:

  1. Port-security was enabled on the entire switch but disabled on this troubled port.
  2. The port duplex and speed was ok (Full/100)
  3. The computer (A) got an dynamic IP address from the DHCP server
  4. The computer (A) mac address was not showing on the port when I check the MAC address table for the problem port (f0/42)
  5. Port security sticky address listing showed that computer (A) mac address was tied to port f0/22

What do you think the root of the problem is?

The computer (A) is not able to communicate with anything on the network because the mac address of the computer (A) was tied to another port so when devices response to the packet, it is returning to the wrong port. so the solution was to remove the sticky address and allow the switch to relearn it over the new connected port (f0/42). Another question resonate in my mind was “How did it get an IP address from the DHCP server?”. But if you review how devices acquire an IP address, it is using broadcast. Therefore, the computer will always get an IP address but it would not be able to communicate with other devices.

I hope this troubleshooting scenario helps someone with understanding port-security on a Cisco switch.

Polycom phone “go off hook” error message

I was called to assist with an incident ticket whereby a Polycom SoundStation IP 6000 conference phone was not able to make or receive calls.

The first thing I did was to check what was the error message when trying to call the phone. The error displayed was “go off hook” and initial impression was that the phone was plugged out but based on the error this could not be the case.

When I check the phone screen, it was showing that the phone was being forwarded to an extension that did not exist hence the error. A little background knowledge, this phone is connected to an Alcatel VoIP Telephony system using the sip protocol. The way this system works is that is an extension does not exist in the system, it immediately give a message “go off hook” so it was strange when I got this message when testing the issue.

So the solution as you probably conclude is to disable the forwarding (which should not even be on a conference phone).

Active Directory Domain Services Offline and Defragmentation

There is a tradition method of taking the Microsoft Active Directory Domain Services (AD DS) offline to perform maintenance on the database is using the Directory Service Restore Mode (DSRM). The steps are listed below:

  1. Log into your Domain Controller (DC)
  2. Open the command prompt
  3. Run the command:

    bcdedit /set safeboot dsrepair

  4. Reboot the DC so the it can enter into safe mode

To exit the safe mode, run the command

bcdedit /deletevalue safeboot

and then reboot for the DC to return to normal mode.

Another method of taking the AD DS offline is by stopping the AD DS service using the services applet (services.msc).

Now to perform defragmentation on the AD DS database is to use the NTDS utility.

Execute the following command at the prompt:

ntdsutil

activate instance ntds

files

compact to c:\

integrity

quit

quit

copy c:\ntds.dit c:\windows\ntds\ntds.dit

del c:\windows\ntds\*.log

Restart the AD DS services.

Restart-Service NTDS

Please note that you don’t have to restart the server after the maintenance of the database.

 

 

Useful Windows PowerShell (PS) command

This blog is an opportunity to record all the PS commands I used in my day-to-day management of my environment.

Get-Help [*command*]

The get-help is a useful command used to get details of the command executed.

Get-Command [*command*]

List the A record of the server

Get-DnsServerRecord -Name [Hostname]-RRType [A]

Add an server name with the IP address to the DNS server. This command should be ran from the DNS server remotely or locally.

Add-DnsServerRecordA -Name [HostName] -IPAddress [IP] -ZoneName [domain-name] -AllowUpdate

Shutdown the server

Stop-Computer -ComputerName [Servername] -Force

Restart the server

Restart-Computer -ComputerName [Servername] -Force

Start a remote PowerShell session on server

Enter-PSSession [Servername]

Exit the existing PowerShell remote session

Exit-PSSession

A Similiar command to ping and trace route in Powershell

Test-NetConnection [-traceroute] [destination-name or address]

A command to execute PS commands on a remote host

Invoke-Command -ComputerName -ScriptBlock { [Powershell command goes here] }

List all the services on a computer

Get-Service

List all the Windows share (SMB) session on a computer

Get-SmbSession

Get the AD User information filtered by Name or login

Get-ADUser -Identity [CID] -Filter ‘Name -like “Firstname*”‘

Set the AD User information

Set-ADUser -Identity [CID] -ParameterField “Value”

Unlock User Account

Unlock-ADAccount -Identity [CID]

Disable User Account

Disable-ADAccount Identity [CID]

Enable User Account

Enable-ADAccount Identity [CID]

Reset User Account Password

Set-ADAccountPassword -Identity [CID] -Reset -NewPassword (ConvertTo-SecureString -ASPlainText “MyPassword” -Force)

Set the user account password settings

Set-ADUser -ChangePasswordAtLogon $true -CannotChangePassword $true -AccountExpirationDate “mm/dd/yyyy” -PasswordNeverExpires $true

Search AD for User Account that has expired password

Search-ADAccount -UsersOnly -PasswordExpired | FT SamAccountName, Name, LastLogonDate

Search AD for Locked Out Accounts

Search-ADAccount -UsersOnly -LockedOut | FT SamAccountName, Name, LastLogonDate

Search AD for Expiring Accounts within 60 days

Search-ADAccount -usersonly -AccountExpiring -TimeSpan 60.00:00 | FT SamAccountName,Name,AccountExpirationDate

Search AD for Inactive Accounts

Search-ADAccount -UsersOnly -AccountInactive | FT SamAccountName, Name, LastLogonDate

Search AD for Disabled Accounts

Search-ADAccount -UsersOnly -AccountDisabled | FT SamAccountName, Name

Search AD for Expired Accounts

Search-ADAccount -UsersOnly -AccountExpired | FT SamAccountName, Name, AccountExpirationDate

Search AD for Accounts password set to never expire

Search-ADAccount -UsersOnly -PasswordNeverExpires | FT SamAccountName, Name, AccountExpirationDate,LastLogonDate

Retrieve AD Group Information based on Filter

Get-ADGroup -Filter ‘Name -like “[string]*”‘ -Properties canonicalName | FT canonicalname

Retrieve members for a specific AD Group

Get-ADGroupMember -Identity ‘[groupname]’ | FT SamAccountName, Name

Retrieve the properties of an object

Verb-Object -properties *

Add Members to an AD Group

Add-ADGroupMember -Identity ‘GroupName’ -Members [UserName]

Create an New OU

New-ADOrganizationUnit -Name [OU-Name] -Path “[DC=Domain,DC=com]”

Create a user in the New OU

New-ADUser -Name “Firstname Lastname” -SamAccountName [CID] -Path “OU=,DC=,D=com”

Create a New AD Group

New-ADGroup -Name “GroupName” -SamAccountName [GroupName] -GroupCategory [Security/Distribution] -GroupScope [DomainLocal/Global/Universal] -Path “OU=,DC=domain,DC=com”

Create a managed service account

New-ADServiceAccount -Name [AccountName] -RestrictToSingleComputer

Add-ADComputerServiceAccount -Identity [COMPNAME] -ServiceAccount [AccName]

Test-ADServiceAccount -Identity [AccName]

Get Windows Security event of lockout accounts from PDC

Get-ADDomainController -Filter “OperationMasterRoles -like ‘*PDCEmulator*'” | FT Hostname, OperationMasterRoles, Site, OperatingSystem

Get-WinEvent -ComputerName [PDCEmulaor] -FilterHashtable @{LogName=’Security’;Id=4740} -ErrorAction Stop

Show the Default Domain Password Policy

Get-ADDefaultDomainPasswordPolicy

Change the Default Domain Password Policy

Get-ADDefaultDomainPasswordPolicy | Set-ADDefaultDomainPasswordPolicy -LockoutDuration -LockoutThreshold -MaxPasswordAge -MinPasswordAge -MinPasswordLength -ComplexityEnabled [$true/$false] -PasswordHistoryCount -LockoutObservationWindow

 

VCP65-DCV – Objective 7.5 – Troubleshoot HA and DRS Configurations and Fault Tolerance

The following reference material from the vSphere 6.5 online documentation will assist you in covering the main information that is needed to know the topics in this objective:

  1. How vSphere HA works
  2. Troubleshooting vSphere HA host states
  3. Troubleshooting Availability
  4. Configure Proactive HA
  5. DRS troubleshooting information
  6. Migration with vMotion
  7. Migration with Storage vMotion
  8. Troubleshooting Storage DRS
  9. Providing Fault Tolerant for Virtual Machines
  10. Troubleshooting Fault Tolerant Virtual Machines
  11. DRS Cluster Validity
  12. Overview Performance Charts for Clusters

VCP65-DCV – Objective 7.4 – Troubleshoot Virtual Machines

The following reference material from the vSphere 6.5 online documentation will assist you in covering the main information that is needed to know the topics in this objective:

  1. Overview Performance Charts
  2. Troubleshoot and Enhance Performance
  3. CPU Virtualization Basics
  4. Memory virtualization Basics
  5. Working with Advanced and Custom Charts
  6. Storage Device Panel
  7. know the commands:
    1. esxtop
  8. EVC requirements for Hosts
  9. Change the EVC mode for a Cluster

VCP65-DCV – Objective 7.3 – Troubleshoot vSphere Upgrades and Migrations

The following reference material from the vSphere 6.5 online documentation will assist you in covering the main information that is needed to know the topics in this objective:

  1. VMware KB 1011641 – vCenter server
  2. VMware KB 653 – ESXi Host
  3. Troubleshooting vCenter Server Installation or Deployment
  4. vCenter Server Upgrade Failed when unable to Stop tomcat service.
  5. Microsoft SQL Database set to unsupported compatibility Mode Causes vCenter server installations or Upgrade to Fail
  6. Export a vCenter server support bundle for troubleshooting
  7. Troubleshooting vCenter server
  8. Introduction to vSphere Upgrade
  9. Upgrade ESXi Hosts
  10. Configure logging options

VCP65-DCV – Objective 7.2 – Troubleshoot vSphere Storage and Networking

The following reference material from the vSphere 6.5 online documentation will assist you in covering the main information that is needed to know the topics in this objective:

  1.  Know the commands
    1. esxtop
  2. Create a Datastore Cluster
  3. Setting the Aggressiveness Level for Storage DRS
  4. Troubleshooting Networking
  5. Troubleshooting Storage
  6. VMFS metadata Updates

VCP65-DCV – Objective 7.1 – Troubleshoot vCenter Server and ESXi Hosts

The following reference material from the vSphere 6.5 online documentation will assist you in covering the main information that is needed to know the topics in this objective:

  1. vSphere 6.5 VCSA appliance monitoring and management (brianjgraf.com)
  2. know the commands:
    • vimtop
    • service-control
  3. Manage Platform Service Controller from the vSphere Web Client
  4. Maintaining a vCenter Server Database
  5. Monitoring the vCenter Server Appliance
  6. Know the VAMI interface options for monitoring performance
  7. Monitoring Host Health Status
  8. Generating Diagnostic Bundles
  9. System Log files
  10. Viewing Log files with the Log Browser
  11. Collecting vCenter Server Log files
  12. ESXi log files Locations
  13. Troubleshooting Overview
  14. vSphere CLI command Overviews
  15. vSphere 6.5 Troubleshooting Guide
  16. Collecting Logs for Troubleshooting a vCener Server Installation or Upgrade
  17. Resources Allocation Shares
  18. VMware Tools Upgrade Fails if VMware Tools is not Installed
  19. Increase Network Latency Observed in FT Virtual Machines