Analytical study of nonlinear oscillatory systems using the Hamiltonian approach technique