Metamaterials are a type of man-made material that have unusual electromagnetic properties not found in nature. The multilayered highly anisotropic hyperbolic dispersion Al:ZnO/ZnO metamaterial is a type of optical metamaterial that has an important quality: the propagating electromagnetic wave can experience dielectric permittivity |ε|<<1, the epsilon- near-zero (ENZ) regime, for TM-polarized light. In this thesis, the optical mode properties of anisotropic rectangular waveguides are investigated. The eigenfrequency data, obtained using MATLAB and confirmed by numerical simulations, shows when the optical modes propagate in regular Zinc Oxide (ZnO) material, the cutoff frequency of the higher order mode has a higher magnitude for each waveguide size. As expected, the cutoff frequency of the mode decreases when the size increases, and the changes of each cutoff frequency of a certain mode are apparent. In comparison, the data obtained by numerical simulations using COMSOL Multiphysics shows that the multilayered Al:ZnO/ZnO rectangular waveguide has three unique properties: i) only lower order modes (TE11, TE12(21)) are allowed to propagate; ii) the cutoff frequency of each mode exhibits strong spectral shifts toward the epsilon-near-zero (ENZ); and iii) the frequency of the modes does not change as much in the ZnO material as the width and height change. For example, from 700 nm to 1000 nm, the frequency of ZnO changes from 1.6 × 1014 Hz to 1.12 × 1014 Hz, which is a 30% difference. For AZO, the frequency change is 6.2 × 1012 Hz from 1.6793 × 1014 Hz to 1.6172 × 1014 Hz, which is a 3.7% difference. The results presented in this thesis provide a better understanding of how the anisotropic waveguide is going to work in the hyperbolic dispersion regime based on Al:ZnO/ZnO nano-layered structures metamaterials. Lowest order modes at the ENZ spectral regime possess high optical confinement, which has a high potential for future photonic devices.