Built in 1874, the Point Fermin Lighthouse was the first navigational light into the San Pedro Bay. Phineas Banning, with the support of many local businessmen, petitioned the Federal Government and the US lighthouse Board to place a lighthouse on the point in 1854. Although the Lighthouse Board agreed funding and land disputes delayed its construction until 1874.

Paul J. Pelz, a draftsman for the US Lighthouse Board, designed the Stick Style Victorian lighthouse. The design was used for six lighthouses built between 1873 and 1874, of which three are still standing, East Brothers in San Francisco Bay, Hereford Light in New Jersey, and Point Fermin. The Stick Style is an early Victorian architectural style and is simpler in design and decoration than the later high Victorian period. It is characterized by its gabled roofs, horizontal siding, decorative cross beams and hand carved porch railings.