The Pahang Club was the earliest building built by the British in 1907. This building is the first site of the British governing Pahang. The house was built for the purpose of Hugh Clifford's residence that held the 'Supritendent' Ulu Pahang at that time. In June 1898, Kuala Lipis replaced Pekan as the Capital of Pahang and this house became the residence for the British Resident.
Hugh Clifford has chosen the Resident's Hill to build a new residence to facilitate the control of the Lipis area and the house was completed in 1922. After Hugh Clifford moved, the house was occupied by the British Police Inspector until 1926. He led the membership of the British Police where his members a total of 100 people comprising Sibu, Dayak, and Malay. In mid-1926 the house was abandoned and the British government made it a special club for British officials.
The house has not been done any modifications aimed at maintaining the authenticity of the art of art. When it became Pahang Club, the house was closely guarded. According to a gardener, Pak Razak is not a man who can go to the house which must be dressed neatly, tied up and the shoes must be shiny. When the house was abandoned, it was once cast into castle but was silenced as the proposal was rejected by the palace party.
In 1957 club membership was only for dignitaries, British, and government officials. When Malaya is independent club membership is open for local people who are financially and irrespective of race. The club's activities are for recreational facilities, sports activities such as tennis, badminton and takraw and social activities.