High Performance Computing most generally refers to the practice of aggregating computing power in a way that delivers much higher performance than one could get out of a typical desktop computer or workstation in order to solve large problems in science, engineering, or business.
A HPC would have all the components which a server would have like Processors, Memory, GPU, High Speed Networking devices like 10G/40G Ethernet with either copper or fiber or Infiniband and Storage. The compute portion of the HPC cluster consists of a Head Node or Front End Node running scheduling and management software. Typical management tools are Bright Cluster Manager, ROCKS, OpenHPC etc.
Research institutes, universities and government labs.
Weather and climate research, bioscience, energy, military etc.