There is a super luxury ac bus which leaves Colombo at 9pm and arrives at Akkaraipattu (50km north of Arugam Bay) at 5am. From there it is possible to take a local bus to Pottuvil or a tuk tuk to Arugam Bay for around 2500LKR. The return bus leaves Akkaraipattu at 8pm and arrives in Colombo at 4.30am.
The bus can be reserved at busseat.lk (opens in a new window). The price of a ticket as of December 2017 was 1100LKR.
Although it has air-conditioning and adjustable seats do not expect to sleep well as it is normal to play music or movies at high volume throughout the journey, even at night! The bus has storage available for luggage and surfboards if required.