There is a direct bus from Matara to Kalmunai which stops in Pottuvil leaving daily at 11:50am. This should give you time to take an early bus from any south coast origin such as Hikkaduwa, Galle, Unawatuna, Mirissa or Weligama and arrive at Matara before it leaves. The bus travels through Tangalle so you can get on there but a seat may not be guaranteed. After arriving in Pottuvil take a tuk tuk to Arugam Bay or wait for a bus to Panama.
Duration and Cost
According to the comments below, the journey time is about seven hours from Matara to Pottuvil and it costs 350LKR.
If you want to leave Matara earlier or later you can take a more regular bus to Monaragala and then change on to the Panama bus which stops in Arugam Bay. See the page Colombo to Arugam Bay by Bus for more info on arriving by bus.