Lionel Finlay said the seeds of Labor's shock Hunter vote last weekend were sown long ago