Assembly of handle set and cylinder lock