If it is only you that will arrive late or leave early then it won't make any difference. You just need to organise transport at your own expense to meet the yachts when you arrive or find your way back to the mainland when you need to leave.
Ferry connections are good, so it isn't usually a problem. Just speak to your TYW skipper for assistance, or drop us an email to firstname.lastname@example.org.
The itinerary is set a couple of weeks prior to departure when you will be able to see exactly where the yachts are on your arrival/departure day.