There's a certain amount of overseeing the planting, the harvest, the collection of crops, and the collection of taxes/render.
There's a manorial court to sit in most months to deal with any crime that has happened that can be handled with fines and floggings. There's also taking any cases that might require death or dismemberment to the Count's (and if necessary, the King's) Court.
There's visiting friends and family and attending court when the Count wishes them to do so.
Potentially if a knight is accused of a crime, the PK might be expected to sit as part of a panel on the case in the appropriate Lord's Court.
There's escorting or accompanying the Count on his progress around the County.
Potentially there's meeting friends/allies/heralds at the borders and escorting them politely to the Count, although this might overlap with military duty and riding the borders.
And there's always the fun and games (especially at the edges of the County) of having to deal with raiders and bandits trying to steal or burn your manor, or that of your friends or neighbours.
Indeed, you might have to defend the manor of a neighbour who is away on military service.
.... and then there's extracurricular raiding of your own.
Hunting for pleasure or for the table is another possibility, or hunting to prevent wildlife from damaging the land (particularly boar, bear, and wolves, although deer can do quite a number on a forested or planted area).
There's overseeing any maintenance of the improvements, buildings, and facilities on your lands, and assessing whether you could build anything else.
There's overseeing the breeding of your horses and dogs (especially at the one manor stage where you might not be able to afford a constable and a master of hounds, and might have to do with your squire acting as groom/hostler, and a dog boy to watch the hounds.
And then you have military training for you, your squire, your levy, and any garrison you might have.