EuroPLoP '08: Shepherding experience
About a month ago, I volunteered as a shepherd for EuroPLoP '08. I shepherded an interesting paper which I hope will be accepted at the conference. I don't think I can talk about the paper here because of confidentiality issues but I can at least describe my experiences with the shepherding process.
I did the shepherding process pretty much based on my own experiences both as a teaching assistant and a peer reviewer for my colleagues' papers. Even though this was my first time shepherding, I think I did a good job based on what my sheep and PC member said. Today, I was pretty much surprised to discover that someone had actually created a pattern language for shepherding -- The Language of Shepherding. I was also pleasantly surprised that most of my actions were inline with the patterns from the article.
The following paragraphs are titled with a pattern from the The Language of Shepherding article and my experiences with that pattern.
The Shepherd Knows the Sheep
Start immediately to build a relationship with the author, and maintain it throughout the shepherding. Contact the author right away, even before reading the patterns, and tell the author about yourself. Tell the author what to expect from you (such as three iterations), and establish what you expect from them in return. Above all, follow through on your commitments!I found that this works very well. In addition, I also asked the sheep about their own experiences with patterns so that I can tailor my comments to their experiences. I also asked about their long-term goals for their pattern; perhaps they are interested in compiling the pattern into a book or actually implementing the pattern in a working system. Having such questions help the me as the shepherd understand how to tailor the comments. Such questions also help remind the sheep of what they need to do to achieve their own goals for the paper.
Three Iterations
There is usually little time for reviews between the start of the shepherding process and the deadline for submission. Therefore, it is a good idea to establish a schedule at the beginning of the shepherding process. I have found that having an iteration be about 1.5 weeks long works out well -- a week is usually a bit too short. This allows for some flexibility in the schedule in case some other commitment pops-up halfway through the shepherding process. Once you have established a schedule (make this concrete by listing the actual dates that you expect an iteration) both parties should agree to stick with it. Having a complete paper at the end of an iteration (vs just having chunks here and there) is also better since it allows both parties to gauge the current state of the paper in its entirety and examine its cohesion.
It also helps to have a plan for each iteration. The first iteration should focus on the big picture of the pattern (and whether or not the paper is really a pattern). Subsequent iterations would be geared toward polishing the paper and making it more cohesive.
The hardest thing about iterations is to be mindful of consistencies. As the paper undergoes various rewrites, some parts can become inconsistent and need to be adjusted. For instance making sure that the same terms are used consistently throughout the paper is an important consistency to check for.
Half a Loaf
I found that small manageable chunks of feedback work well. Small comments tend to be more detailed and concrete and are easier for the sheep to follow. Thus, as the shepherd you are less likely to ramble on too much or dwell into some meta-argument of the paper.
As the original article suggests, it might also be necessary to give a full loaf sometimes. For instance, it is important to provide important feedback as soon as possible even if this might be a big comment. For instance, some changes are big and need to be made since the entire paper hinges on it. It's best to deliver such feedback early even if it would require a lot of explanation/reworking.
This makes it possible to produce a polished paper within the time constraints by addressing the most important issues first.
Small Patterns
Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.As the pattern progresses through comments from the shepherd and other potential peer reviewers, there is the tendency for the pattern to accrue too much description. It then becomes extremely hard to distill the gist of the pattern.
-- Antoine de Saint-Exupery, French writer (1900 - 1944)
So it is prudent to keep the pattern short. It is best to prune pathological cases from the pattern that practitioners of the pattern are unlikely to encounter. Or it could also be a symptom that the pattern should be broken down into smaller ones. Anyway, a huge pattern is a kind of "pattern smell" and should be examined thoroughly.
Here are some of my own techniques that were not actually mentioned in the original article but which I found helpful.
Sheep Bleats
Usually as a shepherd you would be familiar with the domain of the paper. However, sometimes there will be certain details that you are not familiar with. Or maybe the sheep makes a very compelling argument for something that you are almost certain is wrong. Nonetheless you want to avoid unnecessary conflicts/arguments.
Therefore, give your sheep the benefit of doubt. Ask the sheep for resources to justify and support their claims. Not only will this help make the paper stronger but as the shepherd you also get the chance to improve your own knowledge on the domain. Even if the sheep is wrong, this experience has helped clarify any misconceptions in a non-threatening way.
Multiple Examples
It's hard to explain something just by giving a meta-argument of the issue.
Therefore, spend the extra time to give a few examples on the issue and let the sheep experience an "aha" moment. Being able to give multiple examples usually clarifies the issue for both the sheep and the shepherd.
Interlaced Comments
This appeared as a candidate patlet in the original article.
You want to point out a particular section of the pattern that requires work by the author. Usually what you would do is write down the page and paragraph number and then describe the problem in the e-mail. However this has some potential problems:
- it becomes tedious and confusing to refer to multiple sections.
- there is the potential for miscommunication since people in different countries use papers of different sizes. So the page and paragraph number might be off.
Therefore, put the comments in-line in the original document. This could be done using pencil/pen/colored markers/ etc and then have the document scanned for sending. Or it could be done using a tool like Adobe Acrobat or Microsoft Word which allows comments to be embedded. Using such tools also provides automatic comment tracking -- to see which issues have been addressed.
If the use of such tools is not possible it might also be prudent to label the comments so that they can be referred to easily in subsequent follow-ups.
Using a pencil/pen or a tool also allows the embedding of non-text comments e.g. images. drawings, circles which might make it easier to communicate.
Modern Communication
Sometimes it is hard to express something in text. It could either get very verbose or you are afraid that there is a chance for misinterpretation. After all, e-mail doesn't really convey emotion that well.
Therefore, when possible it might be a good idea to offer other ways of communicating i.e. using Skype. Verbal communication is usually better when both parties are fluent in the same language but can be problematic otherwise.
I offered this option to my sheep but we never got around to using it. My friend Maurice had a chance to use it with his sheep and he told me that it worked well. Setting up a Skype session takes a lot prior arrangement from both parties since it requires you to set up a meeting that could span multiple time zones.
As far as I know, The Language of Shepherding has not been revised/updated since its inception in 1999. So hopefully the details of my experiences will highlight the fact that the article is still very much relevant for use circa 2008. Moreover, perhaps someone would also benefit from some of my suggested patlets.
Tweetcomments powered by Disqus